Date: Thu, 19 Jan 2006 11:42:41 +0200 From: Fatos Erozan Subject: Program Evaluation in Language Education
AUTHORS: Kiely, Richard; Rea-Dickins, Pauline TITLE: Program Evaluation in Language Education SERIES: Research and Practice in Applied Linguistics PUBLISHER: Palgrave Macmillan YEAR: 2005
Fatos Erozan, Department of ELT, Eastern Mediterranean University, Famagusta, North Cyprus
This book contributes to the theory and practice of language program evaluation significantly since it provides theoretical background to current understandings of language program evaluation from historical perspectives. Having dealt with theoretical background to language program evaluation, the book considers various case studies with frequent references to the theoretical information presented previously. The book has four main parts. Part 1 ''Departure Points'' explains the research in or theory of program evaluation from historical perspectives to set out the background to current understandings of language program evaluation through exploring three main fields, namely, social programs, educational programs and the specific field of language program evaluation. Part 2 ''Cases and Issues'' presents seven evaluation studies, each of which follows a different format and clarifies different aspects of the evaluation process. These case studies are analyzed with reference to the concepts, frameworks and issues explained in Part 1. Part 3 ''Evaluation Practice and Research'' focuses on evaluation design and decision frames to guide readers' evaluation practices and support their research into evaluation processes. Finally, Part 4 ''Resources'' presents further resources for language program evaluation: books, journals, professional associations, ethical guides and best practice codes, e-mail lists and bulletin boards, and additional internet resources.
The book is arranged in four main parts, the first three of which are organized in different chapters.
Part 1. Departure points Chapter 1 ''Themes and Challenges'' outlines three features of evaluation and explains five challenges for it. The features of evaluation concern (i) its definition which has undergone paradigm shift throughout the history of program evaluation, (ii) the two perspectives of evaluation research, i.e. evaluation as a study which has both research and evaluation functions and evaluation as research into its processes, and finally, (iii) insufficient documentation and not publishing of evaluation reports. The five challenges for evaluation include (i) the purpose of evaluation in its social and political context, (ii) stakeholders (their participation, involvement and investments), (iii) evaluation criteria generating evaluation frameworks, instruments and judgments (Three approaches to specifying criteria for making judgments of worth about language programs are theory-based criteria derived from understandings of language learning process, policy- based criteria established through professional considerations, and constructivist or ethnographic approaches determining criteria through internal program sense-making), (iv) evaluation data validating above- mentioned approaches and instruments and completing the construction of judgments, and finally (v) evaluation use to manage social programs (considering new connections and new applications of evaluation processes and findings such as links between evaluation and research and links between evaluation and management).
Chapter 2 ''Historical Perspectives: Focus on Design and Method'' summarizes the early phases of evaluation theory and practice in social programs generally and in education with emphasis on conceptual aspects that have persisted over time and have the potential to develop current theory and practice. In addition, this chapter presents a chronological outline of a period when the focus of evaluation was on research design, data collection methods and clear evaluation outcomes for decision making and other judgments. This period reflects the shift of emphasis from empirical evaluation studies to more comprehensive accounts of program experience and the use of evaluation to understand develop or improve programs (i.e. paradigm shift from product to process or process and product oriented approaches to evaluation). The authors consider the history of the development of ideas as ''a cumulative layering'' (p.17).
Chapter 3 ''Historical Perspectives: Focus on Context and Use'' focuses on the issues of evaluation use in educational programs, the contribution of constructivism and realism to the use of evaluation, and political and managerial dimensions of the improvement task. Having explained constructivism and realism in evaluation, the authors focus on the issue of political side of program evaluation and the problem of conformity with mandates.
Chapter 4 ''Historical Perspectives: Language Program Evaluation and Applied Linguistics'' considers program evaluation in language education and applied linguistics. The authors outline the evaluation trends in these two fields as: (i) the shift from an exclusive focus on measurement of outcomes to the evaluation of actual curricular experience, (ii) an increased attention to classroom processes, (iii) evaluation as the domain of professional practice, (iv) the development of teachers' and other professionals' skills, and finally, (v) inclusion of baseline and formative evaluation. The authors also explain how the developments in program evaluation correspond to developments in other fields and how the boundaries between research and evaluation in language programs have been redrawn in the direction of a movement from evaluation to research. Moreover, they indicate that evaluation has recently been considered as a platform for internal sense- making and development in language programs that are part of development projects. Finally, they focus on a new dimension of language program evaluation, which is to understand language use as a quality indicator in non-language programs where communication and interaction are main activities (i.e. applied linguistics perspectives in program evaluation).
Part 2. Cases and Issues In Part 2, the authors present seven different evaluation case studies analyzing them with reference to the concepts, frameworks and issues set out in Part 1. In the presentation of each case study, context, aim and the scope of the evaluation, evaluation design or data collection procedures, findings of the evaluation, and finally implications of the study for evaluation were the issues explained and discussed.
Chapter 5 ''Evaluating Teachers' English Language Competence'' explains a case study which aimed at evaluating Basic English teachers' level of English language competence. Rather than being an evaluation of a particular program, the study was conducted to gather data for national strategy development. Design and validation of evaluation procedures such as language tests, class observation bandscales and questionnaires, were the focal points of this case study.
Chapter 6 ''Evaluating a Language through Science Program'' presents the evaluation of the language component of the 'Science Across Europe' Project which aimed to promote the use of a foreign language in working on the science materials and the exchange of information with schools in other countries through the use of information and communication technology. The evaluation study was carried out to gather empirical data about prevailing attitudes towards the program and how it was being implemented with the goal of informing management decisions as regards program development (for the program management) and deciding whether or not there was value for money (for sponsors). Teacher and student questionnaires were the data sources in this evaluation study.
Chapter 7 ''Evaluating the Contribution of the Narrative Speaker Teacher'' describes a large-scale evaluation of an educational pilot program in the secondary school English language curriculum in Hong Kong, which emphasized the importance of use of English in the classroom. The evaluation study aimed to measure learning achievement caused by increased use of English for communication, i.e. the contribution of native speaker teacher as a linguistic and cultural resource. The evaluation used a mixed method approach; that is, it consisted of two separate studies: (i) a conventional experimental study based on learning outcomes, and (ii) a qualitative study of cultural, affective and social facets of school life and learning English (ethnographic case studies).
Chapter 8 ''Evaluating Foreign Language Teaching in Primary Schools'' focuses on the evaluation of a pilot program, an initiative, to introduce a modern foreign language into the primary school curriculum in Ireland. Concerning the evaluation design, as in the previous case study, two main strategies were adopted: a survey of teacher attitudes and a study of 22 classes through assessment of learning outcomes, student attitudes and teachers' assessment of students' competence. The authors identify the strategies developed to elicit attitude and language performance data from young learners and the interface with policy making and program management as the two 'interesting' aspects of this evaluation study.
Chapter 9 ''Evaluating Program Quality in Higher Education'' presents an evaluation case study for quality management in an English for Academic Purposes program in a British university. Stakeholder approach to evaluation was implemented in this study and the purpose of the evaluation was to understand student learning experience and to use this understanding for improving the program. Development-oriented use and accountability (decision making)-oriented evaluation use were in the scope of this case study, and mid- and end-course evaluations were carried out.
Chapter 10 ''Evaluating the Student Experience in Higher Education'' emphasizes methodological aspects of student evaluation, and the case study presented in this chapter explores the student experience since the views of students as service users and clients are viewed as key drivers of policy and practice in current higher education, and therefore, the methods in which these views are constructed and understood are important. As in Chapter 9, a detailed account of an English for Academic Purposes program evaluation is presented in Chapter 10. 'Nominal Group Technique' (group discussion activity) was used as an evaluation instrument. The group discussion as a language learning activity, the ways in which students hold their stake (students' stakeholding), variations in status of different student voices (equal status of all voices), and the role of the teacher were the implications for evaluation raised in this case study.
In Chapter 11 ''Evaluating Assessment Standards and Frameworks'' two evaluation studies, with specific reference to assessment frameworks, are presented. The first study, which adopted documentary evaluation approach, is an evaluation of national and state frameworks for the assessment of learners of English as an additional language. The aim of the study was to provide comparative data on the assessment systems from different contexts of application. The second study, on the other hand, is an empirically driven evaluation of Canadian Language Benchmarks and the Centre for Canadian Language Benchmarks, which helped understand key processes in the implementation of an evaluation study using electronic communication. The authors claim that both studies highlight the relevance of in-depth case studies to capture further insights about the assessment frameworks, both perceptions and use (p. 199).
Chapter 12 ''Stakeholding in Evaluation'' focuses on the issue of 'stakeholding'. Enhancing evaluation utilization is the motivation for extending the involvement of stakeholders. Having defined the concept 'stakeholder', the authors elaborate on different classifications of stakeholders. They argue that students should be given strong voice in program evaluation - adopting not only an informant role but a participant role as well. They demonstrate the importance of stakeholder participation and how it can be achieved, arguing that stakeholder participation ensures validity, utilization and learning, the last of two which are the two main goals for evaluation.
Part 3. Evaluation Practice and Research Aiming to support evaluation practice and research into evaluation processes, this part of the book details evaluation design and decision frames.
Chapter 13 ''Large-scale Evaluations'' focuses on large-scale evaluations and explores the design and implementation issues specifically. Having discussed diversity in large-scale evaluations, the authors elaborate on issues like the limitations of large scale evaluations, ways of framing focal points for an evaluation, validation of evaluation issues and questions, evaluation procedures, the reasons for and the ways of developing evaluation skills, constraints to be worked with in evaluation, and ethical dilemmas. Overall, in this chapter, three aspects of large scale evaluations are emphasized: (i) understanding the evaluation construct, (ii) data management, and (iii) participants and the report (p. 245).
Chapter 14 ''Teacher-led Evaluations'' explains some issues in the design and implementation of teacher-led evaluations or evaluations that teachers can conduct in their own teaching contexts and that correspond with action research and reflective practice. These evaluations are opportunities to evaluate curricular resources, learning materials and classroom tasks, resources like information technology and libraries, and aspects of interaction in the teaching learning process. Moreover, they serve a professional development function. Aspects of the planned or intended curriculum constitute focal points of teacher-led evaluations. After describing the process of conducting these evaluations, the authors present some sample projects to further inform the teachers about purposes and process in evaluations that they can initiate in their contexts.
Chapter 15 ''Management-led Evaluation Projects'' considers management- led evaluations and sets out two frameworks and three sample evaluation designs which readers can use to conduct evaluations which aim to understand program management issues and improve program management processes. The authors argue that ''these evaluations provided opportunities for learning and development within programs, as in terms of program management, they furthered understanding of the contribution of wider institutional policy on resources and support for learning'' (p.271).
Part 4. Resources Chapter 16 ''Resources for Language Program Evaluation'' presents further resources for language program evaluation to help program evaluators to engage with issues of evaluation theory, purpose and practice, examples of data collection instruments and data analysis (i.e. evaluation design), and evaluation reports. These resources are organized in sub categories as: (i) books, (ii) journals, (iii) professional associations, (iv) ethical guides and best practice codes, (v) e-mail lists and bulletin boards, and (vi) additional internet resources.
''Program Evaluation in Language Education'' is a comprehensive and important book, which connects theory/research and practice in program evaluation in language education exploring the relationship between the two. The book is well organized and written with introduction and summary sections in each chapter, which makes it reader friendly. One of the positive attributes of this book is that the authors first explain the theoretical background to language program evaluation in Part 1 and then exemplify and further this explanation by presenting various evaluation case studies in Part 2. Moreover, in Part 3, they particularly focus on evaluation design and implementation of three main types of evaluation again supporting, detailing and clarifying theoretical explanations with sample case studies.
Another important quality of this book is that it is a valuable addition to the literature on the research and practice of language program evaluation. Having reviewed some literature on the evaluation of second language programs and language teacher education programs, it can be seen that not much published research is available. In particular, there are not many published studies which can inform evaluators, teachers or researchers about language program evaluation case studies conducted in different contexts. Therefore, this book can be considered an important addition to the existing case studies of some researchers in the literature (Beretta & Davies, 1985; Spada, 1987; Sharp, 1990; Alderson & Beretta, 1992; Weir & Roberts, 1994; William & Burden, 1994; Mackay, Wellesley & Bazergan, 1995; Frad & Lee, 1997; Lee, 1998; Dushku, 1998; Lewkowicz & Nunan, 1999; Halbach, 1999; Jeffcoate, 2000; Ross, 2003; Kiely, 2003; etc.).
The first part of the book presents dense or complex theoretical information concerning the historical perspectives in language program evaluation. Therefore, readers who do not have sufficient background in program evaluation theory or research may experience some difficulties in following some discussions. Overall, I can highly recommend this book to people who are equipped with some preliminary theoretical background in the field and are already involved in language program evaluation as a researcher, practitioner or the both.
Alderson, J. C., & Beretta, A. (Eds.) (1992). Evaluating second language education. Cambridge: Cambridge University Press,
Beretta, A., & Davies, A. (1985). Evaluation of the Banglore Project. ELT Journal, 39(2), 121-127.
Dushku, S. (1998). ELT in Albania: project evaluation and change. System, 26(4), 369-388.
Fradd, S. H., & Lee, O. (1997). Teachers' voices in program evaluation and improvement: A case study of a TESOL program. Teaching and Teacher Education, 13(6), 563-577.
Halbach, A. (1999). Using trainee diaries to evaluate a teacher training course. ELT Journal, 53(3), 183-190.
Jeffcoate, R. (2000). Teaching English grammar in initial teacher training: A course evaluation. Educational Research, 42(1), 73-85.
Kiely, R. (2003). What works for you? A group discussion approach to program evaluation. Studies in Educational Evaluation, 29, 293-314.
Lee, I. (1998). Supporting greater autonomy in language learning. ELT Journal, 52 (4), 282-289.
Lewkowicz, J. A., & Nunan, D. (1999). The limits of collaborative evaluation. TESOL Quarterly, 33(4), 681-700.
Mackay, R., Wellesley, S., & Bazergan, E. (1995). Participatory evaluation. ELT Journal, 49(4), 308-317.
Ross, S. J. (2003). A Diachronic coherence model for language program evaluation. Language Learning, 53(1), 1-33.
Sharp, A. (1990). Staff/student participation in course evaluation: A procedure for improving course design. ELT Journal, 44(2), 132-137.
Spada, N. (1987). Relationships between instructional differences and learning outcomes: A process-product study of communicative language teaching. Applied Linguistics, 8(1), 137-161.
Weir, C., & Roberts, J. (1994). Evaluation in ELT. Oxford: Blackwell Publishers.
William, M., & Burden, R. (1994). The role of evaluation in ELT project design. ELT Journal, 48(1), 22-27.
ABOUT THE REVIEWER:
ABOUT THE REVIEWER
Fatos Erozan is a senior instructor in the Department of English Language Teaching at Eastern Mediterranean University, North Cyprus. She holds a Ph.D. in ELT (Middle East Technical University, 2005). Her research interests are language program evaluation, curriculum design, and ELT materials evaluation and design.