LINGUIST List 17.205
|
Sat Jan 21 2006
Review: Applied Ling: Kiely & Rea-Dickins (2005)
Editor for this issue: Lindsay Butler
<lindsay linguistlist.org>
|
What follows is a review or discussion note contributed to our Book Discussion Forum. We expect discussions to be informal and interactive; and the author of the book discussed is cordially invited to join in. If you are interested in leading a book discussion, look for books announced on LINGUIST as "available for review." Then contact Sheila Dooley at dooleylinguistlist.org.
|
Directory
1. Fatos
Erozan,
Program Evaluation in Language Education
Message 1: Program Evaluation in Language Education
|
Date: 20-Jan-2006
From: Fatos Erozan <fatos.erozanemu.edu.tr>
Subject: Program Evaluation in Language Education
AUTHORS: Kiely, Richard; Rea-Dickins, Pauline
TITLE: Program Evaluation in Language Education
SERIES: Research and Practice in Applied Linguistics
PUBLISHER: Palgrave Macmillan
YEAR: 2005
Announced at http://linguistlist.org/issues/16/16-2802.html
Fatos Erozan, Department of ELT, Eastern Mediterranean University,
Famagusta, North Cyprus
OVERVIEW
This book contributes to the theory and practice of language program
evaluation significantly since it provides theoretical background to current
understandings of language program evaluation from historical
perspectives. Having dealt with theoretical background to language
program evaluation, the book considers various case studies with frequent
references to the theoretical information presented previously. The book
has four main parts. Part 1 ''Departure Points'' explains the research in or
theory of program evaluation from historical perspectives to set out the
background to current understandings of language program evaluation
through exploring three main fields, namely, social programs, educational
programs and the specific field of language program evaluation. Part
2 ''Cases and Issues'' presents seven evaluation studies, each of which
follows a different format and clarifies different aspects of the evaluation
process. These case studies are analyzed with reference to the concepts,
frameworks and issues explained in Part 1. Part 3 ''Evaluation Practice and
Research'' focuses on evaluation design and decision frames to guide
readers' evaluation practices and support their research into evaluation
processes. Finally, Part 4 ''Resources'' presents further resources for
language program evaluation: books, journals, professional associations,
ethical guides and best practice codes, e-mail lists and bulletin boards, and
additional internet resources.
SUMMARY
The book is arranged in four main parts, the first three of which are
organized in different chapters.
Part 1. Departure points
Chapter 1 ''Themes and Challenges'' outlines three features of evaluation
and explains five challenges for it. The features of evaluation concern (i) its
definition which has undergone paradigm shift throughout the history of
program evaluation, (ii) the two perspectives of evaluation research, i.e.
evaluation as a study which has both research and evaluation functions and
evaluation as research into its processes, and finally, (iii) insufficient
documentation and not publishing of evaluation reports. The five
challenges for evaluation include (i) the purpose of evaluation in its social
and political context, (ii) stakeholders (their participation, involvement and
investments), (iii) evaluation criteria generating evaluation frameworks,
instruments and judgments (Three approaches to specifying criteria for
making judgments of worth about language programs are theory-based
criteria derived from understandings of language learning process, policy-
based criteria established through professional considerations, and
constructivist or ethnographic approaches determining criteria through
internal program sense-making), (iv) evaluation data validating above-
mentioned approaches and instruments and completing the construction of
judgments, and finally (v) evaluation use to manage social programs
(considering new connections and new applications of evaluation processes
and findings such as links between evaluation and research and links
between evaluation and management).
Chapter 2 ''Historical Perspectives: Focus on Design and Method''
summarizes the early phases of evaluation theory and practice in social
programs generally and in education with emphasis on conceptual aspects
that have persisted over time and have the potential to develop current
theory and practice. In addition, this chapter presents a chronological
outline of a period when the focus of evaluation was on research design,
data collection methods and clear evaluation outcomes for decision making
and other judgments. This period reflects the shift of emphasis from
empirical evaluation studies to more comprehensive accounts of program
experience and the use of evaluation to understand develop or improve
programs (i.e. paradigm shift from product to process or process and
product oriented approaches to evaluation). The authors consider the
history of the development of ideas as ''a cumulative layering'' (p.17).
Chapter 3 ''Historical Perspectives: Focus on Context and Use'' focuses on
the issues of evaluation use in educational programs, the contribution of
constructivism and realism to the use of evaluation, and political and
managerial dimensions of the improvement task. Having explained
constructivism and realism in evaluation, the authors focus on the issue of
political side of program evaluation and the problem of conformity with
mandates.
Chapter 4 ''Historical Perspectives: Language Program Evaluation and
Applied
Linguistics'' considers program evaluation in language education and
applied linguistics. The authors outline the evaluation trends in these two
fields as: (i) the shift from an exclusive focus on measurement of outcomes
to the evaluation of actual curricular experience, (ii) an increased attention
to classroom processes, (iii) evaluation as the domain of professional
practice, (iv) the development of teachers' and other professionals' skills,
and finally, (v) inclusion of baseline and formative evaluation. The authors
also explain how the developments in program evaluation correspond to
developments in other fields and how the boundaries between research
and evaluation in language programs have been redrawn in the direction of
a movement from evaluation to research. Moreover, they indicate that
evaluation has recently been considered as a platform for internal sense-
making and development in language programs that are part of
development projects. Finally, they focus on a new dimension of language
program evaluation, which is to understand language use as a quality
indicator in non-language programs where communication and interaction
are main activities (i.e. applied linguistics perspectives in program
evaluation).
Part 2. Cases and Issues
In Part 2, the authors present seven different evaluation case studies
analyzing them with reference to the concepts, frameworks and issues set
out in Part 1. In the presentation of each case study, context, aim and the
scope of the evaluation, evaluation design or data collection procedures,
findings of the evaluation, and finally implications of the study for
evaluation were the issues explained and discussed.
Chapter 5 ''Evaluating Teachers' English Language Competence'' explains a
case study which aimed at evaluating Basic English teachers' level of English
language competence. Rather than being an evaluation of a particular
program, the study was conducted to gather data for national strategy
development. Design and validation of evaluation procedures such as
language tests, class observation bandscales and questionnaires, were the
focal points of this case study.
Chapter 6 ''Evaluating a Language through Science Program'' presents the
evaluation of the language component of the 'Science Across Europe'
Project which aimed to promote the use of a foreign language in working
on the science materials and the exchange of information with schools in
other countries through the use of information and communication
technology. The evaluation study was carried out to gather empirical data
about prevailing attitudes towards the program and how it was being
implemented with the goal of informing management decisions as regards
program development (for the program management) and deciding
whether or not there was value for money (for sponsors). Teacher and
student questionnaires were the data sources in this evaluation study.
Chapter 7 ''Evaluating the Contribution of the Narrative Speaker Teacher''
describes a large-scale evaluation of an educational pilot program in the
secondary school English language curriculum in Hong Kong, which
emphasized the importance of use of English in the classroom. The
evaluation study aimed to measure learning achievement caused by
increased use of English for communication, i.e. the contribution of native
speaker teacher as a linguistic and cultural resource. The evaluation used a
mixed method approach; that is, it consisted of two separate studies: (i) a
conventional experimental study based on learning outcomes, and (ii) a
qualitative study of cultural, affective and social facets of school life and
learning English (ethnographic case studies).
Chapter 8 ''Evaluating Foreign Language Teaching in Primary Schools''
focuses on the evaluation of a pilot program, an initiative, to introduce a
modern foreign language into the primary school curriculum in Ireland.
Concerning the evaluation design, as in the previous case study, two main
strategies were adopted: a survey of teacher attitudes and a study of 22
classes through assessment of learning outcomes, student attitudes and
teachers' assessment of students' competence. The authors identify the
strategies developed to elicit attitude and language performance data from
young learners and the interface with policy making and program
management as the two 'interesting' aspects of this evaluation study.
Chapter 9 ''Evaluating Program Quality in Higher Education'' presents an
evaluation case study for quality management in an English for Academic
Purposes program in a British university. Stakeholder approach to
evaluation was implemented in this study and the purpose of the evaluation
was to understand student learning experience and to use this
understanding for improving the program. Development-oriented use and
accountability (decision making)-oriented evaluation use were in the scope
of this case study, and mid- and end-course evaluations were carried out.
Chapter 10 ''Evaluating the Student Experience in Higher Education''
emphasizes methodological aspects of student evaluation, and the case
study presented in this chapter explores the student experience since the
views of students as service users and clients are viewed as key drivers of
policy and practice in current higher education, and therefore, the methods
in which these views are constructed and understood are important. As in
Chapter 9, a detailed account of an English for Academic Purposes program
evaluation is presented in Chapter 10. 'Nominal Group Technique' (group
discussion activity) was used as an evaluation instrument. The group
discussion as a language learning activity, the ways in which students hold
their stake (students' stakeholding), variations in status of different student
voices (equal status of all voices), and the role of the teacher were the
implications for evaluation raised in this case study.
In Chapter 11 ''Evaluating Assessment Standards and Frameworks'' two
evaluation studies, with specific reference to assessment frameworks, are
presented. The first study, which adopted documentary evaluation
approach, is an evaluation of national and state frameworks for the
assessment of learners of English as an additional language. The aim of the
study was to provide comparative data on the assessment systems from
different contexts of application. The second study, on the other hand, is
an empirically driven evaluation of Canadian Language Benchmarks and the
Centre for Canadian Language Benchmarks, which helped understand key
processes in the implementation of an evaluation study using electronic
communication. The authors claim that both studies highlight the relevance
of in-depth case studies to capture further insights about the assessment
frameworks, both perceptions and use (p. 199).
Chapter 12 ''Stakeholding in Evaluation'' focuses on the issue
of 'stakeholding'. Enhancing evaluation utilization is the motivation for
extending the involvement of stakeholders. Having defined the
concept 'stakeholder', the authors elaborate on different classifications of
stakeholders. They argue that students should be given strong voice in
program evaluation - adopting not only an informant role but a participant
role as well. They demonstrate the importance of stakeholder participation
and how it can be achieved, arguing that stakeholder participation ensures
validity, utilization and learning, the last of two which are the two main
goals for evaluation.
Part 3. Evaluation Practice and Research
Aiming to support evaluation practice and research into evaluation
processes, this part of the book details evaluation design and decision
frames.
Chapter 13 ''Large-scale Evaluations'' focuses on large-scale evaluations
and explores the design and implementation issues specifically. Having
discussed diversity in large-scale evaluations, the authors elaborate on
issues like the limitations of large scale evaluations, ways of framing focal
points for an evaluation, validation of evaluation issues and questions,
evaluation procedures, the reasons for and the ways of developing
evaluation skills, constraints to be worked with in evaluation, and ethical
dilemmas. Overall, in this chapter, three aspects of large scale evaluations
are emphasized: (i) understanding the evaluation construct, (ii) data
management, and (iii) participants and the report (p. 245).
Chapter 14 ''Teacher-led Evaluations'' explains some issues in the design
and implementation of teacher-led evaluations or evaluations that teachers
can conduct in their own teaching contexts and that correspond with action
research and reflective practice. These evaluations are opportunities to
evaluate curricular resources, learning materials and classroom tasks,
resources like information technology and libraries, and aspects of
interaction in the teaching learning process. Moreover, they serve a
professional development function. Aspects of the planned or intended
curriculum constitute focal points of teacher-led evaluations. After
describing the process of conducting these evaluations, the authors
present some sample projects to further inform the teachers about
purposes and process in evaluations that they can initiate in their contexts.
Chapter 15 ''Management-led Evaluation Projects'' considers management-
led evaluations and sets out two frameworks and three sample evaluation
designs which readers can use to conduct evaluations which aim to
understand program management issues and improve program
management processes. The authors argue that ''these evaluations
provided opportunities for learning and development within programs, as
in terms of program management, they furthered understanding of the
contribution of wider institutional policy on resources and support for
learning'' (p.271).
Part 4. Resources
Chapter 16 ''Resources for Language Program Evaluation'' presents further
resources for language program evaluation to help program evaluators to
engage with issues of evaluation theory, purpose and practice, examples of
data collection instruments and data analysis (i.e. evaluation design), and
evaluation reports. These resources are organized in sub categories as: (i)
books, (ii) journals, (iii) professional associations, (iv) ethical guides and
best practice codes, (v) e-mail lists and bulletin boards, and (vi) additional
internet resources.
CRITICAL EVALUATION
''Program Evaluation in Language Education'' is a comprehensive and
important book, which connects theory/research and practice in program
evaluation in language education exploring the relationship between the
two. The book is well organized and written with introduction and summary
sections in each chapter, which makes it reader friendly. One of the positive
attributes of this book is that the authors first explain the theoretical
background to language program evaluation in Part 1 and then exemplify
and further this explanation by presenting various evaluation case studies
in Part 2. Moreover, in Part 3, they particularly focus on evaluation design
and implementation of three main types of evaluation again supporting,
detailing and clarifying theoretical explanations with sample case studies.
Another important quality of this book is that it is a valuable addition to the
literature on the research and practice of language program evaluation.
Having reviewed some literature on the evaluation of second language
programs and language teacher education programs, it can be seen that
not much published research is available. In particular, there are not many
published studies which can inform evaluators, teachers or researchers
about language program evaluation case studies conducted in different
contexts. Therefore, this book can be considered an important addition to
the existing case studies of some researchers in the literature (Beretta &
Davies, 1985; Spada, 1987; Sharp, 1990; Alderson & Beretta, 1992; Weir &
Roberts, 1994; William & Burden, 1994; Mackay, Wellesley & Bazergan,
1995; Frad & Lee, 1997; Lee, 1998; Dushku, 1998; Lewkowicz & Nunan,
1999; Halbach, 1999; Jeffcoate, 2000; Ross, 2003; Kiely, 2003; etc.).
The first part of the book presents dense or complex theoretical
information concerning the historical perspectives in language program
evaluation. Therefore, readers who do not have sufficient background in
program evaluation theory or research may experience some difficulties in
following some discussions. Overall, I can highly recommend this book to
people who are equipped with some preliminary theoretical background in
the field and are already involved in language program evaluation as a
researcher, practitioner or the both.
REFERENCES
Alderson, J. C., & Beretta, A. (Eds.) (1992). Evaluating second language
education. Cambridge: Cambridge University Press.
Beretta, A., & Davies, A. (1985). Evaluation of the Banglore Project. ELT
Journal, 39(2), 121-127.
Dushku, S. (1998). ELT in Albania: project evaluation and change. System,
26(4), 369-388.
Fradd, S. H., & Lee, O. (1997). Teachers' voices in program evaluation and
improvement: A case study of a TESOL program. Teaching and Teacher
Education, 13(6), 563-577.
Halbach, A. (1999). Using trainee diaries to evaluate a teacher training
course. ELT Journal, 53(3), 183-190.
Jeffcoate, R. (2000). Teaching English grammar in initial teacher
training: A course evaluation. Educational Research, 42(1), 73-85.
Kiely, R. (2003). What works for you? A group discussion approach to
program evaluation. Studies in Educational Evaluation, 29, 293-314.
Lee, I. (1998). Supporting greater autonomy in language learning. ELT
Journal, 52 (4), 282-289.
Lewkowicz, J. A., & Nunan, D. (1999). The limits of collaborative evaluation.
TESOL Quarterly, 33(4), 681-700.
Mackay, R., Wellesley, S., & Bazergan, E. (1995). Participatory evaluation. ELT
Journal, 49(4), 308-317.
Ross, S. J. (2003). A Diachronic coherence model for language program
evaluation. Language Learning, 53(1), 1-33.
Sharp, A. (1990). Staff/student participation in course evaluation: A
procedure for improving course design. ELT Journal, 44(2), 132-137.
Spada, N. (1987). Relationships between instructional differences and
learning outcomes: A process-product study of communicative language
teaching. Applied Linguistics, 8(1), 137-161.
Weir, C., & Roberts, J. (1994). Evaluation in ELT. Oxford: Blackwell Publishers.
William, M., & Burden, R. (1994). The role of evaluation in ELT project
design. ELT Journal, 48(1), 22-27.
ABOUT THE REVIEWER
Fatos Erozan is a senior instructor in the Department of English Language
Teaching at Eastern Mediterranean University, North Cyprus. She holds a
Ph.D. in ELT (Middle East Technical University, 2005). Her research interests
are language program evaluation, curriculum design, and ELT materials
evaluation and design.
Respond to list|Read more issues|LINGUIST home page|Top of issue
|
|
Please report any bad links or misclassified data
LINGUIST Homepage | Read
LINGUIST | Contact us
While the LINGUIST List makes every effort to ensure the linguistic relevance of sites listed
on its pages, it cannot vouch for their contents.
|
|
