Corps de l’article

I. Introduction

It can be easily assumed that novice interpreters use different strategies from seasoned professionals. Moser-Mercer (1996:52) argues that it is even more understandable for novice learners to use different strategies from intermediate or advanced learners. Kalina (1994:229-231) also demonstrates that professionals face fewer interference problems and have lower correction rate for minor errors whereas students are over-concerned with even the slightest errors, thus sometimes negatively affecting the entire performance. According to Choi (2004), students will be judged by their performance as soon as they graduate and start to work in the interpretation market. But it is also a clear fact that students are not professionals. These would-be interpreters are in class not just to be judged by their end-product but also to learn. That is, to learn and practice until they get it right (Choi, 2004: 170).

Clear differences between students and professionals indicate a need to develop a different evaluation framework for students. A different framework that goes beyond judging students by their performance per se and branches out to equally assess the learning process is required. To that end, this study attempts to propose an outline for performance assessment and to introduce the learning curve concept as part of assessing the learning process. It should be borne in mind, however, that a more elaborate description of the performance assessment criteria and demonstration of the learning curve assessment criteria are not included in this article. Such issues will be elaborated in further studies in the future.

II. Overview of evaluation and assessment in teaching

2.1. Definition and purpose of evaluation and assessment

The term assessment has often been used interchangeably with the term evaluation. In fact, many of these terms are muddled in most people’s minds including those engaged in teaching. According to Satterly (1989), assessment is a general term that includes all the processes and products that describe the extent and nature of the student’s learning whereas evaluation means making value judgments of the effectiveness of teaching as a whole, which usually occurs after an assessment has been made (as cited in Child, 1973/2004: 361). According to Lefrançois (2000: 486-487), assessment is a term to appraise the student’s performance and to judge the learning process whereas evaluation concerns making value judgments of the adequacy of teaching and the extent to which teaching goals have been met.

In short, it can be concluded that the main purpose of assessment is to measure the student’s progress, to guide students and to examine the learning process whereas the purpose of evaluation is to judge the effectiveness of teaching, to see if the objective of teaching has been met and to inform both teachers and students of what is being expected.

2.2. Types of assessment

According to Child (1973/2004), four types of assessments are used depending on different purposes. They are pretask assessment, formative assessment, diagnostic assessment and summative assessment. First, pretask assessment aims at discovering the level of knowledge and skills of students before learning. It is often required that teachers, especially inexperienced teachers, devise a pretask assessment because it helps them have an idea about the level at which to pitch the course. Second, formative assessment, often used interchangeably with the term formative evaluation, is an on-going assessment method to assess the progress made in knowledge and skills during the learning process. In this type of assessment, the teacher’s intention is to optimize feedback by making students aware of their weak and strong points, thereby guiding them to make improvements. Third, diagnostic assessment occurs particularly to pinpoint the cause of difficulties displayed by students who are struggling in a specific area during the learning process and to help them overcome such obstacles. This type of assessment mostly occurs during the formative assessment period. Child (1973;2004) also stresses that formative and diagnostic assessments are more process-oriented approaches where students benefit the most since it provides feedback to remedy students’ shortcomings and to reinforce their strengths. Fourth, summative assessment, also referred to as summative evaluation, normally occurs in the middle or at the end of a course or lesson to grade the students. Summative assessment, a more product-oriented approach, does not usually analyze difficulties nor provide subsequent feedback to the students but serves as useful information for students, teachers and employers to measure the student’s learning results (Child, 1973;2004: 361-363).

2.3. Limitations

During the evaluation process, many types of measuring instruments and methods are applied. Lefrançois (2000:501) asserts that most of the problems relating to evaluation occur when these measuring instruments or methods fall short of two important standards, which are reliability and validity.

Reliability

According to Child (1973; 2004), reliability in evaluation is often referred to as consistency as well. Child (1973; 2004) points out that inconsistencies arise in one examiner or between several examiners over the same examinee due to fatigue, mood, different expectations and interpretations of the examinee’s answers, experience or propensity of the examiners. To overcome such risks, Child (1973; 2004) underlines the need for the examiners to coordinate the allocation and range of scores based on clearly defined criteria beforehand. Increasing the number of examiners or the length of examination, as mentioned by Child, can help enhance reliability as well. Examiners should also be cautious with borderline scores that should always be reexamined (Child, 1973; 2004: 368-370).

Validity

According to Child (1973; 2004), validity is obtained when a test achieves what it has originally intended to achieve. Tests cannot be unreliable and valid but can be reliable and invalid since a reliable test can reveal unintended results (Child, 1973; 2004: 370-371). Lefrançois (2000) emphasizes that validity is most crucial especially for measuring instruments because test results would be of no use if a test fails to measure what it has intended to measure. In this context, educational or psychological tests are often challenged for their low validity deriving from unexpected variables (Lefrançois, 2000: 501).

III. Evaluation in the interpretation classroom

Choi (2003) describes what usually happens in a typical consecutive interpretation classroom in Korea. The student or the teacher improvises or reads the source text aloud. After the speech, one of the students is chosen to perform or voluntarily performs in front of the others. Students, then, provide oral feedback on their colleague’s performance by indicating what went wrong content wise and expression wise, which is followed by the teacher’s oral feedback (Choi, 2003: 223-224). In the interpretation classroom in Korea, most evaluation occurs through mid-terms, finals or during each class performance.

It is indispensable for would-be interpreters to be trained and evaluated in accordance with professional standards since they receive training in school to eventually work in the professional market. It is, therefore, natural for teaching to focus on interpretation skills, strategies and on how to teach them effectively. This enables students to live up to professional standards and to cope with problems that will occur in their career as interpreters. An extensive range of studies has already been conducted on what should constitute professional standards by Kurz (1993), Pöchhacker (1994) and Moser-Mercer (1996, 1998). In most cases, it would be plausible to say that multiple criteria components introduced in these studies can be explained by accuracy of meaning, appropriate expressions and presentation.

Choi (1998, 2004) describes some of the problems that interpreters encounter in the professional market, which concern terminology, preparation time, method, material and other variables. Regarding the tasks that an interpreter has to face in terms of preparation, she particularly classifies the preparation stage into three subcategories such as preparation of subject matter, terminology and last minute preparation. It has been demonstrated in her study (Choi, 2004) that one of the inevitable challenges that an interpreter has to cope with is the lack of preparation time and material especially during the busy conference season when the demand for interpretation reaches its peak and the interpreter is, thus, given so little time to prepare.

It is true that interpretation skills and strategies are of great significance in teaching. As a result, interpretation strategies and methods have been the bulk of studies pertaining to teaching interpretation. However, evaluating students entirely based on a yardstick geared towards professional standards runs the risk of defeating the very purpose of evaluation and assessment from a pedagogical standpoint. As mentioned before, the purpose of assessment is to measure the student’s progress, to guide students and to examine the learning process while the purpose of evaluation is to judge the effectiveness of teaching and to see if the objective of teaching has been met.

Moser-Mercer (1996) and Gile (2001) have already reiterated the need to evaluate would-be interpreters differently. They both point out that assessment in the classroom has mostly been studied from the standpoint of professionals who usually judge performance on their self-established professional criteria or expectations of the actual users in the market. It is essential for assessment in the classroom to be judged from the standpoint of professionals since students should be aware of what is being expected from them in the professional market and to prepare themselves accordingly. It is, however, equally important for assessment to be studied from the student’s standpoint as well. This is where student self-evaluation plays a vital role.

Evaluation does not only play a determinant role during the course of learning but it also plays an influential role in selecting students before they even start to learn. Moser-Mercer (1994:65-66) suggests that though it is true that every school will have its own way of assessing a student’s aptitude, a consensus can be reached on how to assess students since the ultimate goal shared by any training program is to develop students into competent professionals.

Metacognitive evaluation in this article, which will be discussed in 4., focuses on the implications of self-evaluation. The significance of self-evaluation in translation or interpretation classrooms has already been mentioned by Gile’s (1993) process-oriented approach, Sainz’s (1993) student-focus process, Moser-Mercer’s (1996) diary-based evaluation, Ficchi’s (1999) self-directed learning and Lee-Jahnke’s (2001) self-assessment. Despite the significance of self-evaluation displayed in previous studies, little study has been carried out to validate the hypothesis that self-evaluation can possibly lead to actual improvements in the student’s learning process and performance in the area of interpretation.

IV. Metacognitive evaluation

4.1. Definition of metacognition

Metacognition is described as thinking of thinking, knowing what we do and do not know or learning how to learn. Metacognitive strategies include talking about one’s thinking, writing a thinking journal, self-evaluation and other strategies to solve problems that occur in the learning process (Blakey & Spence, 1990). The Oxford Dictionary of Psychology (2003) defines metacognition as follows;

Knowledge and beliefs about one’s own cognitive processes [...] The term is also sometimes applied to regulation of cognitive functions including planning, checking or monitoring as one plans one’s cognitive strategy for memorizing something, checks one’s accuracy while performing mental arithmetic or monitors one’s comprehension while reading [...]

In a nutshell, metacognition in learning can be described as the awareness of the learning process and the ability to adapt to challenges that occur during this process through effective strategies, thereby helping learners improve their learning capacity.

4.2. Purpose of metacognitive evaluation

It is true that it is often challenging to discern whether the student has a problem in comprehension or in expression. Sometimes problems could concern both comprehension and expression. The student could have mistranslated a certain sentence due to shortcomings in processing the previous sentence. It is evident that many variables exist to explain the cause of poor performance. The purpose of metacognitive evaluation is to encourage students to think about such problems by reflecting upon themselves through self-evaluation. Through the Metacognitive Model, which will be illustrated later, students are taught to self-evaluate. Metacognitive evaluation also aims at assessing the learning results of self-evaluation to ultimately assess the learning process.

V. Metacognitive evaluation method

5.1. Performance assessment

5.1.1. Definition of performance assessment

Lefrançois (2000: 488) describes performance assessment as an assessment that judges the actual performance of students in a teaching environment very similar and close to real life. Performance assessment is also referred to as authentic assessment, which Wiggins (1990) defines as directly judging student performance on worthy and practical intellectual tasks. It would not be an exaggeration to say that the bulk of evaluation in the interpretation classroom is based on performance assessments since most classes seek to simulate the actual situation that a professional will have to cope with in real life by often using authentic speeches, conducting mock conferences or providing students with first-hand experience to interpret real speakers themselves in school.

5.1.2. Assessment method

Assessment criteria

Performance assessment takes the form of summative assessment that occurs, as stated in 2.2., in the middle or at the end of a course or lesson to grade the students. To be able to assess performance, it is imperative to quantify performance on clearly defined criteria. As mentioned before, multiple criteria components in judging an interpreter’s performance can boil down to accuracy of meaning, appropriate expressions and presentation. In this article, assessment on accuracy of meaning mostly focuses on omission, addition and mistranslation that occur between the source and target texts. Appropriate expressions concerns grammar, terminology in the target text. Presentation can consist of multiple sub-parameters such as voice, speed, articulation, and rhythm, to mention a few.

Considering that the aforementioned criteria are still far from sufficient, further study will be carried out to specify the categories that constitute the performance assessment criteria.

Rating scale

It is true that rating performance has been highly disputed due to its subjective nature. Teachers often have to make sure that evaluation is valid and reliable, which has been the major challenge in evaluation as mentioned in 2.3.. Quantifying performance quality can somewhat minimize the risks that are inherent in assessments by making performance quality more measurable. The five-point Likert-type rating scale (as cited in Christiansen, 2005), commonly used as a data gathering tool in psychology and teaching, is employed as an assessment tool to quantify performance. Accuracy, expressions and presentation, the main elements of the performance assessment criteria, are scored respectively. For measurement purposes, each rating is assigned with points ranging from 6 to 10. For instance, if a student performance is excellent in accuracy, good in expressions and acceptable in presentation, the student receives 10 points for accuracy, 9 points for expressions and 8 for presentation. The rating scale will also be further elaborated in future studies. Meanwhile, an example of the rating scale sheet is illustrated in Table 1.

Table 1

Five-point scale based on the Likert-type rating scale

Five-point scale based on the Likert-type rating scale

-> Voir la liste des tableaux

Assessment criteria composition

After having been scored respectively through the five-point scale, the three categories are also given different weight value. Accuracy of meaning can be assumed to be the most determinant factor, followed by expressions and presentation when assessing student performance since students, particularly novice students, should be carefully guided to concentrate their efforts more on grasping the accurate sense than on demonstrating flowery expressions and superior presentation skills. Therefore, accuracy is multiplied by a factor of 5 and expression by a factor of 3. Presentation is multiplied by the lowest factor of 2 to make sure that nobody with excellent presentation skills will be able to compensate for poor accuracy. Following is a sample of the performance assessment sheet based on the aforementioned criteria and the rating scale.

Table 2

Performance assessment sheet

Performance assessment sheet

-> Voir la liste des tableaux

5.2. Learning curve assessment

5.2.1. Definition of learning curve

According to the Oxford Dictionary of Psychology (2003),

Learning curve is a mathematical expression of the change in behavior occurring as a function of practice, usually represented by a graph showing a measure of performance on the vertical axis and amount of learning represented by trials or time on the horizontal axis [...] The concept was introduced by the US psychologist Louis Leon Thurstone [...]

Constructing a learning curve could enable both teachers and students to better understand the learning process. In his study, Thurstone (1919) states that the level of attainment obtained per unit of practice decreases as practice increases. He admits that this tendency, called the law of diminishing returns, may not necessarily apply to the learning of foreign languages and other complex processes of learning where learning curve positively accelerates in the initial stage, remains steady during the course of learning with irregular changes from time to time. His study on learning curves advocates that continued learning in some fields may not demonstrate a rise of the practice limit where attainment stabilizes at a certain level. However, that limit of practice in the initial stage of learning can be, instead, achieved at a faster rate or with more consistency in learning (Thurstone, 1919:11-12). Atherton describes the learning curve as the amount learned against the number of trials or over time. Atherton also refers to consistency as the most vital factor upon which the progress of learning is assessed since novice learners can get the performance right by chance, which people call “beginner’s luck” (Atherton, 2003).

5.2.2. Assessment method

Five-Stage Metacognitive Model

Learning curve assessment is mainly based on the following Five-Stage Metacognitive Model.

Five-Stage Metacognitive Model

  1. Stage 1: self-evaluation (S[1]) / feedback (T[2])

  2. Stage 2: problem-finding (S,T) / student profiling (T)

  3. Stage 3: prioritization (S,T)

  4. Stage 4: practice (S)

  5. Stage 5: revaluation (S) / monitor (T)

Choi, 2004:181

According to Choi (2004), in stage 1, students record their performance in class and listen to it after class. They submit their tapes and self-evaluated texts to the teacher. The teacher listens to the tapes again and provides additional feedback. In stage 2, the student and teacher identify the root causes of the student's poor performance together, whether it be comprehension, expression, speed, note-taking skills, lack of practice or other shortcomings. As pointed out by Lim (2001;220), one of the problems students, particularly novice students, often face is that they have a tendency to want to cling to words and fail to disassociate the words from the idea. In parallel with the problem-finding process, the teacher develops the student’s individual file, what is referred to as a student portfolio by Lefrançois (2000: 487), containing the samples of their self-evaluated texts, their fundamental weaknesses, strengths, habits, personality traits, anything that the teacher believes is worthwhile to keep track of the student’s learning process. In stage 3, students prioritize the problems with the help of the teacher when multiple problems are interwoven since they cannot address them all at once. In stage 4, students concentrate their practice on the most critical issue based on the assessment made in stage 1, 2 and 3. In stage 5, students revaluate themselves while the teacher continues to monitor them on a regular basis (Choi, 2004: 182).

One of the weaknesses of this method, however, is that student performance is only audio-taped and not video-taped. For technical reasons, visual presentation skills such as eye contact and gestures, which are critical factors in consecutive interpretation, are not measured in performance assessment. However, they are dealt with in class to make sure that students do not disregard the significance of these skills.

Assessment procedure

A pretask assessment is carried out at the beginning of the course before they embark on learning. First-time learners in consecutive interpretation undergo the same memory test from their B language into A. If a class consists of second-time learners who have already learned consecutive interpretation in the previous semester, students will take a simple consecutive interpretation test. This is essential for the teacher to obtain information about the student’s outset level. Learning curve assessment also takes the form of formative and diagnostic assessments since the student is subject to the metacognitive model that enables both the teacher and the student to detect the student’s weaknesses, strengths and causes of poor performance. Summative assessment is also employed since the results of the mid-terms and finals are used to compare the degree of progress made between the pretask assessment, mid-term and final examinations.

While performance assessment concentrates on the results of performance, learning curve assessment puts more focus on the degree of progress made in performance during the course of learning. Other variables including self-evaluation capabilities may well play a significant role in assessing the learning patterns of individuals, which will determine the learning curve assessment criteria. Such issues will be specified in future studies.

VI. The Metacognitive Evaluation Method

Based on what has been discussed above, the Metacognitive Evaluation Method can be summarized as follows:

Table 3

The Metacognitive Evaluation Method

The Metacognitive Evaluation Method

-> Voir la liste des tableaux

VII. Conclusion and discussions

The purpose and method of metacognitive evaluation in the consecutive interpretation classroom have been discussed. On-going study, however, is required to elaborate the performance and learning curve assessments criteria. Empirical study based on research updates on the actual results of performance and learning curve assessments are also needed to support their criteria. The learning curve assessment criteria, in particular, needs to be specified through thorough and careful investigation to demonstrate the student’s learning process and to go as far as to translate qualitative data into a subsequent learning curve. This is indispensable to validate the hypothesis that the learning curve assessment in parallel with the performance assessment mechanism under the Metacognitive Evaluation Method can serve as a moderate step towards the development of a different method to assess students in the classroom and perhaps carefully predict their potential to make further advance in their career as professional interpreters.