Malcolm Williams. Translation Quality Assessment: An Argumentation-Centred Approach. Ottawa, University of Ottawa Press, 2004, 188 p.

This is a book about assessing the quality of non-literary translations produced in a workplace rather than an educational setting. Its basic contention is that Translation Quality Assessment (TQA) systems currently in use are microtextual rather than macrotextual: they look for errors at the sub-sentence level; they “are not designed to assess each passage as an integral part of a whole … or evaluate the logic and coherence existing within the sample passage” (p. xvii).

The book begins with a statement of issues. Williams notes that there is no agreement about criteria: how important are typos? elegant style? minor shifts in meaning? He announces the issues he will focus on: Is it alright to use samples rather than assess an entire text? What is a major as opposed to a minor error? If a system involves counting errors, what about the problem that a text with x errors will be satisfactory but a text with x+1 errors unsatisfactory? Finally, if you are rating many different factors, how do you derive an overall assessment: will each of the major errors and each of the minor errors have the same weight?

Chapter 1 reviews existing TQA systems used in either a professional or educational context. Williams first considers quantitative systems, those that count errors and output a rating which is interpreted in terms such as ‘deliverable to client’ vs ‘worth revising’ vs ‘unusable’. He considers the SICAL system formerly used by the Canadian federal government’s Translation Bureau, the Ontario Government’s system, and the system used by the Canadian Translators and Interpreters Council to mark certification examinations. He then looks at non-quantitative models of TQA proposed by Reiss, Nord and House. He notes that many systems (though not those actually in use in a professional context in Canada) refer to discourse-level errors. In particular a student-evaluation system proposed by Bensoussan & Rosenhouse recognizes that the message can get through despite micro-level errors; and a system suggested by Larose sees the seriousness of an error in terms of the level of macrostructure affected by it.

For some reason there is a rather lengthy discussion of Toury’s concept of norm and its elaboration by Nord and Chesterman. Norm in this sense is the target culture’s concept of what a good translation is; Toury derives it by observing how people do actually translate; it is not given prescriptively ahead of time. This, as Williams himself notes, is a concept from descriptive-explanatory Translation Studies rather than applied TS (under which TQA falls).

Chapter 2 presents Argumentation Theory. Argumentation is used in a broad sense that covers any sort of text (narrative, dialogue, description as well as argument proper). Williams proposes to use a system taken from Stephen Toulmin which picks out a text’s argument schema in terms of six elements: the Claim, Grounds, Warrant, Backing, Qualifiers and Rebuttals (CGWBQR) that constitute the message of a text. In a text of any length, there may be several CGWBQR sequences embedded in larger sequences, e.g. the Grounds could have its own internal CGWBQR structure. Williams gives a short (8-line) French text and proposed English translation, analyses the CGWBQR of the French, and finds that the translator has misconstrued C and W.

Chapter 3 discusses the relationships among the parts of an argument. Parts of varying size are linked by relations such as problem-solution, conclusion-reason, opinion-evidence, question-answer. The progression of the argument is signposted by various devices such as connector words. Williams rightly emphasizes the enormous importance of connectors, which are typically overlooked in micro-level assessment systems. He gives a French passage along with a translation in which some connectors are well handled, some not. A French-English translator can easily throw the reader right off the track of the argument by mishandling words like d’ailleurs, donc or en effet.

Some readers may find the remainder of Chapter 3, which looks at further aspects of textual rhetoric, somewhat heavy going. The text (originally a doctoral thesis) contains many categorizations and subcategorizations. I found myself getting lost occasionally, and some of the signposts were not helpful: “the third component of rhetoric” introduces the section on figures of speech on p. 58, but what were the first two components? (On the table on p. 64, figures appear to be the fifth component). Some paragraphs are laden with lists:

Figures … will more often than not be examples of what Dubois et al. call metalogisms or manipulation of logical relations (litotes, hyperbole, repetitions, pleonasms, antithesis, euphemism, irony, paradox, etc.). Metalogisms are translatable, unlike many other types of figures. Semantic figures (metasememes in Dubois’s terminology) include synecdoche, simile, metaphor, metonymy and oxymoron.

p. 60

These listings neither summarize previous material nor introduce detailed study of each figure. The editor should have intervened here.

Chapter 4 discusses the concept of major error. The SICAL system defines it as a complete failure to render an essential part of the message, but does not say what counts as essential. In Williams’ approach, what is essential is the six elements of the argument schema. As a result, an error is major only if it affects the rendering of one of these elements.

Chapter 5, which occupies one-third of the book, contains detailed analyses of four texts. To fully appreciate Williams’ approach, it is necessary to read these analyses, but that will not be possible for readers with no French. As with most TS books and journals published in Canada, it’s assumed that readers know both of Canada’s official languages. Perhaps it is time to adopt the practice of glossing material which is not in the language of the publication.

Williams starts by establishing the argument schema of the source text. Then he reads the translation without reference to the source text, paying particular attention to coherence. He then checks the translation to see if it reflects the schema. Finally, he checks for preservation of the various rhetorical structures discussed in Chapter 3.

A 400-word passage from each translation is then assessed using two microtextual systems—SICAL and the Ontario Government system. The result: with three of the four texts, the overall judgment of satisfactoriness obtained with Williams’ system differed from that obtained with the microtextual system. Also, in three cases, major errors did not affect the rating because they fell outside the 400-word passages selected for applying the microtextual system. Williams is quite right that arbitrary sampling of passages for quality control may completely overlook the crucial parts of a text.

Most interestingly, there were several cases where an error deemed minor from a microtextual viewpoint was seen to be major when looked at in terms of the argument schema, and vice versa. Since there is no space here to cite lengthy passages from the texts, let me illustrate with an imagined text in which an automobile is described in French as ‘bleue’ (blue) and in the English translation as ‘brown’. (Perhaps the French word for brown appeared nearby in the text, or perhaps the translator’s concentration lapsed and he wrote a colour word beginning with ‘b’.) Is this a major error? From the point of view of a language teacher, or a strict interpretation of a microtextual system, it is. But from a macrotextual point of a view, it may or may not be major. If the text is a police stolen car report, then it’s an error which could well make the translation useless. But it’s not hard to imagine a context in which the colour of the car is completely immaterial to the text’s argument schema.

This approach means that what is traditionally considered a major error, like writing nonsense or writing the opposite of what the source text says, may turn out not to be critical. (Part way through the book, Williams makes a terminological switch, distinguishing critical errors from major and minor errors. A critical error makes a translation unusable because of failure to render a component of the argument schema. Unfortunately this term, ‘critical’ as it is to Williams’ argument, is missing from the book’s subject index!)

Chapter 6 introduces an interesting method of assigning a numerical value to the assessment. The rating grid has two sets of parameters. The core parameters, which must be rated for every text, cover the argument schema components and the various rhetorical features of Chapter 3. Then there are what Williams calls the field― or use-specific parameters. These are things like terminology, formatting and TL quality, and it is up to the evaluator to decide which of these to include. So an evaluator may decide to rate or not to rate a feature such as conciseness or plain language, depending on the text type and the use to be made of the translation. This does seem like a good idea, since rendering the main argument does not exhaust what is expected of a translator. On the other hand, it may strike the reader as a somewhat mechanical grafting of features of microtextual assessment systems onto argumentation-based TQA.

Williams assigns a numerical rating using a method inspired by Larose and ultimately Nida. Each of the core and use-specific parameters is given a weighting as well as a minimum requirement for that text (using 10=excellent, 8=very satisfactory, and so on). The argument schema always gets a weighting of 3 and a minimum requirement of excellent (=10), for a required score of 30 on this parameter. With one of the texts from chapter 5, Williams gives ‘typography’ a weighting of 2 and a requirement of very satisfactory (=8), so that a score of 16 is required; for another text, he gives ‘typography’ a weighting of 1 and a requirement of satisfactory (=6), so that a score of 6 is required. The total minimum required score thus varies with the text. Appendix 1 gives a sample scoring of one of the texts from chapter 5.

Chapter 7 is about standards. Existing national standards such as Germany’s DIN 2345, Williams points out, govern procedures for achieving quality rather than making normative statements about what constitutes quality. The author proposes three standards: publication standard, information standard and minimum standard. In all three, all components of the argument schema must be rendered. They differ with regard to the need to meet target-language quality criteria (this being vital for publication), and to the permissibility of non-critical defects.

The Conclusion emphasizes Williams’ main point that in argumentation-based TQA, words and phrases are analysed not for their correspondence to microunits of the source text but for their contribution to the argument schema. “The stumbling block to acceptance and adoption of translation quality standards over the years has, in my view, been the failure to justify quantitative standards and assessments with reference to the highest norm of understandability, effective communication, and by extension, text usability. In other words, the criticisms levelled at texts deemed unsatisfactory were not justified with respect to the norm of communication” (p. 152).

Williams’ approach eliminates the problem of borderline cases with microtextual systems (1 extra minor error can put a text over the allowable limit, making it unsatisfactory) because the scoring rests heavily on critical defects, those pertaining to the successful rendering of the CGWBQR schema. Also, Williams claims that reference to the argument schema “leaves little margin for variation and inconsistency between assessments” (p. 150). This is probably true—the argument of a text will usually not be a matter for debate, whereas a system in which the score relies heavily on counts of minor error will be more open to subjectivity. The book ends with a TQA terminology listing, a list of works referred to in the text, and a list of further readings under six headings.

Williams was formerly in charge of running the federal Translation Bureau’s SICAL system, and is therefore eminently qualified to write about the problems of a microtextual approach. However he might have distinguished between the system itself and the translators who make the assessments. It is indeed a bad thing that the SICAL system did not explicitly draw the assessor’s attention to the macrotextual argument; however it does not follow that the assessors regularly ignored connector words and the like. They were, after all, experienced translators who would surely take text macrostructure into account when they translated; it would be odd if they completely ignored it when assessing.

As someone who assesses translations regularly, I kept wondering, as I read Williams’ book: how long would it take to quality control a 5000-word source text and translation using his method? To apply it requires first identifying the CGWBQR for the entire source text; second, making a detailed analysis of those passages in the translation that directly express these six elements; and third, assigning the weightings and the minimum required scores to the various parameters, and making the calculations. The third step could perhaps be skipped by rating the translations in purely business terms; that is, in terms of the number of minutes per 1000 words required to fix the translation using Williams’ criteria. Even then, however, I do not see how the first two steps could be done in a reasonable amount of time, i.e. how a large translating organization could quality-control the many tens of thousands of words of contracted work that arrive at its offices every day. The method could, however, profitably be applied in semi-annual audits of already delivered work done in-house. An organization could randomly select a few delivered texts and use Williams’ method to identify problems that might call for training.

Despite a few weaknesses, this book is a significant contribution to the literature on assessment because of its insistence on attending not to every little detail but to just those details which are important in a translation.