Language Resources and Evaluation

Margaret King

TIM(ex-ISSCO)/ETI
University of Geneva
54 rte des Acacias
CH-1227 Carouge(GE).
Margaret:Kingissco.unige.ch

Abstract

This paper discusses the importance of three areas (dictionaries, corpora and evaluation methodologies) where collaborative work is putting in place part of the infrastructure needed for successful research and development in language engineering. It reviews past and present efforts, examines some of the major difficulties and problem areas and concludes with the need for a long term policy committed to the production, maintenance and distribution of language resources for a variety of languages.

Keywords: 
Language resources, dictionaries, corpora, evaluation, standardisation.

1. Introduction.
The presence of some five hundred people at the first Language Resources and Evaluation Conference in Granada in May of this year attests to the degree of importance currently being accorded to these issues. In this paper, I want to look at why this should be so, sketch some of the efforts underway and discuss some of the issues involved. Language resources will be interpreted here to mean electronic dictionaries and corpora, large collections of text in electronic form. 
The paper suffers from two main limitations. First, I shall concentrate mainly on work on written language. Spoken language is at least as important, and with changes in technology enabling more and more public use of computer based information systems is only likely to increase in importance. Because this is so, a great 

deal of effort is being invested in exactly the resources and evaluation area. But I can claim no expertise in speech, and there are important differences between spoken and written language which make it hazardous for the non-expert to blunder in where angels fear to tread. Those interested in the spoken language area are referred to the proceedings of the International Conferences on Spoken Language Processing, to the proceedings of the Eurospeech conferences or to Gibbon (1996).
Secondly, I shall concentrate mainly on work in Europe. A great deal has been done in North America, in Japan and in the French speaking community which will be unfairly neglected here. This in part reflects my own European perspective, but is also a consequence of space and time constraints. Those seeking further information in these areas are referred to Zampolli , Calzolari and Palmer (1994). Even within Europe, there are some efforts which should be mentioned and are not. The most glaring of these omissions is the EuroWordNet project (Vossen 98, Alonge 98, Roventini et al 98), whose aim is to produce a multilingual data base which records basic semantic relations between words.
Finally the Proceedings of the Granada Conference (Rubio et al., 1998) provide a wealth of papers about both speech and written language, without geographical constraints.

2. Dictionaries.
Thinking about electronic dictionaries is perhaps the easiest way both to grasp the importance of language resources and to understand some of the important issues in their creation and maintenance.
Almost every application of human language technology from the humble spelling checker to the most ambitious AI understanding system requires a dictionary, and for a realistic application the dictionary is likely to be very large. Yet dictionary coding, especially for the more ambitious applications, is a time consuming and expensive process; for example, literally hundreds of man years have gone into the coding of the dictionaries used by the European Commission's version of the Systran machine translation system. 
Would it not, then, be a good idea to invest in the creation of an electronic  dictionary for each language, which could then serve as a common resource for all those working on that language?
This insight is behind the Japanese Electronic Dictionary Project which has been working for some time now to establish such a resource for Japanese. 
It was also behind the organisation of a workshop "On Automating the Lexicon" organised in Grosetto, Italy in 1986, which proved to be seminal in promoting European efforts in the area. The workshop had a large group of influential sponsors; the European Commission, the Council of Europe, ACL, AILA, ALLC and EURALEX. Its aim was to survey research efforts, current practices and potential developments in work on the lexicon, machine readable dictionaries and lexical data bases, with special consideration of the problems caused by working in a multilingual environment. Most importantly, the workshop was to make recommendations. These recommendations, which were transmitted to the European Commission and also widely distributed, are summarised in Zampolli (1994) as follows: 

•	To establish procedures for creating mulitfunctional databases from the information  contained both implicitly and explicitly in those traditional dictionaries that exist in machine readable form.
	
•	To develop computational tools for more efficient handling of lexical and lexicographical data, and to provide 'workstation' environments within which these tools may be used by lexicologists and lexicographers.
	
•	To explore the possibility of creating multifunctional lexical data bases capable of general use, despite divergences of linguistic theories and differences in computational and applicational frameworks.
	
•	To study the possibility of linking lexical data bases and large text files, in both monolingual and multilingual contexts, in order to determine the most effective ways of exploiting the relationships among the various lexical elements.

What are the principle obstacles to such a programme? The first and most important is already given in the summary above: dictionary coding is not done in an intellectual vacuum. Both the information to be encoded and the organisation of that information depend both on the application for which the dictionary is intended and on an underlying linguistic theory of some sort, which may or may not be explicit. 
A couple of examples will help to make this more concrete. Consider first the dictionary associated with the kind of spelling checker integrated into most text processing systems. All that the spelling checker does is recognise whether or not a string of characters constitutes a legal word of the language. The decision is based on a character by character comparison of the characters in the text with a list of legal words: the dictionary. (More sophisticated spelling checkers may include an element of morphological analysis and/or generation, but the underlying principle remains the same). Dictionary coding here is reduced to a minimum, and consists of nothing more than listing the legal words.
But now imagine that the dictionary is also to be used for grammar checking. Even if the grammar checker can do little more than, for example, recognise potential problems of agreement ('the boys was swimming') some grammatical information on at least parts of speech will have to be added to the dictionary. The names of parts of speech are not universal: a noun for one person may be a substantive for another, one person may distinguish between count nouns and mass nouns (in order, for example to pinpoint 'butters are bad for you' as a possible mistake), yet another may sub-classify adjectives in order to be able to spot mistakes in adjective ordering ('the French black big cat' as opposed to ' the big French black cat', for example). 
Clearly, the more complex the information, the more difficult this problem will become. As an extreme example, imagine trying to reach consensus on how to define and structure the semantic information necessary to disambiguate the pronoun in 'The farmer's wife sold the cow because she needed money' where the interpretation is exactly opposite in 'The farmer's wife sold the cow because she didn't give enough milk.'
Despite the acknowledged difficulties, from the late 80's and through the 90's a number of European projects sought to resolve the problems. Space prevents us from listing all the projects which made some contribution to the problem, but amongst those making a direct attack, many people will have heard of ACQUILEX (Calzolari & Briscoe 95) and MULTILEX (Khatchadourian et al. 94) within the Esprit programme, GENELEX (Antoni-Ley et al. 94) within the Eureka programme and MULTEXT within the LRE programme. 
However, if we look more closely at these projects, we find that they are mainly concerned either with feasibility studies or with direct technical attacks on some specific aspect of the problem. They are not conceived as sharing a common platform or as contributing to the creation of resources for the common good. (This should not be interpreted as a reproach, at least not of the projects themselves: neither aim was included in their goals). Nonetheless, they served to create awareness of a set of needs: the list given below is based on Calzolari (98):

•	actual development of language resources was (over)due - even more so in comparison with what was happening in the US (see, for example, the Linguistic Data Coding (LDC) initiative and the COMLEX lexicon (Grishman et al 93).
	
•	projects were still creating fragments of linguistic resources rather than reusing what had already been done
	
•	any consistent and public funded development effort ought not to privilege only a handful of European Union languages, but should treat all equally
	
•	a common platform of real size linguistic resources was required to ensure the development of real-world systems, coping with actual texts and not fictitious toy samples
	
•	the cost of developing large size linguistic resources was prohibitive for small and medium sized companies, who nonetheless desperately needed them
	
A consequence of this realisation was the putting in place of a strategic plan with the aims of 

•	defining technical standards and recommendations for best practice
	
•	creation of linguistic resources for all European languages
	
•	distribution of linguistic resources.
 
This strategic plan concerns corpora and evaluation as well as dictionaries; we shall return to it in those contexts as well. For the moment we limit ourselves to concrete realisation of the plan in the area of dictionaries.
The EAGLES (Expert Advisory Group on Language Engineering Standards) initiative was launched in February 1993, within the European Commission's Language Research and Engineering Programme (LRE). The first phase lasted until 1995, and the second phase is now drawing to its close.
In the first phase, an EAGLES group working on the lexicon made proposals for the encoding of lexical entries with a common core of morpho-syntactic distinctions in a multi-layered structure, with applications for all the European Union countries. Similarly, a proposal for a preliminary classification scheme and a set of standardised basic notions for encoding subcategorization information, using a frame-based structure, was drawn up, based on a comparison of how different systems and theories in different European languages classify and deal with subcategorization. (Subcategorization is a way of describing the syntactic combinatorial properties of linguistic items: we might, for example, say that 'dies' in the sense of 'ceasing to be alive' does not subcategorize for a direct object - one cannot say 'Brutus dies Caesar' -  whilst 'make' does subcategorize for a direct object - one cannot say 'Bill Gates makes', and stop there.)
The current round of EAGLES work on the lexicon is working towards a proposal for a basic set of notions in the area of lexical semantics. This is seen as an extension of the work on subcategorization. The proposed extension (see Sanfilippo et al., 1998) addresses the annotation of:

•	semantic class of subcategorizing expressions, e.g. verbs.
	
•	semantic class of subcategorized terms, e.g. nouns.
	
•	integration of semantic and syntactic information into subcategorization structures.

A classic example of the type of information it is hoped to deal with is offered by the 'wrongness' of the sentence 'Piety plays football': only concrete animates can play football.
The work has a special focus on requirements for Information Systems and Machine Translation, which, because of their major role in the computer industry, are thought to provide an excellent basis for focusing standardisation work in the human language technology area.
The EAGLES proposals for the lexicon have been applied in a number of different projects and over an impressive number of languages, thus contributing to the definition of a de facto standard, which is the first step towards a resource which can be commonly shared. Amongst these are the PAROLE project in the Language Engineering Programme, which has produced lexicons of 20'000 lemmas for all of the following languages: Catalan, Danish, Dutch, English, Finnish, French, German, Greek, Italian, Portuguese, Spanish and Swedish. Entries are described at morphological and syntactic (subcategorisation) levels, and in a few cases also at the semantic level. The lexical resources developed are declarative and theory and application independent. The model is designed to ensure that application dependent models of data and dictionaries for specific applications can be derived from the generic model.  All the lexicons are publicly available and are distributed through ELRA, the European Language resources Association.
A new project, SIMPLE, started only in May of this year, aims at adding a semantic layer to a subset of the existing morphological and syntactic layers. The semantic lexicons should cover about 10'000 word meanings for each of the twelve languages covered by PAROLE. The main types of information to be encoded for nouns, verbs and adjectives are: domain information, the semantic type of the head and the semantic type of the arguments of predicates. Currently, the formal representation of the 'conceptual core' of the lexicons is being designed, and the core structured set of meaning-types to be used as a common starting point and a shared device to build the harmonised language specific lexicons is being built.
Before turning to other types of linguistic resources, it is worth spending a little time on one issue which is particularly pertinent to dictionaries. In all that precedes, we have been talking about reusability: avoiding duplication of work by making sure that what is created takes a form allowing it to be re-used. One obvious source of material to be re-used would be existing dictionaries, printed or, these days, on CD-ROM. 
Of course, several of the issues already raised apply equally well to printed dictionaries: each dictionary has its own coding system, for example. Nonetheless, many attempts have been made to extract information from conventional dictionaries. Ide and Véronis (1995) discuss these efforts, and come to the conclusion that most existing dictionaries are inconsistent in the way they are coded, some of the information they contain is inappropriate, other types of information are missing. In addition, conventional dictionaries are highly structured objects: creating the software tools needed to carry out automatic extraction is a far from simple task.
In addition to all this, if information automatically extracted from a conventional dictionary were to be included in a commercial system, quite tricky copyright questions would also have to be solved. Up until now, and quite understandably, the publishing houses have been very reluctant to relinquish control of data which has taken years to amass. 
One recently started project, DicoPro in the Multilingual Information Society programme will attack some of these issues from another perspective. The project aims to make a number of commercial dictionaries available over the web through a uniform interface. 
This brings us to another issue which is more acute with dictionaries than with, say, corpora. Once a dictionary has been created, it has to be maintained. Language changes all the time, and with increased global communication, it changes more and more quickly. The publishing houses solve this problem by putting out new editions. If an electronic dictionary has been created, who is responsible for its maintenance and updating? The European Commission has perhaps partly solved this problem by helping to create the European Language Resources Association (ELRA), as a way of cataloguing and distributing language resources. It may be that the Association could also takes on the maintenance role for any resource it distributes.

3. Corpora.
Whilst it is easy to see the importance of electronic dictionaries, the utility of large electronic corpora is perhaps less intuitively obvious. But in fact electronic corpora furnish the raw materials for a host of applications.
Many of these rely on statistical methods, which can be used to determine relatively superficial phenomena over a large quantity of data. Part of speech tagging is one of the best established techniques in this area: given a sequence of words as input, taggers assign a sequence of part of speech tags with a very high success rate. The programs consist of a lexical component which assigns a set of potential tags to each word, and a component which chooses amongst competing tags for any given word in the sequence on the basis of probabilities of sequences of tags, calculated by previously training the tagger. For example,  in the sequence 'He coaches football' 'coaches' is ambiguous (as far as a computer programme is concerned) between the verbal sense intended here and the noun sense, as in 'We'll be travelling in coaches'. The noun sense is eliminated by a tagger on the grounds that the sequence Pronoun + Noun + Noun has a very low probability in English.
Tagging serves as the basis for a very wide range of subsequent tasks, for example for parsing, as a basis for identifying phrasal expressions for terminology extraction, for use with spelling or grammar checkers, and in applications such as speech recognition, information retrieval and computational lexicography.
Concordancing programs allow large corpora to be searched for every occurrence of a word or a phrase in its context of use. They have long been used in the humanities, and their utility for practical tasks such as lexicon building or translation is clear.
Translators are also increasingly making use of computerised translation aids, which rely essentially on the existence of multilingual corpora, parallel texts which exist in two or more language versions. For parallel texts to be exploited, they must first be aligned. That is, a correspondence must be established between the linguistic units of the text in one language and the linguistic units of the 'same' text in a different language. This is not so simple as it sounds. Quite often, a sentence in one text will correspond to two or more sentences in the parallel text, or the order of linguistic elements may differ from one language to another. Quite reliable alignment algorithms have now been developed, and are used with notable success in translation memory systems. The system compares an incoming text with previously translated  texts stored in its memory, and retrieves the previous translation for any fragment it recognises. 
Multilingual corpora have also been used for machine translation in the sense of fully automatic translation. Example based translation, first advocated by Nagao in (Nagao 84) relies on a data base of structured bilingual texts which are automatically matched according to lexical and structural regularities and various measures based on, for example, the use of a thesaurus. (Brown et al 88) report on a statistically based attempt to do fully automatic machine translation.
The examples given here are just the tip of the iceberg: many more and much more discussion can be found in (.Armstrong 94).
Talking of the importance of corpora is somewhat reminiscent of the empirical and statistical methods which were popular in both theoretical linguistics and in early work on machine translation in the 50's. But at that time, there were major obstacles to both the creation and the exploitation of electronic corpora. First, documents did not then exist in electronic form. Creating a corpus meant painstakingly converting a paper version to an electronic version using data entry methods which were far from user friendly. Secondly, computer capacity was minuscule compared to today, and few could afford the memory needed to store large amounts of data. In consequence, older corpora for English,  such as the Brown corpus (Francis & Kucera 82), tend to be quite small by current standards. Other corpora, especially for languages other than English, are held in the national language centres and are not easily available to external research workers or developers. 
A major factor in making corpus collection feasible is the advent of text processors and of document preparation software. It is therefore perhaps not surprising that the first collaborative efforts coincide with the era when document processing was first becoming widespread. In 1989 the ACL established a Data Collection Initiative (DCI). The acquisition work was carried out on a volunteer basis and in a rather opportunistic fashion, depending on availability rather than concerns of balance or the desire to be representative of some specific type of text. Text preparation was done by a few individuals (Liberman 89). In 1991, the ACL published its first CD-ROM, containing over 600 Kb of mostly American English data including a large collection of newspaper articles from the Wall Street Journal, a dictionary of English donated by Collins Publishers and some grammatically annotated data from the Penn Treebank Project. (Marcus et al 93).
Two issues about data acquisition and collection are buried in the last paragraph. The first concerns the physical form of the text. Even when textual data is available in electronic form, it may not be directly useable as part of a corpus. Older texts, which were prepared for printing and with no other purpose in mind are usually stored on tapes in a complex and undocumented form. It is often every hard to establish what the correspondence is between the logical structure of the text and the typographical structure. Even more recent texts may contain system specific formatting codes, and clearly one and the same corpus should not contain texts with different and incompatible formatting codes. The solution adopted by the DCI was to clean up the texts, removing any system specific coding, and mark up the text using SGML, a language specifically designed to be a 'neutral' coding of text structure and formatting information. This problem is becoming less severe, with the spread of mark-up languages, but still needs to be taken into account.
The other problem concerns a theoretical issue, and will not go away. Any corpus, no matter how it is collected, is representative of something. For example, one of the earliest bilingual corpora was the Canadian Hansard, the daily record of debates in the Canadian Parliament. It is obviously representative of the language used by Canadian Parliamentarians, and not, say, of the language of schoolchildren, newspapers or scientific articles. One of the problems associated with creating a corpus that is intended to be useful to a wide community of researchers and developers is to decide what to put in it.
The European Corpus Initiative, established in 1991 (Thompson 92) aimed at acquiring a large multilingual corpus for research work in Europe. An additional goal was to acquire a set of parallel texts (texts and their translations), given the importance in the European setting of multilingual document production and the growing interest in translation studies. A large amount of data (at least 5 million words of text for each of the major languages) was collected. A variety of parallel corpora came from international organisations ( English, French and Spanish) and from the Swiss banks (English, French and German). The data is available on CD-ROM:
The early volunteer projects were followed by more official efforts. The Linguistic Data Consortium was set up in the US in 1992 with an initial start-up grant from the Advanced Research Projects Agency (ARPA). In its first year the LDC produced nearly 100 CD-ROMs. In Europe, an initial feasibility study was carried out in a project called the Network for European Corpora, which was followed up by the RELATOR project and by PAROLE, the current project already mentioned in connection with dictionaries.
The central goal of PAROLE is to produce an initial set of harmonised corpora and lexicons for all of the European Union languages. For each of the following languages a corpus of at least 20 million words and a lexicon of 20'000 lemmas has been produced: Catalan, Danish, Dutch, English, Finnish, French, German, Greek, Italian, Portuguese, Spanish (lexicon only) and Swedish. Smaller corpora have been created for Belgian French (20 million words), Irish (15 million words) and Norwegian (3 million words).
As effort to build corpora has intensified, issues of standardisation have come to play a central role. An early effort in this direction was the Text Encoding Initiative, which produced guidelines for coding material for all types of text mark-up. 
An EAGLES group working on corpora has subsequently produced a Corpus Encoding Standard, based on the TEI standard. This standard was used for the coding of the corpora produced by the PAROLE project. The exchange format is SGML. Each corpus will be accessible for consultation, possibly via the Internet. A subset of 3 million words of each corpus will also be distributable, i.e. a physical copy of it can be given to users. The European Language Resources Association (ELRA), will take care of the distribution. There may be some restrictions on what users may have access to what as a result of restrictions imposed by the holders of copyright of the source texts when they agreed that the texts should be included in the corpus.
So far, apart from one brief mention of the Penn Tree Bank Project,  we have talked as though a corpus was nothing more than a large collection of strings of words. In fact, the value of a corpus can be increased by adding information to it of various types. The EAGLES Corpora Group has been working towards the definition of a framework for morpho-syntactic annotation of corpora that will ensure the compatibility and interchangeability of concrete annotation schemes based on it. 20'000 running words of each of the PAROLE corpora are tagged (and checked) at the morpho-syntactic level, following the EAGLES guidelines. These tagged sections are amongst the distributable part of the corpus. Preliminary general recommendations for syntactic annotation of corpora are also available, focusing on surface syntactic bracketing. 
In the current phase of EAGLES work, attention is focused on the annotation of dialogue, thus working towards an integration of different perspectives of written and spoken language annotation.
Let us close this section by emphasising once again that changes in computer power and storage coupled with the ever-increasing availability of texts in electronic form provide radically new possibilities, both for research and for development. Having corpora available allows us to study 'real' language and encourages us to develop robust tools to deal with it. As public access to computer based systems becomes wider and wider, robustness grows in importance. At the moment we are just creating the first large corpora for a variety of languages and learning to exploit them. I believe that as our experience grows so will our imagination and our creativity, and that we shall discover ways of using these new resources in applications yet to be thought of.

4. Evaluation.
The essential issue in providing and maintaining well designed large scale language resources is one of infrastructure. Their existence is part of the infrastructure which makes research and development in language engineering possible. Evaluation too plays a critical role in this perspective.
Evaluation is important at every stage of a product's life cycle. Right at the beginning, when the research  which will eventually enable the development of a product is being planned, there will probably be a research proposal, which will be evaluated by whoever is expected to pay for the research. When the research project is completed, somebody will evaluate its results. A developer will carry out an evaluation of whether the results obtained can in fact usefully be integrated into a commercial product. During development of the product, its commercial potential will be evaluated. Finally, the purchaser will evaluate the product, before he buys it if the purchase is expensive, and inevitably after he has bought it.
Although there had been a number of important evaluations carried out, mainly of machine translation systems, evaluation has only emerged as an important topic in its own right over the last few years. This was primarily a consequence of the ARPA/DARPA series of evaluation guided research programmes started in the late 1980s and continued into the 1990s. These programmes brought together teams working in a particular area and defined a common task which all teams should tackle. The definition of the task was accompanied by the definition of metrics according to which success in fulfilling the task would be judged. The programmes covered a variety of different technical areas, including speech recognition, document retrieval, fact extraction and machine translation.
Since the main purpose of the exercise was to advance the core technology of the area in question, the main emphasis was on the functionality of the system being evaluated. Other aspects, such as user acceptability or even the context in which the system was meant to be used were deliberately neglected and were intended to play no part in the evaluation. 
Interestingly, although several of these programmes were very successful, leading to a set of commonly accepted metrics and to genuinely friendly sharing of data and even software (Sparck Jones 98), others found a great deal of difficulty in defining acceptable metrics. (White et al 94 and King 96, 97) discuss some of the reasons why this should be so, In general, the successful programmes continue, and have succeeded in the goal of advancing the technology, while those where no consensus could be found have been dropped.
One point to notice is that although, of course, the evaluation designers of each programme talked to one another, evaluation design was seen as being specific to each programme: there was no general framework which could serve as guidelines for all evaluation designers.
In Europe, it was once again the EAGLES initiative which played a major role in emphasising the importance of evaluation. But there, the perspective was rather different. First, the EAGLES initiative is primarily concerned with standards. Evaluation was seen as a way of defining de facto standards which products should meet. Secondly, the European Commission has a strong interest in promoting European industry. Evaluation was therefore seen as a way of helping the developer to ensure that his product would meet market needs and find market acceptability whilst at the same time providing the consumer with a way of deciding what product was best for him. The direct consequence of putting together these points of view was to define the task of the Evaluation Group as one of developing a general methodology for the design of evaluations which would be applicable to all evaluations in the field of language engineering, whether they be of systems, of products or even projects. In other words, the group worked at a greater level of abstraction than their American colleagues. In order to prevent abstraction becoming too divorced from practical reality, certain types of products - spelling checkers, grammar checkers and translation memory systems - were chosen as test cases: the general methodology would be validated by applying it to the concrete evaluation of these products. Much of this validation work was done in the framework of a Language Research and Engineering project, TEMAA.
There is an obvious tension between designing a general methodology and trying to ensure that users are taken as central to the evaluation task; after all, every user and every context of use is different. This tension was resolved by working in terms of classes of typical users, much in the same way that consumer magazines will relate products to typical groups of consumers. A car, for example, may be assessed for its suitability for a large family, or for a user heavily concerned with economy, or for a user who frequently travels long distances in it.
In line with the EAGLES initiative's emphasis on standards, a natural place for the Evaluation Group to start was by considering whether any other relevant standards already existed. ISO standard 9126 concerns the evaluation of software product. It provides a standard for the definition of quality and guidelines for an evaluation procedure. The first phase of EAGLES work used a 1991 version of this standard as a basic text. In the second round, we have been fortunate enough to have had access to a new draft of the standard and to have been able to discuss it with its technical editor, Nigel Bevan. Where relevant, the main differences between the two drafts will be mentioned in what follows.
An early EAGLES decision was to distinguish between at least two basic types of evaluation: 

•	adequacy evaluation, where the performance of a product is seen from the point of view of the user or purchaser and 
	
•	progress evaluation, where development progress is measured against plans, specifications and previous versions of the same system. 

The ISO standard is mostly concerned with development of software, and therefore with progress evaluation, although a claim is made that the standard should be more widely applicable. In contradistinction, it was decided within EAGLES to focus on adequacy evaluation, thereby testing the applicability of the ISO standard to a different kind of evaluation and simultaneously complementing the ARPA/DARPA emphasis on testing functionality divorced from context of use.
Quality, in the ISO 9126 standard, is defined in terms of quality characteristics, of which there are six. Each characteristic is a generic label, which is made concrete and specialised by breaking it down into a set of sub-characteristics, each of which in its turn can be further broken down. At the leaves of the hierarchical structure thus obtained are metrics: recipes for obtaining a value for that sub-characteristic when performing a specific evaluation. The values obtained can be combined to find  values for nodes at higher levels of the structure. 
The logical structure inherent in the ISO 9126 standard was translated in EAGLES into a feature structure, of the type familiar to computational linguists. The TEMAA project's application of the EAGLES methodology to the evaluation of spelling checkers will provide a concrete example of a fragment of one such feature structure. 
In the case of spelling checkers, the top level quality characteristic functionality is assumed to have three sub-characteristics:

•	recall: the degree to which the checker accepts all the valid words of the language
	
•	precision: the degree to which the checker rejects all invalid words
	
•	suggestion adequacy: in case of invalid words, does the checker provide correct suggestions?
	
	Space will not allow us to look at how all three are further broken down. As a typical example, precision was decomposed into:
	
•	typographical errors
•	spelling errors
•	medium related errors (e.g. dictation software errors, OCR errors)
	
	each of which was in turn broken down again. For example, typographical errors was broken down into
	
•	insertion
•	deletion
•	substitution
•	transposition
	
	again broken down into, for example,
	
•	doubling of a letter
•	deletion of a letter
•	interchanging of two letters

It can readily be seen that defining the feature structure relevant to a given type of system is a long and fastidious task requiring a great deal of thought. It is also easy to see that doing the definition clarifies the mind about what should be measured and how, thus helping to make the overall evaluation better founded, and that once a feature structure has been defined it can serve as a model for any other system that falls into the same general class.
Doing the definition also helps to suggest metrics. For example, in the case above,  a program was created which allowed illegal 'words' to be generated from legal words, according to rules defined by the user. A session with the spelling checker was then automatically simulated during which the checker checked the generated 'mistakes'. Since it was known what the correct form was intended to be, it was then possible to give a score to  the spelling checker based on whether the mistake was identified as a mistake, whether the correct suggestion was made, whether the correct suggestion was first in the list of suggestions made and so on.
Users, in the EAGLES paradigm, are also described via feature structures. Specific values (or more often limiting values) reflect the user's particularities, which are further reflected in the ways in which values lower down in the structure are combined to give values for nodes at higher levels. For example, if some user is totally uninterested in medium related errors, the contribution of that sub-characteristic to the precision sub-characteristic which is its dominating node is set at zero.
The fact that both users and systems are formally represented using the same descriptive mechanism allows automatic comparison of users and systems, and thus identification of what systems will meet a given user's needs.
The feature structure created for a class of products can be seen as a model of quality for that class of systems. Formalised descriptions have the obvious advantage of being more easily standardised., thus making it easier to check conformity. They also, as the concrete example above shows, make it easier to automate at least some of the measuring and reporting process. 
Thinking of a feature structure as a model of quality brings us to another important point. Quality is not a single, unchanging characteristic of software: it means different things to different people at different times of the product's life cycle. The EAGLES work was concerned with evaluation of whether a product would meet the needs of a particular class of users, and the formal description therefore took into account the purpose of the evaluation. If the purpose of the evaluation was to determine whether a product satisfied a set of requirements specifications, the quality model would be different and different attributes would be chosen to play a role in the formalisation. However, the basic modelling tool would remain unchanged.
The first round of EAGLES work also placed a lot of emphasis on establishing the validity of metrics. Based on work in the social sciences, two types of validity were distinguished. Internal validity is inherent to the metric itself. A classic example is that of reading tests based on the use of specifically chosen vocabulary. The validity of the test relies only on whether the vocabulary has been correctly chosen. External validity relies on correlation with some external criterion. A classic example here is the set of criteria used by insurance companies to determine the size of life insurance premiums. Factors such as the existence of hereditary disease in the family, whether the applicant has already undergone major surgery and so on are held to have a strong correlation with life expectancy.  The importance of choosing and validating metrics is also reflected in the new ISO 9126 draft, which is accompanied by separate documents on metrics.
The new draft also introduces a new notion, that of 'quality in use'. This is quality as perceived by the user, and is a combination of a number of quality characteristics, including functionality, usability and reliability. 
The current round of EAGLES Evaluation work is mainly one of consolidation and dissemination. A workshop was help in Brussels in November of 1997, where presentations were made of the new ISO draft, of related work in speech and questions of experimental design and of the positive experiences gained from one of the ARPA evaluation programmes. Major topics of discussion were quality in use and the definition of metrics. 
A web site has been set up which will, we hope, become in time a common, publicly accessible repository of information about evaluation, about resources for evaluation and about evaluation results. We have recently set up a discussion list for discussion of matters relating to evaluation. The web site address is http://www.cst.ku.dk/projects/eagles2.html. 
Any one wanting to join the discussion list should send an e-mail to eagcst.ku.dk. The web address for EAGLES in general is: http://www.ilc.pi.cnr.it/EAGLES/home/html
One of the major aims of the current round of EAGLES work is to produce a 'Handbook of Current Evaluation Practices'. Much of the material has already been gathered, and first drafts of the some of the chapters will be presented in a second workshop to be held in Geneva on September 8/9. It is expected that a final version of the Handbook will be available by the end of the year.

5. Conclusion.
A theme underlying all three of the topics covered in this paper is the importance of infrastructure. The sort of resources we have been talking about cannot be developed or distributed without an appropriate infrastructure being put into place, and yet the resources themselves are part of the infrastructure required for long term development and research in the human language technology area. The need for infrastructure implies a need for long term policy and long term commitment, especially on the part of the funding agencies. In the absence of such a policy, the resources which have been developed will not be maintained, made available or used, and no new resources reflecting new needs and changes in language will be developed. So far, both the North American and the European agencies have quite a good track record in this respect, and the research and development community must be grateful to them. Let us hope this happy situation will continue.

References.

(Alonge 98) A. Alonge.  Data on Verb Semantics in the EuroWordNet Database. Proceedings of the 1st LREC Conference, ELRA, 1998..
(Antoni-Ley et al 94) M.H. Antoni-Ley, G. Francopoulo, L. Zaysser. A Generic Model for Reusable Lexicons: The Genelex Project. Literary and Linguistic Computing 9(1), 1994.
(Armstrong 94) S. Armstrong Warwick. Acquisition and Exploitation of Textual Resources for NLP. In Zampolli et al, op. cit., 1994.
(Brown et al 88) P. Brown, J. Cocke, S. Della Pietra, F. Jelinek, R Mercer, P. Roossin. A Statistical Approach to Machine Translation. Coling 1988.
(Calzolari 98) N. Calzolari, An Overview of Written Language Resources in Europe: a few Reflections, Facts and a Vision. Proceedings of the 1st LREC, 1998.
(Calzolari & Briscoe 95) ACQUILEX-I and -II: Acquisition of Lexical Knowledge from Machine Readable Dictionaries and Text Corpora. Cahiers de Lexicologie LXVII (2), 1995.
(Francis & Kucera 82) W. Francis, H. Kucera. Frequency Analysis of English Usage. Houghton Mifflin, Boston, Massachusetts, 1982.
(Gibbon 96) D. Gibbon, (ed.), Natural Language Processing and Speech Technology. Results of the 3rd. Konvens Conference, Bielefeld, October 1996. Mouton de Gruyter, Berlin and New York, 1996.
(Ide & Véronis 95) N. Ide, J. Véronis, Knowledge Extraction from Machine Readable Dictionaries. In P. Steffens (ed.) Machine Translation and the Lexicon, Springer, 1995.
(Khatchadourian et al 94) H. Khatchadourian, N. Modiano, G. Heyer, K. Waldhor, Use and Importance of Standard in Electronic Dictionaries: the Compilation Approach for Lexical Resources. Literary and Linguistic Computing 9(1), 1994.
(King 96) M. King. Evaluating Natural Language Processing Systems. In Y. Wilks (ed.) Special Edition of the CACM on Natural Language Processing. vol. 39, 1., 1996.
(King 97) Evaluating Translation. In C. Hauenschild and S. Heizman (eds,), Machine Translation and Translation Theory. Mouton de Gruyter, 1997.
(King & Maegaard 98) M. King, B. Maegaard. Issues in Natural Language Systems Evaluation. Proceedings of the 1st LREC Conference, ELRA, 1998
(Liberman 89) M. Liberman. Text on Tap: the ACL/DCI. Proceedings of the 1989 DARPA Speech and Natural Language Workshop. Cape Cod, Massachusetts, 1989.
(Marcus et al 93) M. Marcus, B. Santorini, M. Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2), 1993.
(Nagao 94) M. Nagao. A framework of a mechanical translation between Japanese and English by analogy principle. In A. Elithorn and R. Banerji (eds) Artificial and Human Intelligence. North Holland, 1994.
(Roventini et al 98) A. Roventini, C. Peters, N. Calzolari, F. Bertagna. Building a Semantic Network for Italian using Existing Lexical Resources. Proceedings of the 1st LREC Conference, ELRA, 1998..
(Rubio et al 98) A. Rubio, N. Gallardo, A. Castro, A. Tejada,  Proceedings of the First International Conference on Language Resources and Evaluation, Granada, Spain. ELRA, 1998.
(Sanfilippo et al 98) EAGLES Preliminary Recommendations on Semantic Encoding. Preliminary Report. EAGLES, 1998.
(Sparck Jones 98) Report on Coherent Approaches to Evaluation, a presentation made at the EAGLES Workshop, Brussels, 1997.  Available at http://www.cst.ku.dk/projects/eagles2/html
(Thompson 92). H. Thompson European Corpus Initiative. ELSNEWS, 1(1), 1992.
(Vossen & Bloksma 98). Categories and Classifications in EuroWordNet. Proceedings of the 1st LREC Conference, ELRA, 1998..
(Walker et al 95) D. Walker, A. Zampolli, N. Calzolari, (eds.),  Automating the Lexicon: Research and Practice in a Multilingual Environment. Proceedings of a workshop held in Grosseto. Oxford, Oxford University Press, 1995.
(White et al 94) J.S. White, T. O'Connell, F.E.O'Mara. The ARPA MT Evaluation Methodologies: Evolution, Lessons and Further Approaches. Proceedings of the First Conference of the Association for Machine Translation in the Americas, Columbia, Maryland, 1994.
(Zampolli 94) A. Zampolli. Preface to Zampolli, Calzolari and Palmer, op.cit.
(Zampolli et al 94) A. Zampolli, N. Calzolari, M. Palmer, M. (eds.), Current Issues in Computational Linguistics: in Honour of Don Walker. Pisa, Giardini and Kluwer, 1994.