Dias 1		
	
Dias 2		The chapter on test materials will be a somewhat updated version of the one in the EAGLES-I report and we expect it to have the same structure as in the outline of this talk.

It is also clear that this topic is closely related to the question of sharing resources and data which will be discussed tomorrow.
	
Dias 3		Since the EAGLES I report there seems to have been an explosion of interest in the creation and dissemination of language corpora.  
The chapter will to some extent be updated to reflect this surge of activity.  But most of the information will most probably be treated under the heading “Sharing Resources” which will also provide links and references to relevant sites.
This slide is not meant to be exhaustive - but rather indicative of the range of resources potentially available.  So that there are some rather more specific corpora available
	
Dias 4		The most serious problem with “general” and generally available test sets is  representativeness.  They have not usually been assembled with a specific set of user needs in mind.
While every test set is representative of something this might not be the relevant text type for the evaluation. So care must be taken in choosing to re-use such sets.  
On the other hand - with the advent of more restricted  corpora (e.g. from one newspaper) the evaluator may be lucky enough to find the very corpus they are looking for. or put together a number of test sets
It is clear that evaluators may also assemble their own customer/evaluation specific test sets.  It could be interesting to discuss how folk might go about sharing such specific test sets.
	
Dias 5		Test suites are used to test specific system behaviours wrt to certain linguistic phenomenon.
A major problem is the complexity of their construction.  The evaluator has to decide what to test to make sure that they can really test the behaviour they want to test. 
Such test suites very quickly grow very large especially when, instead of treating one phenomenon per input the interaction of different phenomena is to be tested 
As they grow large there is also a problem with both maintaining the test suite and analysing results.
Test suites, being defined to test specific systems may be inadvertantly tuned towards that system and so not completely re-usable for other evaluations.
Help is at hand from TSNLP and now its follow-up project DiET which provide guidelines for construction of test suites
	
Dias 6		Test collections are defined as sets of inputs and their related expected outputs.
In the EAGLES-I report examples of test collections arising from the competitive MUC and TREC evaluations are given.
In MUC thegeneral task is to fill out a template with information extracted from documents. The expected output is defined in terms of  correctly filled out templates for the documents.
In TREC the input is a set of documents plus so-called topics which the documents must deal with. For each document a judgement as to its relevance wrt to the topic is given i.e. yes/no.
	
Dias 7		Design and construction is extremely costly because so much is to be done by hand in the construction and definition of judgement. so it important to be able to share and re-use resources

 Can such large scale collections be achieved outside an evaluation driven research paradigm? i.e. can it only be competitive/comparative evaluation
 SENSEVAL apparantly bases itself on the ARPA/DARPA model The question of proofing and translation tools is maybe more a question of how wewant to define the notion of “test collections”

Dias 8