This page contains an annotated draft of the table of contents for the new EAGLES handbook which will be discussed at the workshop in Geneva. Sections 1 and 2 are largely based on updating the EAGLES-I final report
Introduction
(How the document is structured, how to use it, relation
to EAGLES I
report, how it came about...)
1. Summary of the EAGLES Framework
(relatively short describing the overall framework in terms
of
the main actions/steps you need to evaluate NLP products in the new
ISO terms with references to the relevant subsections in Section 2
where they are discussed in much more detail.)
2. Detailed Description of the EAGLES Framework
2.1 EAGLES and the new ISO standard
(will replace the old ISO standard used in EAGLES-I)
2.2 Formalisation and Automation
2.3 User profiling/requirements analysis
(will re-use most of the EAGLES-I work but also be extended to include
new material on usability -see "Quality in Use")
2.4 Metrics
(this covers both measures and methods fromEAGLES-I)
2.5 Tools
2.6 Test materials
2.7 Sharing resources
(This is not something that was looked at in EAGLES-I, but the LREC
conference (and Moncton) would be a good place to start.)
3. Case Studies
(These case studies would be examples of how the framework
has been
applied.)
3.1 Spelling Checkers
(to be based on TEMAA. This will be a description of the actual
evaluations which were carried out on Danish and Italian spelling
checkers)
3.2 Grammar Checkers
(be an amalgamation of some TEMAA/EAGLES work on defining
the grammar checker evaluation model)
3.3 Translation Memories
(This would involve bringing together material from various appendices
of EAGLES-I)
3.4 Other case studies
4 Current Practices Survey
For each approach surveyed:
- a descriptive section based on their materials
- an analytic section based on trying to recast the approach in EAGLES
terms
- a lessons to us all section, that tries to draw conclusions