[Date Prev][Date Next][Thread Prev][Thread Next][Chronological Index][Thread Index]

EAGLESEVAL: Outstanding Questions

Dear Colleagues,

    As promised in my previous mail, to get the discussion going, here
are some questions which were left open at the end of the LREC
workshop, supplied by Steven Krauwer.  More outstanding questions will
be mailed to the list over the next couple of days. Wherever possible, the
person who asked the question is identified. Please check that you are
not misrepresented.  
    At the end of this message I also append a few notes on how to interact
with the list.


Patrick Paroubek (LIMSI):
To what extent can the performance loss reported by Spoken
Language Dialogue Systems developers when shifting from lab
conditions to real world conditions be attributed to insufficient
definition of measure taking conditions? (Could compare gold
standard definition for text). Ergonomists involved?

Lin Chase (LIMSI):
(1) Has there been discussion of confidence measures for human
    annotated data?

(2) By recording only categorial decisions (what class is this
    word/phrase/document) we lose important information that we
    might retain by allowing the human annotator to say if she/he
    was uncertain about that judgment.

(3) The Americans all seem to assume that "the evaluator" of the
    technology (or "the organizer") is also the funding agency.
    This alignment of performance demand with financial incentive
    seems to be a major characteristic of the US programs. Are we
    in Europe going to be successful if we don't choose a similar

(4) Raj Reddy, who runs the speech recognition research group at
    Carnegie Mellon University, always says that the only real
    test of whether you've succeeded in developing a system is
    that if you give it to users, when you come to take it away
    they fight you for it. What can we do (say with spoken
    language information systems) to use this kind of
    "evaluation" more?

Marc Blasband (NS):
(1) Question to Joseph Mariani:
    Effort to involve end-users in his programme?

(2) Question to ??:
    European efforts in German, Spanish, Italian? In other languages?


Just a few words on how this list functions (apologies if I am stating the
obvious here!):

1. This list is not moderated

2. To send a mail to the discussion list as a whole:
  - email EAGLESeval@cst.dk. 
  - I suggest that we follow Jean Veronis' excellent example and prefix
     the subject field with "EAGLESEVAL:"

2. To reply to a specific email on the list

  - use your mailer's "reply" function to reply privately to the original
    sender of the email

  - to reply to the list as a whole you must ensure that you also copy
    your reply to EAGLESeval@cst.dk.

In a couple of weeks the list will also be accessible via the EAGLES II
website: http://www.cst.dk/eagles/index.html.

		 send an email to eag@cst.dk 

Nancy Underwood
Center for Sprogteknologi