EAGLESEVAL: Outstanding Questions
- To: EAGLESeval@pia.cst.dk
- Subject: EAGLESEVAL: Outstanding Questions
- From: Nancy Underwood
- Date: Fri, 17 Jul 1998 13:27:22 METDST
Dear Colleagues,
As promised in my previous mail, to get the discussion going, here
are some questions which were left open at the end of the LREC
workshop, supplied by Steven Krauwer. More outstanding questions will
be mailed to the list over the next couple of days. Wherever possible, the
person who asked the question is identified. Please check that you are
not misrepresented.
At the end of this message I also append a few notes on how to interact
with the list.
--------------------
QUESTIONS FROM THE FLOOR:
Patrick Paroubek (LIMSI):
-------------------------
To what extent can the performance loss reported by Spoken
Language Dialogue Systems developers when shifting from lab
conditions to real world conditions be attributed to insufficient
definition of measure taking conditions? (Could compare gold
standard definition for text). Ergonomists involved?
Lin Chase (LIMSI):
------------------
(1) Has there been discussion of confidence measures for human
annotated data?
(2) By recording only categorial decisions (what class is this
word/phrase/document) we lose important information that we
might retain by allowing the human annotator to say if she/he
was uncertain about that judgment.
(3) The Americans all seem to assume that "the evaluator" of the
technology (or "the organizer") is also the funding agency.
This alignment of performance demand with financial incentive
seems to be a major characteristic of the US programs. Are we
in Europe going to be successful if we don't choose a similar
approach?
(4) Raj Reddy, who runs the speech recognition research group at
Carnegie Mellon University, always says that the only real
test of whether you've succeeded in developing a system is
that if you give it to users, when you come to take it away
they fight you for it. What can we do (say with spoken
language information systems) to use this kind of
"evaluation" more?
Marc Blasband (NS):
-------------------
(1) Question to Joseph Mariani:
Effort to involve end-users in his programme?
(2) Question to ??:
European efforts in German, Spanish, Italian? In other languages?
----------------------------------------------------------------------
Finally:
Just a few words on how this list functions (apologies if I am stating the
obvious here!):
1. This list is not moderated
2. To send a mail to the discussion list as a whole:
- email EAGLESeval@cst.dk.
- I suggest that we follow Jean Veronis' excellent example and prefix
the subject field with "EAGLESEVAL:"
2. To reply to a specific email on the list
- use your mailer's "reply" function to reply privately to the original
sender of the email
- to reply to the list as a whole you must ensure that you also copy
your reply to EAGLESeval@cst.dk.
In a couple of weeks the list will also be accessible via the EAGLES II
website: http://www.cst.dk/eagles/index.html.
-----------------------------------------------------------------
IF YOU DO NOT WISH TO BE INCLUDED ON THIS LIST
send an email to eag@cst.dk
-----------------------------------------------------------------
--
Nancy Underwood
Center for Sprogteknologi