[Date Prev][Date Next][Thread Prev][Thread Next][Chronological Index][Thread Index]

Re: EAGLESEVAL: A question from George Doddington.

At 5:09 PM +0200 8/25/98, Maghi King wrote:

>Eduard Hovy wrote:

>> At 4:33 PM +0200 8/11/98, Maghi King wrote:
>> >A question from George Doddington:
>> >Several presenters have argued for moving evaluation from the technical
>> >(lower) level toward the application (user) level. If evaluation is
>> >viewed as a means of making progress, this implies that these presenters
>> >believe that it is more important (or easier?) to make progress at the
>> >application level than on the underlying problems. Is this a fair
>> >representation of the presenters' perspective?
>> This is, of course, a loaded question.  A counter is: Does Mr. Doddington
>> believe it is better to make progress on the underlying problems even if
>> they have no effect at the application level?
>And now I can't resist: what would it be like to make progress on an
>underlying level and NOT have an effect on the application level? 
>I can imagine going in for something that is really overkill, like a
>full blown understanding system to deal with simple queries that could
>be dealt with just as well by very straight forward methods, but surely
>you'd still have an effect on the application level, if it was only the
>negative effect of slowing evrything down unnecessarily?

I can think of several places in which people make progress on underlying 
levels that have no real effect on the application level.  For example, 
many groups have insisted that detailed syntactic parsing is the key, 
without demonstrating significant increases in output quality (or a 
significant decrease in speed).  Ditto for the heavily semanticized 
Interlingua-based approaches, who can spend years building ontologies 
and fancy lexicons but still show no improvement on general text in the 
large.  (I happen myself to believe that semantics is the right direction 
to go, but I do not believe that we will get adequately large semantic 
knowledge bases and/or lexicons using current manual techniques.) 


Eduard Hovy
email: hovy@isi.edu          USC Information Sciences Institute 
tel: 310-822-1511 ext 731    4676 Admiralty Way 
fax: 310-823-6714            Marina del Rey, CA 90292-6695 
project homepage: http://www.isi.edu/natural-language/nlp-at-isi.html