Re: A possible metric for Machine Translation?
Hi Ed!
Need to think about some of what you say, but there's one possible
misunderstanding that I wanted to clear up right away.
So please see little comments inserted below.
Eduard Hovy wrote:
>
> Hi M,
>
> At 1:59 PM +0200 9/4/98, Maghi King wrote:
> >So I was wondering if there might not be some feature (quality
> >subcharacteristic in ISO jargon) of MT systems that was just a minimal
> >indicator of quality, was pertinent to most applications/uses of MT and
> >could be tested relatively easily.
> >
> >One that springs to mind is preservation of truth values.
>
> That is, for an input sentence meaning
> "This sentence is an incorrect translation"
> the following translation
> "Sentence this one incorrect conversion changing been"
> is better than
> "This sentence is a correct translation"
> because its truth value polarity is the same as the original?
This is the bit I want to think about; will come back to it later.
>
> There are probably a dozen plausible "minimal indicators" one could choose,
> each one with a strong case.
>
> I think translation evaluation is simply not reducible to a single metric.
>
And this is weher there might be a mis-understanding.
I totally agree with you, Ed, I don't think any kind of evaluation can
be reduced to a single metric. (Ther's going out on a different kind of
limb). I'm very convinced by the ISO picture of working out a
hierarchical structure of attributes, each of which will have one or
more metrics attached (some attributes at higher levels have metrics
which are combinations of values obtained on lower levels, but they
still have metrics).
So the truth preservation metric that I was musuing about would be
typically one among many: but one that might be useful for very many
evaluations, so worth spending time on to get it right.
I agree too that there may well be other "minimal indicators": wouldn't
it be nice to know what they are and to be able just to pick them up and
plug them into the right place in the overall evaluation scheme?
Maghi
--
Please note my new e-mail address (old address was king@divsun.unige.ch)
Maghi King | E-mail: Margaret.King@issco.unige.ch
ISSCO, University of Geneva | WWW: http://issco-www.unige.ch/
54 route des Acacias | Tel: +41/22/705 71 14
CH-1227 GENEVA (Switzerland) | Fax: +41/22/300 10 86