BILAG 4

-------------------------------------------------

TRÆNING AF BRILL-TAGGEREN

-------------------------------------------------

1) Del træningskorpus i 2 lige store dele:

cat "KORPUS" | perl RULE_BASED_TAGGER_V1.14/Utilities/

divide-in-two-rand.prl TAGGED-CORPUS TAGGED-CORPUS-2

(INPUT = "KORPUS" OUTPUT = TAGGED-CORPUS og TAGGED-CORPUS-2)

2) Fjern taggene fra korpus:

cat "KORPUS" | perl RULE_BASED_TAGGER_V1.14/Utilities/

tagged-to-untagged.prl > UNTAGGED-CORPUS

(INPUT = "KORPUS" OUTPUT = UNTAGGED-CORPUS)

3) Lav en liste over alle ord i det utaggede korpus:

cat UNTAGGED-CORPUS|perl RULE_BASED_TAGGER_V1.14/Utilities/

wordlist-make.prl |sort +1 -rn| awk "{print $1}" > BIGWORDLIST

(INPUT = UNTAGGED-CORPUS OUTPUT = BIGWORDLIST)

4) Lav en liste over alle ord m. tag fra 1. del af det taggede træningskorpus:

cat TAGGED-CORPUS|perl RULE_BASED_TAGGER_V1.14/Utilities/

word-tag-count.prl|sort +2 -rn > SMALLWORDTAGLIST

(INPUT = TAGGED-CORPUS OUTPUT = SMALLWORDTAGLIST)

5) Lav en liste over alle ordpar i det utaggede korpus:

cat UNTAGGED-CORPUS|perl RULE_BASED_TAGGER_V1.14/Utilities/

bigram-generate.prl|awk "{ print $1 $2 }" > BIGBIGRAMLIST

(INPUT = UNTAGGED-CORPUS OUTPUT = BIGBIGRAMLIST)

6) Nu kommer delen hvor de leksikalske regler skal læres:

perl RULE_BASED_TAGGER_V1.14/Learner_Code/unknown-lexical-learn.prl

BIGWORDLIST SMALLWORDTAGLIST BIGBIGRAMLIST 300 LEXRULEOUTFILE

(INPUT = BIGWORDLIST SMALLWORDTAGLIST BIGBIGRAMLIST

OUTPUT = LEXRULEOUTFILE)

OBS! husk at lave de initiale tags om fra NN/NNP til N/EGEN

i Laerner_Code/unknown-lexical-learn.prl

Laerner_Code/unknown-lexical-learn-continue.prl og

Tagger_Code/start-state-tagger.c

7) Kopier hele det taggede korpus over i en ny fil:

cp parole TAGGED-CORPUS-ENTIRE

(INPUT =parole OUTPUT = TAGGED-CORPUS-ENTIRE)

8) Lav et træningsleksikon:

cat TAGGED-CORPUS |perl RULE_BASED_TAGGER_V1.14/Utilities/

make-restricted-lexicon.prl > TRAINING.LEXICON

(INPUT = TAGGED-CORPUS OUTPUT = TRAINING.LEXICON )

9) Lav det leksikon som skal bruges til tagging når træningen

er afsluttet:

cat parole.ny.FSTL |perl RULE_BASED_TAGGER_V1.14/Utilities/

make-restricted-lexicon.prl > FINAL.LEXICON

(INPUT = TAGGED-CORPUS-ENTIRE OUTPUT = FINAL.LEXICON)

10) Lav et utagget træningskorpus:

cat TAGGED-CORPUS-2| perl RULE_BASED_TAGGER_V1.14/Utilities/

tagged-to-untagged.prl > UNTAGGED-CORPUS-2

(INPUT = TAGGED-CORPUS-2 OUTPUT = UNTAGGED-CORPUS-2 )

 

 

 

 

 

11) Her laves et "dummy-korpus" (korpus i initialt stadium??):

tagger ../../ trained_tagger/TRAINING.LEXICON ../../ trained_tagger/UNTAGGED-CORPUS-2

../../trained_tagger/BIGBIGRAMLIST ../../ trained_tagger/LEXRULEOUTFILE /dev/null -w ../../

trained_tagger/BIGWORDLIST -S > ../../ trained_tagger/DUMMY-TAGGED-CORPUS

(INPUT = TRAINING.LEXICON, UNTAGGED-CORPUS-2,

BIGBIGRAMLIST, LEXRULEOUTFILE

OUTPUT = DUMMY-TAGGED-CORPUS)

OBS! Vær opmærksom på at man SKAL stå nede i direktoriet hvor "tagger" ligger:

RULE_BASED_TAGGER_V1.14/Bin_and_Data

12) Nu skal de kontekstuelle regler laves:

RULE_BASED_TAGGER_V1.14/Bin_and_Data/contextual-rule-learn TAGGED-CORPUS-2

DUMMY-TAGGED-CORPUS CONTEXT-RULEFILE TRAINING.LEXICON

(INPUT = TAGGED-CORPUS-2, DUMMY-TAGGED-CORPUS, TRAINING.LEXICON

OUTPUT = CONTEXT-RULEFILE)

---------------------------------------------------------

TAGGING MED BRILL-TAGGEREN

---------------------------------------------------------

Stå i direktoriet RULE_BASED_TAGGER_V1.14/Bin_and_Data/.

Start taggeren:

tagger ../../ trained_tagger/FINAL.LEXICON ../../ trained_tagger/CORPUS ../../ trained_tag-ger/BIGBIGRAMLIST ../../ trained_tagger/LEXRULEOUTFILE trained_tagger/CONTEXT-RULEFILE > ../../ trained_tagger/CORPUS-TAGGED

INPUT = CORPUS det korpus der skal tagges dvs utagget

OUTPUT = CORPUS-TAGGED samme korpus nu tagget

Der er forskellige options man kan bruge:

-h :: help

-w wordlist :: provide an extra set of words beyond those in LEXICON.

See below.

-i filename :: writes intermediate results from start state tagger

into filename

-s number :: processes the corpus to be tagged "number" lines at

a time. This should be specified if memory problems

result from trying to process too large a corpus at

once. For example, on a Sparc 10 with 32 meg RAM,

I usually process 250,000 words at a time. On a

machine with 48 meg, I typically do 500,000 words.

(note that "number" is the number of lines, not

words). These numbers are just guidelines. You

can test out what works best for you if you plan to

tag large corpora.

-S :: use start state tagger only.

-F :: use final state tagger only. In this case,

YOUR-CORPUS is a tagged corpus, whose taggings will

be changed according to the final-state-tagger

contextual rules. YOUR-CORPUS should be a tagged

corpus ONLY when using this option.

The tagger writes to standard output.

 

---------------------------------------------------------

TEST AF BRILL-TAGGEREN

---------------------------------------------------------

1) Hvis taggeren skal testes, skal der bruges:

- et tagget test-korpus ("test")

- samme korpus i utagget form ("test.untagged")

Brug:

cat test | perl RULE_BASED_TAGGER_V1.14/Utilities/

tagged-to-untagged.prl > test.untagged

(INPUT = parole.test.FSTL OUTPUT = test.untagged)

2) Så tagges "test.untagged":

tagger ../../ trained_tagger/FINAL.LEXICON ../../ test.untagged ../../ trained_tagger/BIGBIGRAMLIST

../../ trained_tagger/LEXRULEOUTFILE ../../ trained_tagger/CONTEXT-RULEFILE > ../../trained_tagger/test.tagged

(INPUT = test.untagged OUTPUT = test.tagged)

(Stå i direktoriet RULE_BASED_TAGGER_V1.14/Bin_and_Data/)

3) De 2 taggede korpora ("test" & "test.tagged") kan nu sammenlignes:

perl PROG.dif

(INPUT = test, test.tagged OUTPUT = test.dif)

(Stå i hjemmekataloget)

4) Recall og Precision udregnes ....... , fejl rettes og forbedringer laves