'[f18dZddlZddlZddlmZddlmZmZddlm Z dZ Gdde Z Gd d Z Gd d Z Gd deZdS)z9 A reader for corpora whose documents are in MTE format. N)reduce)TaggedCorpusReaderconcat) XMLCorpusViewc.|||SN)findall)rootpathnss J/var/www/piapp/venv/lib/python3.11/site-packages/nltk/corpus/reader/mte.pyxpathr s <<b ! !!c"eZdZdZddZddZdS) MTECorpusViewz0 Class for lazy viewing the MTE Corpus. Nc4tj||||dSr)r__init__)selffileidtagspec elt_handlers r rzMTECorpusView.__init__stVWkBBBBBrc httdtj||||S)Nc |duSrxs r z*MTECorpusView.read_block.. !4-r)listfilterr read_block)rstreamrrs r r!zMTECorpusView.read_blocks; ''(vw LL     rr)NN)__name__ __module__ __qualname____doc__rr!rrr rrsJCCCC      rrc2eZdZdZdddZdZdZdZdZd Z d Z e d Z e d Z e d Ze dZe dZe dZe dZe dZe dZdZdZdZdZdZdZdZdZdZdS) MTEFileReaderz Class for loading the content of the multext-east corpus. It parses the xml files and does some tag-filtering depending on the given method parameters. zhttps://www.tei-c.org/ns/1.0z%https://www.w3.org/XML/1998/namespace)teixmlz{https://www.tei-c.org/ns/1.0}z'{https://www.w3.org/XML/1998/namespace}zTEI/text/body/div/div/p/s/(w|c)zTEI/text/body/div/div/p/szTEI/text/body/div/div/pc||_dSr)_MTEFileReader__file_path)r file_paths r rzMTEFileReader.__init__2s$rc|jSr)textclseltcontexts r _word_eltzMTEFileReader._word_elt5s xrcHfdt|djDS)Nc<g|]}|dSr)r4.0wr1s r z+MTEFileReader._sent_elt..;'HHH1 a&&HHHr*rr r0s` r _sent_eltzMTEFileReader._sent_elt9+HHHHc30G0GHHHHrcHfdt|djDS)Nc<g|]}|dSr)r>r8sr1s r r:z+MTEFileReader._para_elt..?r;rr<r=r0s` r _para_eltzMTEFileReader._para_elt=r?rc>d|jvr |jdfS|jdkr|jdkr|j|jdfS|jdkr7|jdkr,|jt|jdfSt jdt jdd|jzdz}| |jdrK|jdkr|j|jdfS|jt|jdfSdS) Nanamsd universal^-.z.*$) attribr/_MTEFileReader__tags_MTEFileReader__tagsetMTETagConvertermsd_to_universalrecompilesubmatch)r1r2r3tagss r _tagged_word_eltzMTEFileReader._tagged_word_eltAs  " "Hb> ! :    5 5Hcj/0 0 Z2  #,+"="=Ho>>sz%?PQQR R:cBF3SZ$@$@@5HIIDzz#*U+,, <5((Hcj&788'88E9JKK trc ttdfdt|djDS)Nc |duSrrrs r rz0MTEFileReader._tagged_sent_elt..[rrc<g|]}|dSr)rWr7s r r:z2MTEFileReader._tagged_sent_elt..\)PPP1%%a..PPPrr<rr rr r0s` r _tagged_sent_eltzMTEFileReader._tagged_sent_eltWM ''PPPPc38O8OPPP     rc ttdfdt|djDS)Nc |duSrrrs r rz0MTEFileReader._tagged_para_elt..drrc<g|]}|dSr)r]rBs r r:z2MTEFileReader._tagged_para_elt..er[rr<r\r0s` r _tagged_para_eltzMTEFileReader._tagged_para_elt`r^rcNd|jvr |jdfS|j|jdfS)NlemmarG)rMr/r0s r _lemma_word_eltzMTEFileReader._lemma_word_eltis/ #* $ $Hb> !Hcj12 2rcHfdt|djDS)Nc<g|]}|dSr)rer7s r r:z1MTEFileReader._lemma_sent_elt..r)NNN##At,,NNNrr<r=r0s` r _lemma_sent_eltzMTEFileReader._lemma_sent_eltp+NNNNeCcf6M6MNNNNrcHfdt|djDS)Nc<g|]}|dSr)rirBs r r:z1MTEFileReader._lemma_para_elt..vrhrr<r=r0s` r _lemma_para_eltzMTEFileReader._lemma_para_elttrjrcVt|jtjtjSr)rr,r( word_pathr4rs r wordszMTEFileReader.wordsx$  m5}7N   rcVt|jtjtjSr)rr,r( sent_pathr>rps r sentszMTEFileReader.sents}rrrcVt|jtjtjSr)rr,r( para_pathrDrps r paraszMTEFileReader.parasrrrcVt|jtjtjSr)rr,r(rorerps r lemma_wordszMTEFileReader.lemma_words$  m5}7T   rc|t_|t_t|jtjtjSr)r(rOrNrr,rorWrtagsetrVs r tagged_wordszMTEFileReader.tagged_words4!' #   m5}7U   rcVt|jtjtjSr)rr,r(rtrirps r lemma_sentszMTEFileReader.lemma_sentsr{rc|t_|t_t|jtjtjSr)r(rOrNrr,rtr]r}s r tagged_sentszMTEFileReader.tagged_sentsrrcVt|jtjtjSr)rr,r(rwrmrps r lemma_paraszMTEFileReader.lemma_parasr{rc|t_|t_t|jtjtjSr)r(rOrNrr,rwrbr}s r tagged_paraszMTEFileReader.tagged_parasrrN)r#r$r%r&r tag_nsxml_nsrortrwr classmethodr4r>rDrWr]rbrerirmrqrurxrzrrrrrrrr r(r(!s.6  B.F 6F1I+I)I%%%[II[III[I[*  [   [ 33[3 OO[OOO[O                             rr(c FeZdZdZdddddddd d d d d d ZedZdS)rPzu Class for converting msd tags to universal tags, more conversion options are currently not implemented. ADJADPADVCONJDETNOUNNUMPRTPRONVERBrLX) ASRCDNMQPVrLrKc~|ddks|dn|d}|tjvrd}tj|S)z This function converts the annotation from the Multex-East to the universal tagset as described in Chapter 5 of the NLTK-Book Unknown Tags will be mapped to X. Punctuation marks are not supported in MSD tags, so r#rK)rPmapping_msd_universal)tag indicators r rQz MTETagConverter.msd_to_universalsB#&a&C--CFFSV OAAAI4Y??rN)r#r$r%r&r staticmethodrQrrr rPrPsq               @ @\ @ @ @rrPcheZdZdZddZdZddZddZddZdd Z dd Z dd Z ddZ ddZ ddZdS)MTECorpusReaderz Reader for corpora following the TEI-p5 xml scheme, such as MULTEXT-East. MULTEXT-East contains part-of-speech-tagged words with a quite precise tagging scheme. These tags can be converted to the Universal tagset Nutf8cBtj||||d|_dS)a. Construct a new MTECorpusreader for a set of documents located at the given root directory. Example usage: >>> root = '/...path to corpus.../' >>> reader = MTECorpusReader(root, 'oana-*.xml', 'utf8') # doctest: +SKIP :param root: The root directory for this corpus. (default points to location in multext config file) :param fileids: A list or regexp specifying the fileids in this corpus. (default is oana-en.xml) :param encoding: The encoding of the given files (default is utf8) z 00README.txtN)rr_readme)rr fileidsencodings r rzMTECorpusReader.__init__s& #D$BBB% rc|j}nt|tr|g}tfd|}td|}|st d|S)Nc|jvSr)_fileids)rrs r rz+MTECorpusReader.__fileids..s1 #5rc |dvS)N)z oana-bg.xmlz oana-mk.xmlrrs r rz+MTECorpusReader.__fileids..s 1,J#Jrz$No valid multext-east file specified)r isinstancestrr printrrs` r __fileidszMTECorpusReader.__fileidssr ?mGG  % % iG5555w??JJGTT : 8 9 9 9rc`tfd|DS)z :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of words and punctuation symbols. :rtype: list(str) cg|]F}ttjj|GSr)r(osr join_rootrqr8frs r r:z)MTECorpusReader.words..O   bgll4:q99::@@BB   rr_MTECorpusReader__fileidsrs` r rqzMTECorpusReader.wordssI     00      rc`tfd|DS)z :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of sentences or utterances, each encoded as a list of word strings :rtype: list(list(str)) cg|]F}ttjj|GSr)r(rr rrrurs r r:z)MTECorpusReader.sents.. rrrrs` r ruzMTECorpusReader.sentsI    00      rc`tfd|DS)a :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as lists of word string :rtype: list(list(list(str))) cg|]F}ttjj|GSr)r(rr rrrxrs r r:z)MTECorpusReader.paras..rrrrs` r rxzMTECorpusReader.parasrrc`tfd|DS)a :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of words, the corresponding lemmas and punctuation symbols, encoded as tuples (word, lemma) :rtype: list(tuple(str,str)) cg|]F}ttjj|GSr)r(rr rrrzrs r r:z/MTECorpusReader.lemma_words..%O   bgll4:q99::FFHH   rrrs` r rzzMTECorpusReader.lemma_wordsrrrHrGcdksdkr0tfd|DStddS)a; :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of tagged words and punctuation symbols encoded as tuples (word, tag) :rtype: list(tuple(str, str)) rIrHcg|]H}ttjj|ISr)r(rr rrrr8rrrVr~s r r:z0MTECorpusReader.tagged_words..8Z""',,tz1"="=>>KKrUnknown tagset specified.Nrrrrrr~rVs` ``r rzMTECorpusReader.tagged_words+ [ FeOO"^^G44   - . . . . .rc`tfd|DS)aB :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of sentences or utterances, each encoded as a list of tuples of the word and the corresponding lemma (word, lemma) :rtype: list(list(tuple(str, str))) cg|]F}ttjj|GSr)r(rr rrrrs r r:z/MTECorpusReader.lemma_sents..Krrrrs` r rzMTECorpusReader.lemma_sentsBI    00      rcdksdkr0tfd|DStddS)aH :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of sentences or utterances, each each encoded as a list of (word,tag) tuples :rtype: list(list(tuple(str, str))) rIrHcg|]H}ttjj|ISr)r(rr rrrrs r r:z0MTECorpusReader.tagged_sents..^rrrNrrs` ``r rzMTECorpusReader.tagged_sentsQrrc`tfd|DS)am :param fileids: A list specifying the fileids that should be used. :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as a list of tuples of the word and the corresponding lemma (word, lemma) :rtype: list(List(List(tuple(str, str)))) cg|]F}ttjj|GSr)r(rr rrrrs r r:z/MTECorpusReader.lemma_paras..qrrrrs` r rzMTECorpusReader.lemma_parashrrcdksdkr0tfd|DStddS)a :param fileids: A list specifying the fileids that should be used. :param tagset: The tagset that should be used in the returned object, either "universal" or "msd", "msd" is the default :param tags: An MSD Tag that is used to filter all parts of the used corpus that are not more precise or at least equal to the given tag :return: the given file(s) as a list of paragraphs, each encoded as a list of sentences, which are in turn encoded as a list of (word,tag) tuples :rtype: list(list(list(tuple(str, str)))) rIrHcg|]H}ttjj|ISr)r(rr rrrrs r r:z0MTECorpusReader.tagged_paras..rrrNrrs` ``r rzMTECorpusReader.tagged_parasws [ FeOO"^^G44   - . . . . .r)NNrr)NrHrG)r#r$r%r&rrrqrurxrzrrrrrrrr rrs & & & &                       ////.     ////.     //////rr)r&rrR functoolsrnltk.corpus.readerrrnltk.corpus.reader.xmldocsrrrr(rPrrrr rs9 99999999444444"""     M   "H H H H H H H H V"@"@"@"@"@"@"@"@J|/|/|/|/|/(|/|/|/|/|/r