'[fbdZddlZddlZddlmZddlmZmZmZddl m Z GddeZ dS)z{ A reader for corpora that consist of Tweets. It is assumed that the Tweets have been serialised into line-delimited JSON. N) CorpusReader)StreamBackedCorpusViewZipFilePathPointerconcat)TweetTokenizercTeZdZdZeZ dedfdZd dZd dZ d dZ dZ dS) TwitterCorpusReadera7 Reader for corpora that consist of Tweets represented as a list of line-delimited JSON. Individual Tweets can be tokenized using the default tokenizer, or by a custom tokenizer specified as a parameter to the constructor. Construct a new Tweet corpus reader for a set of documents located at the given root directory. If you made your own tweet collection in a directory called `twitter-files`, then you can initialise the reader as:: from nltk.corpus import TwitterCorpusReader reader = TwitterCorpusReader(root='/path/to/twitter-files', '.*\.json') However, the recommended approach is to set the relevant directory as the value of the environmental variable `TWITTER`, and then invoke the reader as follows:: root = os.environ['TWITTER'] reader = TwitterCorpusReader(root, '.*\.json') If you want to work directly with the raw Tweets, the `json` library can be used:: import json for tweet in reader.docs(): print(json.dumps(tweet, indent=1, sort_keys=True)) Nutf8ctj||||||jD]N}t |t rt j|dkrtd|dO ||_ dS)a :param root: The root directory for this corpus. :param fileids: A list or regexp specifying the fileids in this corpus. :param word_tokenizer: Tokenizer for breaking the text of Tweets into smaller units, including but not limited to words. rzFile z is emptyN) r__init__abspaths_fileids isinstancerospathgetsize ValueError_word_tokenizer)selfrootfileidsword_tokenizerencodingrs N/var/www/piapp/venv/lib/python3.11/site-packages/nltk/corpus/reader/twitter.pyr zTwitterCorpusReader.__init__:s dD'8<<<MM$-00 : :D$ 233 :&&!++ !8!8!8!8999,E-cdtfd|ddDS)a( Returns the full Tweet objects, as specified by `Twitter documentation on Tweets `_ :return: the given file(s) as a list of dictionaries deserialised from JSON. :rtype: list(dict) cRg|]#\}}}|j|$S))r) CorpusView _read_tweets).0rencfileidrs r z,TwitterCorpusReader.docs..YsD   'T3d&7#FF   rT)rr )rrs` rdocszTwitterCorpusReader.docsNsM    +/==$+M+M      rc||}g}|D]_} |d}t|tr||j}||P#t $rY\wxYw|S)z Returns only the text content of Tweets in the file(s) :return: the given file(s) as a list of Tweets. :rtype: list(str) text)r$rbytesdecoderappendKeyError)rr fulltweetstweetsjsonor&s rstringszTwitterCorpusReader.strings_sYYw''   E V}dE**6;;t}55D d####     sA A** A76A7cX||}|jfd|DS)z :return: the given file(s) as a list of the text content of Tweets as as a list of words, screenanames, hashtags, URLs and punctuation symbols. :rtype: list(list(str)) c:g|]}|S)tokenize)r t tokenizers rr#z1TwitterCorpusReader.tokenized..{s'666! ""1%%666r)r.r)rrr,r4s @r tokenizedzTwitterCorpusReader.tokenizedrs8g&&( 6666v6666rcg}tdD]E}|}|s|cStj|}||F|S)zS Assumes that each line in ``stream`` is a JSON-serialised object. )rangereadlinejsonloadsr))rstreamr,ilinetweets rrz TwitterCorpusReader._read_tweets}sgr ! !A??$$D  Jt$$E MM%  r)N) __name__ __module__ __qualname____doc__rrrr r$r.r5rr1rrr r s>(J !1A1AF....(    "& 7 7 7 7     rr ) rCr:rnltk.corpus.reader.apirnltk.corpus.reader.utilrrr nltk.tokenizerr r1rrrGs  //////VVVVVVVVVV((((((sssss,sssssr