Text streaming in Gensim - inputstream

Question

Gensim uses text streaming to minimize memory requirements. This is at the cost of performance due to endless disk IO. Is there a trick to on the fly copy the complete file from disk (one disk IO) to a temporary in-memory file?
I like to keep the code as is (no recoding into a list structures), but this is not a great way of debugging functionality
Expected result: much faster code
Some more background on the question
The original code is at https://github.com/skipgram/modern-nlp-in-python/blob/master/executable/Modern_NLP_in_Python.ipynb. The example code is taken from the phrase modelling section
I'm calculating the unigrams. All reviews are at
review_txt_filepath = os.path.join(intermediate_directory,'review_text_all.txt'),
all unigrams should go to
unigram_sentences_filepath = os.path.join(intermediate_directory, 'unigram_sentences_all.txt')
The crucial routines are
def punct_space(token):
return token.is_punct or token.is_space
def line_review(filename):
# generator function to read in reviews from the file
with codecs.open(filename, encoding='utf_8') as f:
for review in f:
yield review.replace('\\n', '\n')
def lemmatized_sentence_corpus(filename):
# generator function to use spaCy to parse reviews, lemmatize the text, and yield sentences
for parsed_review in nlp.pipe(line_review(filename),
batch_size=10000, n_threads=4):
for sent in parsed_review.sents:
yield u' '.join([token.lemma_ for token in sent
if not punct_space(token)])
The unigrams are calculated as
with codecs.open(unigram_sentences_filepath, 'w', encoding='utf_8') as f:
for sentence in lemmatized_sentence_corpus(review_txt_filepath):
f.write(sentence + '\n')
Doing this for 5000 lines requires some patience, 1h30m ;-)
I'm not that familiar with iterables, but do I understand it correctly that I first have to read the
actual file (on disc) into a variable "list_of_data" and process that
with (review_txt_filepath, 'r', encoding='utf_8') as f:
list_of_data = f.read()
with codecs.open(unigram_sentences_filepath, 'w', encoding='utf_8') as f:
for sentence in lemmatized_sentence_corpus(list_of_data):
f.write(sentence + '\n')
So the strategy is
1. read all data into a list in memory
2. process the data
3. write the results to disc
4. delete the list from memory by setting list_with_data = ()
A problem with this is obviously that line_review is doing the file reading

Answer 1

Most gensim interfaces actually take iterable sequences. Examples which emphasize streaming-from-disk just happen to use iterables that read each item as needed, but you could use an in-memory list instead.
Essentially, if you do have enough RAM to have your whole dataset in memory, just use the IO-reading iterable to read things once into a list. Then, feed that list to the gensim class where it expects any iterable sequence.
This shouldn't involve any "recoding into a list structure" – but it is using the Python list type to hold things in memory. It's the most natural way to do it, and likely the most efficient, especially in algorithms which do multiple passes over tokenized text.
(The less-idiomatic approach of, say, loading the entire file into a raw byte array, then performing repeated reading of that, file-style, to the individual items needed by the algorithm is a clunkier approach. It may similarly save on repeated IO cost, but will likely waster effort on repeated re-parsing/tokenizing of items that will be processed repeatedly. You'll want to keep each item as a Python object in memory, if you have the memory, and that requires putting them in a list.)
To be more specific in answering, you'd need to provide more details in the question, like which specific algorithms/corpus-reading-styles you're using, ideally with example code.