Downloader race condition with multiple processes

[naktinis]

When calling nltk.download followed by model usage from multiple processes at once, at least four types of errors can happen:

• Error with downloaded zip file
• [Errno 2] No such file or directory
• zipfile.BadZipFile: Bad CRC-32 for file
• LookupError

I can reproduce this every time using the following script (but you may need to tweak the sleep time or number of processes to match your hardware and internet connection):

import os
import nltk
import time
import random
import shutil
import multiprocessing

def download_and_run():
    print("Downloading...")
    time.sleep(random.random() * 2)
    nltk.download('punkt', force=True, download_dir='models')
    nltk.data.path.insert(0, 'models')
    print(nltk.sent_tokenize("Some text to split. Another sentence."))

if __name__ == "__main__":
    shutil.rmtree('models', ignore_errors=True)

    processes = []
    for i in range(10):
        processes.append(multiprocessing.Process(target=download_and_run))

    for p in processes:
        p.start()

    for p in processes:
        p.join()

The issue can be solved by writing model files atomically, as proposed in this pull request: #3247.