Catch KeyboardInterrupt and gracefully exit a ThreadPoolExecutor in Python

Introduction

The standard library module concurrent.futures in Python 3 provides an easy way to speed up completing a task for every element in a list. I’ve used it for searching every file in a folder for a certain string (check out pathlib too if you don’t use it already) and for downloading every webpage from an index.

However, it can be scary to try to stop a threaded program while it’s running, with ctrl+c. You might be interrupting not just one but dozens of different subroutines at a bad point and causing issues.

So, you can try to add a try-catch block like this:

try:
    with ThreadPoolExecutor() as executor:
        executor.map(func, iterables)
except KeyboardInterrupt:
    print('Gracefully exiting!')
    executor.shutdown(cancel_futures=True)
    print('Finished shutting down the thread pool.')

However, you’ll still get an error message and other unexpected behaivour when you interrupt the program.

Exception ignored in: <module 'threading' from '/usr/lib/python3.9/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 1415, in _shutdown
    atexit_call()
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 31, in _python_exit
    t.join()
  File "/usr/lib/python3.9/threading.py", line 1033, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.9/threading.py", line 1049, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt:

I did a lot of googling and attempted fixes before I found the solution. I found the solution by reading the documentation carefully, which should be a lesson to me…

Reference:

• Docs for Executor.map - read carefully https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor.map
• Code where I encountered and solved this issue (in sync_all_projects) https://github.com/gnu-mirror-unofficial/gnu-mirror/blob/main/mirror.py

Solution

The answer is that Executor actually returns a generator immediately - it doesn’t block the calling thread. This means that if you use Executor.map like the code block above, the main thread will finish immediately, and only the Executor’s threads will be keeping the program running. Not a tidy situation. Demo:

with ThreadPoolExecutor() as executor:
    executor.map(func, iterables)
    print('executor.map already returned.')

To fix it, iterate over the generator returned by Executor.map. It will block for each Future completed.

try:
    with ThreadPoolExecutor() as executor:
        futures = executor.map(func, iterables)
        for f in future:
            # you can do anything you want with future here.
            pass
except KeyboardInterrupt:
    print('Gracefully exiting!')
    executor.shutdown(cancel_futures=True)
    print('Finished shutting down the thread pool.')

Thanks for reading.