Taming PyInstaller: My Journey to a Working Backtesting Executable

When I set out to build an executable for my backtesting application using PyInstaller, I ran into a series of errors that taught me a lot about packaging Python applications with nonstandard dependencies. In this post, I’ll share my discoveries—from initial module-not-found errors to finally resolving a missing data file issue—with detailed explanations and a reproducible build script.

Environment Setup

I’m working with Python 3.12.2 inside a dedicated virtual environment located in a subfolder called venv. This isolated setup ensures that all dependencies are neatly contained, and it helps prevent conflicts with system-wide packages. In the following post, I detail the steps I took to build an executable for my backtesting application using PyInstaller.

---

The Early Hurdles: ModuleNotFoundError

At first, I encountered a basic error when running the executable:

ModuleNotFoundError: No module named 'backtesting'

This was confusing because my development environment had the backtesting module installed. I suspected that PyInstaller might not be detecting the module correctly. I tried several approaches:

1. Using the --hidden-import Flag:
   I rebuilt the executable with: pyinstaller --onefile --hidden-import=backtesting demo.py Unfortunately, this did not solve the issue.
2. Verifying the Environment:
   I confirmed that the module was installed in my active Python environment: pip show backtesting Everything looked good, so I moved on.
3. Checking for Naming Conflicts:
   I ensured that my project did not include a local file or folder named backtesting that could interfere with the import.

Despite these attempts, the error persisted, and it seemed that PyInstaller was somehow missing the module during runtime.

---

A New Twist: Dynamic Imports and Data Files

After further investigation with debug logging enabled, I discovered that PyInstaller was indeed detecting the module. However, a different error emerged when running the executable:

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/_MEI3utdP7/backtesting/autoscale_cb.js'

This error indicated that while the Python code for backtesting was bundled, one of its essential data files—autoscale_cb.js—was missing. The backtesting package uses this JavaScript file for plotting and dynamic scaling, and its absence caused the executable to crash.

I tried several solutions:

1. Using --collect-all:
   I attempted to include all data files from the package with: pyinstaller --onefile --collect-all backtesting demo.py Unfortunately, this didn’t resolve the issue.
2. Creating a Custom Hook:
   I created a custom hook file (hook-backtesting.py) with: from PyInstaller.utils.hooks import collect_all datas, binaries, hiddenimports = collect_all('backtesting') Despite this, the file was still not found at runtime.

It became clear that the key was to explicitly add the missing JavaScript file into the bundle.

---

The Breakthrough: Adding the Missing File

After some research, I discovered that the --add-data flag could be used to manually include non-code files. I located the file on my system at:

/home/peter/.pyenv/versions/3.12.2/lib/python3.12/site-packages/backtesting/autoscale_cb.js

I then built the executable with the following command:

pyinstaller --onefile \
  --add-data "/home/peter/.pyenv/versions/3.12.2/lib/python3.12/site-packages/backtesting/autoscale_cb.js:backtesting" \
  demo.py --log-level=DEBUG --clean

Let’s break down what this command does:

• --onefile: Bundles everything into a single executable.
• --add-data:
  • Source Path: The absolute path to autoscale_cb.js on my system.
  • Destination Folder: The file is placed in a folder named backtesting within the executable. This ensures that when the backtesting package tries to load the file via its relative path, it finds it.
• demo.py: The main script of my application.
• --log-level=DEBUG and --clean: These options provide detailed logging and ensure a clean build environment, which is invaluable for troubleshooting.

After rebuilding with this command, the executable ran perfectly—backtesting could now locate autoscale_cb.js, and the FileNotFoundError was resolved.

---

Documenting the Final Build Script

To streamline future builds, I encapsulated the working command into a shell script. Here’s my build.sh script:

#!/bin/bash
# build.sh
# This script builds the demo executable using PyInstaller,
# ensuring that the necessary backtesting data file is included.

# Clean previous builds and run PyInstaller with debug logging.
pyinstaller --onefile \
            --add-data "/home/peter/.pyenv/versions/3.12.2/lib/python3.12/site-packages/backtesting/autoscale_cb.js:backtesting" \
            demo.py \
            --log-level=DEBUG \
            --clean

How to Use the Build Script

1. Make the Script Executable: chmod +x build.sh
2. Run the Script: ./build.sh
3. Verify the Executable: The resulting executable is in the dist/ directory. Run it with: ./dist/demo

---

Conclusion

My journey with PyInstaller was a learning experience. I encountered initial issues with missing modules and dynamic data files, and I tried various solutions—from hidden imports to custom hooks. Ultimately, the solution was to manually include the missing autoscale_cb.js file using the --add-data flag.

This process taught me the importance of understanding how external data files are bundled with Python executables. For anyone packaging applications with similar dynamic or nonstandard dependencies, I hope my discoveries and documented build script can serve as a helpful guide.

Addendum for python-dotenv

I ran into another special case, well maybe not so special since I use it all the time, but if you are using python-dotenv (with module name dotenv) I had to do this:

pyinstaller --onefile \
--add-data "/home/peter/projects/Telegram-REST-Server/venv/lib/python3.12/site-packages/dotenv:dotenv" \
telegram_rest_server.py

As you can see, I am using a virtual env in subfolder ./venv

If I did not do this when building, I always got an error:

>> ./dist/telegram_rest_server --port 8999
Traceback (most recent call last):
File "telegram_rest_server.py", line 16, in
from dotenv import load_dotenv
ModuleNotFoundError: No module named 'dotenv'
[PYI-434894:ERROR] Failed to execute script 'telegram_rest_server' due to unhandled exception!