
Taming PyInstaller: My Journey to a Working Backtesting Executable
When I set out to build an executable for my backtesting application using PyInstaller, I ran into a series of errors that taught me a lot about packaging Python applications with nonstandard dependencies. In this post, I’ll share my discoveries—from initial module-not-found errors to finally resolving a missing data file issue—with detailed explanations and a reproducible build script.
Environment Setup
I’m working with Python 3.12.2 inside a dedicated virtual environment located in a subfolder called venv
. This isolated setup ensures that all dependencies are neatly contained, and it helps prevent conflicts with system-wide packages. In the following post, I detail the steps I took to build an executable for my backtesting application using PyInstaller.
The Early Hurdles: ModuleNotFoundError
At first, I encountered a basic error when running the executable:
ModuleNotFoundError: No module named 'backtesting'
This was confusing because my development environment had the backtesting
module installed. I suspected that PyInstaller might not be detecting the module correctly. I tried several approaches:
- Using the
--hidden-import
Flag:
I rebuilt the executable with:pyinstaller --onefile --hidden-import=backtesting demo.py
Unfortunately, this did not solve the issue. - Verifying the Environment:
I confirmed that the module was installed in my active Python environment:pip show backtesting
Everything looked good, so I moved on. - Checking for Naming Conflicts:
I ensured that my project did not include a local file or folder namedbacktesting
that could interfere with the import.
Despite these attempts, the error persisted, and it seemed that PyInstaller was somehow missing the module during runtime.
A New Twist: Dynamic Imports and Data Files
After further investigation with debug logging enabled, I discovered that PyInstaller was indeed detecting the module. However, a different error emerged when running the executable:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/_MEI3utdP7/backtesting/autoscale_cb.js'
This error indicated that while the Python code for backtesting
was bundled, one of its essential data files—autoscale_cb.js
—was missing. The backtesting package uses this JavaScript file for plotting and dynamic scaling, and its absence caused the executable to crash.
I tried several solutions:
- Using
--collect-all
:
I attempted to include all data files from the package with:pyinstaller --onefile --collect-all backtesting demo.py
Unfortunately, this didn’t resolve the issue. - Creating a Custom Hook:
I created a custom hook file (hook-backtesting.py
) with:from PyInstaller.utils.hooks import collect_all datas, binaries, hiddenimports = collect_all('backtesting')
Despite this, the file was still not found at runtime.
It became clear that the key was to explicitly add the missing JavaScript file into the bundle.
The Breakthrough: Adding the Missing File
After some research, I discovered that the --add-data
flag could be used to manually include non-code files. I located the file on my system at:
/home/peter/.pyenv/versions/3.12.2/lib/python3.12/site-packages/backtesting/autoscale_cb.js
I then built the executable with the following command:
pyinstaller --onefile \
--add-data "/home/peter/.pyenv/versions/3.12.2/lib/python3.12/site-packages/backtesting/autoscale_cb.js:backtesting" \
demo.py --log-level=DEBUG --clean
Let’s break down what this command does:
--onefile
: Bundles everything into a single executable.--add-data
:- Source Path: The absolute path to
autoscale_cb.js
on my system. - Destination Folder: The file is placed in a folder named
backtesting
within the executable. This ensures that when the backtesting package tries to load the file via its relative path, it finds it.
- Source Path: The absolute path to
demo.py
: The main script of my application.--log-level=DEBUG
and--clean
: These options provide detailed logging and ensure a clean build environment, which is invaluable for troubleshooting.
After rebuilding with this command, the executable ran perfectly—backtesting
could now locate autoscale_cb.js
, and the FileNotFoundError was resolved.
Documenting the Final Build Script
To streamline future builds, I encapsulated the working command into a shell script. Here’s my build.sh
script:
#!/bin/bash
# build.sh
# This script builds the demo executable using PyInstaller,
# ensuring that the necessary backtesting data file is included.
# Clean previous builds and run PyInstaller with debug logging.
pyinstaller --onefile \
--add-data "/home/peter/.pyenv/versions/3.12.2/lib/python3.12/site-packages/backtesting/autoscale_cb.js:backtesting" \
demo.py \
--log-level=DEBUG \
--clean
How to Use the Build Script
- Make the Script Executable:
chmod +x build.sh
- Run the Script:
./build.sh
- Verify the Executable: The resulting executable is in the
dist/
directory. Run it with:./dist/demo
Conclusion
My journey with PyInstaller was a learning experience. I encountered initial issues with missing modules and dynamic data files, and I tried various solutions—from hidden imports to custom hooks. Ultimately, the solution was to manually include the missing autoscale_cb.js
file using the --add-data
flag.
This process taught me the importance of understanding how external data files are bundled with Python executables. For anyone packaging applications with similar dynamic or nonstandard dependencies, I hope my discoveries and documented build script can serve as a helpful guide.
Happy packaging!