05/02/2022 | News release | Distributed by Public on 05/02/2022 13:18
C (and less commonly, Fortran) libraries are a necessary evil in Python since they significantly speed up compute-intensive routines, such as math calculations. This makes them indispensable to data analysis and machine learning packages, from TensorFlow to NumPy to Scikit-Learn.
Luckily, most Python packages are "pure Python," which means they don't rely on external libraries written in a different language. You can easily install them by simply running pip install . However, after installing and trying to run a package such as Pillow, you might get an error like:
x86_64-linux-gnu-gcc: error: build/temp.linux-x86_64-3.4/libImaging/Jpeg2KDecode.o: No such file or directory
This is because Pillow is one of those Python packages that requires C libraries to run, and your system is attempting to build the missing C files from source code. But creating a local build environment isn't usually part of the standard process when installing Python, so what's going on?
These days, Python developers of packages that incorporate C code typically build a pre-compiled binary wheel for most popular operating systems (OS), and then make the binaries available via the Python Package Index (PyPI). This way, when you run pip install pip can select the appropriate pre-compiled wheel for your platform and install both the Python code and the required C libraries.
While installing pre-built wheels is a convenient shortcut, they raise two concerns:
This blog post is meant to help you understand your options when it comes to managing C libraries associated with your Python deployments, as well as introducing options if you need to build Python packages yourself.
The C and/or Fortran code associated with a Python package is typically distributed in the package's source code archive, which you'll need to separately download and install. But you'll also need to build the source code for your OS before you can use the package, which means you'll need to create an appropriate build environment on your local system that includes:
While this kind of setup is expedient, it has a number of drawbacks. For example:
Because of the last point, organizations often centralize the build process in a single team that builds the Python artifacts required by all teams. This method comes with its own set of tasks and requirements, including:
If all this sounds like a lot of work to set up and maintain, and then rebuild the packages required for each project over time as vulnerabilities are found and new versions are released, you'd be correct. Luckily, there's a better way.
The ActiveState Platform provides a cloud-based secure build service that automatically builds Python packages from source code, including any linked C and Fortran libraries. As such, the ActiveState Platform:
And it does it all in a secure, repeatable manner so you can be sure that the artifacts it produces are trustworthy. There's no need for you to have OS expertise, or even Python experience to be able to securely build all the packages your teams require from source.
See how it works:
If you'd like to understand more about how easy it is to automatically build all the Python components your teams require securely from source code:
Experienced Product Marketer and Product Manager with a demonstrated history of success in the computer software industry. Strong skills in Product Lifecycle Management, Pragmatic Marketing methods, Enterprise Software, Software as a Service (SaaS), Agile Methodologies, Customer Relationship Management (CRM), and Go-to-market Strategy.