python: error while loading shared libraries:
I recently decided to live on the edge and actually update to the current version of my operating system. I made my backups and found a good book to keep busy during the five hours my laptop needs for such a Herculean task. Everything went smoothly, and I was able to continue working without issue.
A bit later I wanted to work on a django project that I hadn't touched for a while. I activated the virtual environment and tried to run the server
$ python manage.py runserver
python: error while loading shared libraries: libpython3.7m.so.1.0: cannot open
shared object file: No such file or directory
That... was not what I was hoping for.
I checked a few other projects with virtual environments and found that all of the virtual environments that I created before updating my OS gave the error, while those created after the update were fine.
Now, the solution turned out to be painfully obvious, but I made a few newbie errors that slowed me down a fair bit. At least the approximately 1.2 million hits returned by google for "python: error while loading shared libraries:" (noticed while writing this that ddg doesn't give the number of results) assure me that I'm not alone. Hopefully, this note can help some future newbie.
What are .so files anyway?
Since I didn't know anything about this .so file, I initially went after a "red herring". I noticed that the working venvs had a file called pyvenv.cfg while the ones returning an error message did not. This file is mentioned in PEP 405 and, as expected, contains a home key pointing to the parent directory of the python install.
home = /usr/bin
include-system-site-packages = false
version = 3.8.5
My first thought was that the other venvs needed this file. So I copied it into a venv that wasn't working and tried running python again, both leaving the version number as 3.8.5, and after changing it to 3.7. Both attempts still gave the missing .so file error. At this point I was curious: what is this .so file anyway?
The 'so' stands for 'shared object'. As far as I currently understand, these files are important for linking external libraries and for defining how code is laid out and linked at a low level. The analagous file on Windows systems is .dll (dynamic link library). The libpython*.so files are mentioned as part of PEP 384 -- Defining a Stable ABI. For those of us lacking a formal education in computer science, ABI stands for 'application binary interface'.
The quest for the .so
After being convinced that .so files are properly important, it seemed like a good idea to find where mine were. The most likely suspects were the \lib, \lib64 and \usr directories. I searched fairly generally for .so files containing 'libpython' and \lib64 turned out to be a good guess
$ cd /lib64
$ find . -name 'libpython?.*.so*'
./libpython3.8.so.1.0
./python2.7/config/libpython2.7.so
./libpython3.8.so
./libpython2.7.so
./libpython2.7.so.1.0
ok, so there are shared object files for python libraries, but none for python3.7, and none with an 'm' in the name. To be honest, I'm still not sure what the differences between libpython?.?.so and libpython?.?m.so are. Anybody who can clarify the matter is welcome to send an e-mail :)
I wasn't sure where libpython3.7m.so.1.0 should come from, and most of the threads I found assumed that you had the file and just needed to update some reference to it. Searching on pkgs.org for "libpython3.7m.so.1.0" turned out to be more useful. It brought up packages for python 3.7 interpreters. Finally, I had the lightbulb moment that I must be missing python3.7.
Having a look at the python3.7 package for my architecture
shows that the required .so file is indeed part of this package and checking for an installation of python 3.7 with the which comand confirmed that I didn't have a python 3.7 interpreter. It must have been automatically removed during the OS update and replaced with 3.8.
Righto, then I just need to follow the friendly 'install HowTo' on pkgs.org...
$ dnf install python37
No match for argument: 37
Of course not. Do you know python3.7?
$ dnf install python3.7
Dependecies resolved.
================================================================================
Package Architecture Version Repository Size
================================================================================
Installing:
python37 x86_64 3.7.9-1.fc32 updates 19 M
Transaction Summary
================================================================================
Install 1 Package
Great! And the .so files?
$ cd /lib64
$ find . -name 'libpython3.*.so*'
./libpython3.8.so.1.0
./libpython3.7m.so.1.0
./libpython3.8.so
./libpython3.7dm.so.1.0
./libpython3.7dm.so
./libpython3.7m.so
I still don't know what all these files do in great detail (*.7dm.so?) but at least the file I wanted is indeed there. Let's go back to the introDjango venv:
$ source venv/bin/activate
(venv) [katie@linux introDjango]$ python
Python 3.7.9 (default, Aug 20 2020, 14:07:20)
[GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
It lives!
pyvenv.cfg and final thoughts
On one hand, problem solved. If you see the 'shared library not found' error, then either make a new venv (you did make a requirements.txt with pip freeze before updating the OS... right?) using one of your available python interpreters, or install the missing python version. On the other hand, I didn't understand why some of my virtual environments have a pyvenv.cfg while others do not.
Having inconsistencies between venvs sounded suspiciously like a possible issue between keyboard and chair. I had a faint memory of learning that virtual environments are a good idea, but being unsure about which package I should use to make my virtual environments. There are quite a few options, after all. So I had a look at my bash history:
$ cd ~/
$ grep -n 'venv venv' .bash_history
710:python3 -m venv venv
924:python3 -m venv venv
980:python3 -m venv venv
$ grep -n 'virtualenv venv' .bash_history
237:virtualenv venv -p python3.7
457:virtualenv venv
579:virtualenv venv
644:virtualenv venv
There you go: I mixed the venv and virtualenv packages. I checked some of the lines around of these commands in my bash history, for instance:
$ sed -n '235, 237p' .bash_history
mkdir introDjango
cd introDjango/
virtualenv venv -p python3.7
where 'introDjango' did not have a pyvenv.cfg. This basically confirmed that the venv directories containing pyvenv.cfg were the ones made with the standard library venv package.
Even though it tripped me up a bit, I think it's great that one can switch between venv and virtualenv without noticing.
But I also think this highlights the double edged sword that is 'user-friendliness'. I was allowed to be lazy about keeping track of my python versions, required files and initial project setup, and it ultimately left me more lost when something went wrong than I should have been. At least I'll know better next time!