Fixing Incorrect Python Path on Ubuntu on AWS EC2

How does Python find the files it imports?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Check out my series on Automating Cybersecurity Metrics | Code.

🔒 Related Stories: Application Security | Secure Code | AWS Security | EC2

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

NOTE: This post turns out to be related to Python 2. Python 3 seems like it should be working differently. Some of the mechanisms below work in Python 2 and 3, but Python 3 seems to have different precedence. I wrote about that in this post:

Then I further summarize the issue in this post:

If you want to read where it all started and a few tricks for checking which packages Python is using…continue…but just know there’s a bit more to it and the blog post I read here seems to focus on Python 2. Things changed in Python 2.5 and Python 3.

Here’s where I started.

I’m having various problems with Python packages as I wrote in a couple of posts in my Bugs that Bite blog.

Python doesn’t seem to be using the correct libraries. As a short term fix I copied one package directly into my application but I don’t want to end up copying all of the python lib directory in there as I need new packages so let’s see if we can fix this for real.

My question was, how does Python find packages? Luckily I am not the only inquiring mind that wants to know. I love Google. This is a great post:

So following along I execute python3 to get the command prompt and type these commands. I had to change the print statement to work with python3.

So python will find any packages at those locations. Intersting that the directory I need is in there. Recall that I need my app to pick up this file:

/usr/lib/python3.10/ipaddress.py

The order and locations where Python looks for files

From the post I’m reading:

As the docs explain, sys.path is populated using the current working directory, followed by directories listed in your PYTHONPATH environment variable, followed by installation-dependent default paths, which are controlled by the site module.

So that says:

1. Working directory — which is why I dropped the python package into my app. The program looks there first before going elsewhere.
2. The PYTHONPATH environment variable.

So I echo out PYTHONPATH to see if it’s set. Nope.

echo $PYTHONPATH

3. Paths controlled by the site module.

Adding missing library paths for Python to find files

As it states in the article you can append something to the syspath:

# Add the home directory to sys.path
sys.path.append(home_dir)

Great, but the directory is already on the path.

Figuring out what path is used for a particular module

From my last post the problem I’m having is with the ipaddress.py file. Apparently you can check the file location python is referencing. So I check on the command line:

Well, that is correct, actually. Hmm.

The article also mentions using imp. Note that is being changed as of Python 3.12.

So here’s the weird thing. I created my __pycache__ directory. I re-run the code that failed yesterday on IP address. And today it works. I’m still having the session and permissions problem, but the system seems to be finding the correct python libs for ipaddress.py now — and I didn’t change anything. What?

Oh well, this has been an interesting blog post written for no reason, but now I know more about how Python finds the files it uses. I could have gone further to look at the code in this file (but for the version of Python I’m using) but I didn’t need to in this case.

'/usr/lib/python2.7/site.pyc'

What’s odd is…how did my code magically start working again?

Follow for updates.

Teri Radichel | © 2nd Sight Lab 2023

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests, Assessments, Phone Consulting ~ 2nd Sight Lab

Need Help With Cybersecurity, Cloud, or Application Security?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test or security assessment
🔒 Schedule a consulting call
🔒 Cybersecurity Speaker for Presentation

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ Twitter: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab
❤️ YouTube: @2ndsightlab