Python Project Layout


One of the biggest problems for me, when I was a beginner, was how to structure my Python project. Finally I’ve learned the basics when I was reading Learn Python the Hard Way. You need to have a proper structure for your project in order to make things easier for you and the contributors.

This article will present you with a project skeleton that will help you get up and running with the new projects faster.

Anatomy of a simple Python project

The most basic setup of your project should contain the following:

.
├── bin
├── docs
├── src
├── tests
├── CHANGELOG.md
├── README.md
├── requirements.txt
└── setup.py

bin/ is directory in which you can place helper scripts for your project.

docs/ is self explanatory - put your documentaiton here. If it’s suitable for your project, you can use sphinx.

src/ is the “source” code of your project. Do not put your projects scripts in the root of your project. That is messy. I would recommend making src/ even for a small project. Alternativelly, this directory can be named after your project. And this “alternative” way is probably more common in the Python ecosystem. Django project named their source directory django/.

tests/ is for keeping your unit tests, and integration tests in one place.

You should always version your library/app, and keep up to date CHANGELOG.md. If you’re not sure how to do it, use SemVer (semantic versioning), and see these recommendations.

Always keep a comprehensible README.md file. It should contain instructions how to run the project, what this project is about, and how to contribute. Also, I like to put some tips and tricks, and explanations about scripts I keep in the bin/. From time to time you will forget some quircks about your project, and this could serve as a refresher. README.md is just as important as your docstrings and comments.

Depending on how you manage your depenendices you will need requirements.txt or something similar to that. Never commit a Python project without dependencies. This makes it hard for people to run your project.

Finally, probably the least understood part of the project: setup.py. If you’re a complete beginner, this will be the most challenging part. In this case setup.py is used to package your application. When do you need it? In case you’re building e.g. an API wrapper library. In fact, anytime you’re building a library you’re gonna need it. However, if you’re building a web app using a framework such as Django, you probably won’t need it.

Sometimes you can add examples/ directory with working app/script example. But this can be an overhead, because you have to maintain it.

This covers the most basic setup. But in production you will probably need much much more.

“Advanced” layout

The quoatation marks in the subtitle are here because this is not really an advanced layout. In fact, any project layout will depend on technologies it uses. For example, let’s assume you are using Docker to bundle your application, flake8 to lint your application, and bandit to check the security of your code. Of course, you’re keeping your source code versioned controled with git. In this case, this will be the layout of your project:

.
├── bin
├── docs
├── src
├── tests
├── docker
│   └── entrypoint
│       └── server.sh
├── .gitignore
├── .dockerignore
├── .bandit
├── .flake8
├── CHANGELOG.md
├── README.md
├── Dockerfile
├── requirements.txt
└── setup.py

.flake8 and .bandit are configuration files for the linter and static security analysis, respectively.

When docker is in question, I like to put my entrypoint scripts in the docker/ directory. In case I have more than one entrypoint script (e.g. you have to start an app server, and a celery server) I like to put them in the entrypoint/ subdirectory.

Just give me an example

The example is available in this repository. First, make the project skeleton directory:

mkdir -p ~/repos/python-project-skeleton
cd ~/repos/python-project-skeleton
git init

Then, create all basic files:

mkdir bin src docs tests
touch README.md CHANGELOG.md setup.py .flake8

Dependencies of the project will be managed with pip-tools.

mkdir requirements.in requirements
python3.7 -m venv env
echo 'env/' >> .gitignore
. env/bin/activate
pip install pip-tools

Finally, create python packages for your library, and tests:

mkdir src/project
touch src/project/__init__.py tests/__init__.py

Now, we will need to install the dependencies of the project. Let’s say we’re building an API wrapper. You will need to have requests as your base dependency, and optionally you can include flake8 to check your code’s PEP8 compliance.

touch requirements.in/base.txt
echo 'requests==2.23.0' >> requirements.in/base.txt
touch requirements.in/dev.txt

The content of your dev requirements should be (dev.txt):

-r base.txt

flake8==3.7.9

Now, compile and install your dependencies:

pip-compile -o requirements/dev.txt requirements.in/dev.txt
pip-sync requirements/dev.txt

Now, you are almost ready to build your project! Let’s write some dummy code, and try to test it.

# src/project/project.py
import requests


class ApiWrapper:

    def __init__(self, url):
        self.url = url
    
    def get(self, endpoint):
        return requests.get(f'{self.url}/{endpoint}').json()

And then add unit test for it:

# tests/test_project.py
import unittest
from unittest import mock
from project.project import ApiWrapper


class MockHTTPResponse:

    def __init__(self, return_value):
        self.return_value = return_value

    def json(self):
        return self.return_value



class ApiWrapperTestCase(unittest.TestCase):

    def setUp(self):
        self.wrapper = ApiWrapper('https://api.example.com')

    @mock.patch(
        'requests.get',
        mock.MagicMock(
            return_value=MockHTTPResponse([{'1': 'user1'}, {'2': 'user2'}])
        )

    )
    def test_get_endpoint(self):
        response = self.wrapper.get('users/')
        self.assertEqual(response, [{'1': 'user1'}, {'2': 'user2'}])

Now, if you try to run tests, they will fail. And they will fail badly:

$ pipenv run coverage run -m unittest discover
E
======================================================================
ERROR: tests.test_project (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.test_project
Traceback (most recent call last):
  File "/usr/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/user/repos/python-project-skeleton/tests/test_project.py", line 3, in <module>
    from project.project import ApiWrapper
ModuleNotFoundError: No module named 'project'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

Our project package cannot be imported. In order to make this project skeleton work, the final step is to write our setup.py and install our package as a dev dependency i.e. as an editable pip package.

setup.py

Finally the dreaded setup.py. When you set this file, you can use pip to install your library as editable in your virtualenv. This means that any changes made in your code will be reflected in scripts that are using the library i.e. you can develop and unit test at the same time, without a need to package application in between.

import os
from setuptools import find_packages, setup


# This allows setup.py to be run from any path
os.chdir(os.path.normpath(os.path.join(os.path.abspath(__file__), os.pardir)))

setup(
    name='project',
    packages=find_packages(where='src'),
    package_dir={'': 'src'},
    include_package_data=True,
    description='My API Wrapper',
    long_description='You can use Python file API to load your README here',
    author='Petar Gitnik',
    install_requires=['requests'],
    classifiers=[
        'Programming Language :: Python',
        'Programming Language :: Python :: 3.7',
    ],
)

Since this article is only concerned with project layout, and not with Python project packaging, I will not go into the details about setup.py. Suffice to say, now we can update our dev.txt with our project as dependency. Add the following lines in your dev.txt:

-e .

Now if you run tests, everything is OK:

$ python -m unittest discover
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

What about lib/?

I’m too lazy to remember proper command for running unit tests, so I usually put it in the lib/. For example:

#!/bin/bas
# Don't forget to activate virtual environment before using this script
python -m unittest discover

That’s it. You’re ready to start building your projects.

python 

We're not spammers, and you can opt-out at any moment. We hate spam as much as you do.

powered by TinyLetter

See also