Skip to content

To effectively develop a C-extension module for Python using a template repository, the initial setup is crucial for ensuring a smooth development process. Below is an extensive guide that covers each step, from creating a Python virtual environment to configuring your environment with Poetry.

Create a Python Virtual Environment

The first step in setting up your development environment is to create a Python virtual environment. A virtual environment allows you to isolate your project’s dependencies from other projects on your system, ensuring that you can work with the exact versions of libraries needed without interference.

To create a virtual environment, you can use the following command:

python -m venv .venv

This command will create a new directory named .venv in your project’s root folder. This folder will contain a copy of the Python interpreter and a site-packages directory where all the dependencies you install will reside.

Choosing the Right Python Version for the Virtual Environment

When creating a virtual environment, it's essential to use the Python executable that matches your Minimally Supported Python Version (MSPV). For instance, if you have chosen Python 3.8 as your MSPV, you should explicitly use Python 3.8 to create the virtual environment:

python3.8 -m venv .venv

This ensures that your module is compatible with the chosen Python version, which is particularly important when writing C extensions, as compatibility with Python’s ABI (Application Binary Interface) must be maintained.

Using a Custom Script to Create the Virtual Environment

For convenience, this template repository provides a custom script, scripts/create_venv.sh, to automate the creation of the virtual environment. This script ensures that all necessary steps are followed, reducing the chances of error.

To use the script, run the following command:

bash scripts/create_venv.sh

This script will automatically detect your Python version and create the .venv folder accordingly. Using this script is recommended if you want to standardize the setup process across different environments.

Activate the Virtual Environment

Once the virtual environment is created, the next step is to activate it. Activation is necessary to ensure that your terminal uses the Python interpreter and libraries within the virtual environment rather than those installed globally.

On macOS/Linux:

To activate the virtual environment, use the following command:

source .venv/bin/activate

This command modifies your terminal session, prepending the virtual environment’s binary directory to your system’s PATH. After activation, the terminal prompt will usually change to indicate that the virtual environment is active.

On Windows:

If you are working on a Windows system, the command to activate the virtual environment is slightly different:

.\.venv\Scripts\activate

Again, this will modify your terminal session to use the virtual environment’s Python interpreter and libraries.

Configure Poetry to Use the Virtual Environment

Poetry is a dependency manager and build tool for Python projects, and it plays a crucial role in managing the dependencies of your C-extension module. By default, Poetry creates virtual environments outside the project directory, which can lead to confusion when managing multiple projects.

To ensure that Poetry uses the virtual environment created in your project directory, you can configure it by running:

poetry config virtualenvs.in-project true

This command tells Poetry to always create the virtual environment inside the project directory. This makes it easier to manage your dependencies and ensures that everything related to your project is self-contained within the project folder.

Why Use Poetry with a Virtual Environment?

Using Poetry in conjunction with a virtual environment provides a robust setup for developing C-extension modules. Poetry’s lock file ensures that your dependencies are pinned to specific versions, which is critical for maintaining consistency across different development environments. Additionally, by configuring Poetry to use an in-project virtual environment, you streamline the development process, making it easier to share your project with collaborators and deploy it on different systems.

Explanation of create_venv.sh

This Bash script is designed to streamline the process of creating a Python virtual environment using a specific Python version managed by pyenv. It is particularly useful for developers who want to ensure that their development environment is consistent with a specific Python version, either the exact version or the latest patch version within a given major.minor version series.

Below, I will break down the script step-by-step and provide an extensive elaboration on its functionality and purpose.

Script Overview

The script performs the following primary tasks:

  1. Install a specified Python version using pyenv, if it’s not already installed.
  2. Determine the most appropriate Python version to use based on the user’s input.
  3. Create a virtual environment using the specified or determined Python version.
  4. Handle potential conflicts, such as an existing virtual environment, by prompting the user for action.
  5. This script is highly customizable and user-friendly, with prompts to guide the user through each step, making it a powerful tool for managing Python environments in complex development workflows. By using this script, developers can ensure that their Python environment aligns with project requirements, minimizing compatibility issues and streamlining the setup process for new or existing projects.

Detailed Breakdown of the Script

1. Handling Input and Providing Usage Instructions

if [ "$#" -ne 1 ]; then
    echo "Usage: $0 <python_version>"
    echo "Example: $0 3.8 or $0 3.8.17"
    exit 1
fi

The script begins by checking if exactly one argument (the desired Python version) is provided by the user. If not, it outputs a usage guide and exits. This ensures that the user understands how to properly use the script and avoids running it with incorrect or incomplete input.

2. Setting Up Environment Variables

PYTHON_VERSION="$1"
PYENV_ROOT="$HOME/.pyenv"

Here, the script sets the PYTHON_VERSION variable to the user-provided version and defines PYENV_ROOT to the default location where pyenv is installed. Adjust this path if your pyenv installation is located elsewhere.

3. Checking for an Exact Version Match

EXACT_VERSION_DIR="$PYENV_ROOT/versions/$PYTHON_VERSION"

if [ -d "$EXACT_VERSION_DIR" ]; then
    LARGEST_VERSION="$PYTHON_VERSION"

The script checks whether the exact Python version specified by the user is already installed by pyenv. If it finds a matching directory, it sets LARGEST_VERSION to the user-specified version, indicating that this version will be used to create the virtual environment.

4. Finding the Largest Version in a Major.Minor Series

for version in "$PYENV_ROOT/versions/$PYTHON_VERSION."*; do
    if [ -d "$version" ]; then
        version_number="${version##*/}"
        if [[ -z "$LARGEST_VERSION" || "$version_number" > "$LARGEST_VERSION" ]]; then
            LARGEST_VERSION="$version_number"
        fi
    fi
done

If the exact version is not found, the script searches for the highest patch version within the specified major.minor series (e.g., 3.8.x). This ensures that the latest bug fixes and security patches within that series are used, which is often the recommended practice.

5. Prompting to Install a Missing Version

if [ -z "$LARGEST_VERSION" ]; then
    read -p "Python version '$PYTHON_VERSION' not found. Do you want to install it using pyenv? (y/n): " response
    if [[ "$response" == "y" || "$response" == "Y" ]]; then
        install_python_version
        exec "$0" "$PYTHON_VERSION"
        exit 0
    else
        echo "Exiting without installing Python version '$PYTHON_VERSION'."
        exit 1
    fi
fi

If no matching version is found, the script prompts the user to install the specified version using pyenv. If the user agrees, the script installs the version and then restarts itself to ensure that the new version is used. If the user declines, the script exits without making any changes.

6. Handling Existing Virtual Environments

if [ -d ".venv" ]; then
    read -p ".venv folder already exists. Do you want to override it? (y/n): " response
    if [[ "$response" != "y" && "$response" != "Y" ]]; then
        echo "Exiting without creating a new virtual environment."
        exit 1
    fi
    rm -rf .venv
fi

Before creating a new virtual environment, the script checks if a .venv folder already exists. If it does, the user is asked whether they want to override it. If the user chooses not to override the existing virtual environment, the script exits. Otherwise, it deletes the old .venv directory to make way for the new environment.

7. Creating the Virtual Environment

PYTHON_EXEC="$PYENV_ROOT/versions/$LARGEST_VERSION/bin/python"

if [ -x "$PYTHON_EXEC" ]; then
    echo "Creating virtual environment using $PYTHON_EXEC..."
    "$PYTHON_EXEC" -m venv .venv
    echo "Virtual environment created at $(pwd)/.venv"
    echo "You can activate it by running the command: \`source ./.venv/bin/activate\`"
    exit 0
else
    echo "Error: Python executable not found in $PYENV_ROOT/versions/$LARGEST_VERSION/bin/"
    exit 1
fi

Finally, the script uses the determined or specified Python version to create the virtual environment. It checks if the Python executable is present and, if so, creates the .venv directory using the venv module. The script then provides instructions on how to activate the newly created virtual environment.