Referencing requirements.txt for install_requires in setuptools setup.py

Fundamental Differences in Dependency Management

In Python project development, while both requirements.txt and setup.py files involve dependency management, they serve fundamentally different design purposes and application scenarios. The install_requires parameter in setup.py aims to declare the minimum compatible dependency version ranges required for package operation, providing flexible installation options for package users. In contrast, requirements.txt serves as a deployment manifest, precisely specifying all packages and their exact versions needed in a particular environment to ensure deployment consistency and repeatability.

Reverse Dependency Declaration Strategy

To avoid duplication in dependency declarations, a reverse strategy can be employed: completely define dependencies in setup.py while including only a single dot character . in requirements.txt. This approach triggers installation of the current directory package via pip install -r requirements.txt, automatically resolving dependencies defined in setup.py to achieve unified dependency management.

requirements.txt Parsing Implementation

Although directly parsing requirements.txt files is not recommended, it can be achieved through pip's internal API in specific scenarios. Here's an implementation example based on pip 9.0.1:

from pip._internal.req import parse_requirements
from pip._internal.network.session import PipSession

install_reqs = parse_requirements('requirements.txt', session=PipSession())
requirements = [str(ir.requirement) for ir in install_reqs]

setup(
    name='example_package',
    install_requires=requirements,
    # Other setup parameters
)

It's important to note that this method cannot handle environment markers and relies on pip's internal API, which may cause compatibility issues during version updates.

Historical Version Compatibility Solution

In versions prior to pip 6.0, public API could be used for parsing:

from pip.req import parse_requirements

install_reqs = parse_requirements(<requirements_path>)
reqs = [str(ir.req) for ir in install_reqs]

setup(
    install_requires=reqs
)

This method can handle comments and file references (-r option) in requirements.txt, but similarly carries API stability risks.

Limitations of Direct File Reading

The simple file reading approach, while straightforward to implement, has significant limitations:

import os
from setuptools import setup

with open('requirements.txt') as f:
    required = f.read().splitlines()

setup(install_requires=required)

This method cannot handle advanced features such as comment lines, line continuation characters, or environment markers, making it suitable only for the simplest dependency list scenarios.

Best Practice Recommendations

Considering the advantages and disadvantages of various solutions, the following strategy is recommended: For library projects, prioritize using setup.py to define宽松 dependency ranges; for application deployment, use pip freeze > requirements.txt to generate precise dependency manifests. In scenarios where requirements.txt parsing is necessary, thoroughly evaluate API stability risks and consider implementing custom parsing logic to handle complex situations.