Psycopg2 keeps failing to build

[marvelous-benji]

• opencdms version: 0.10
• Python version: 3.8
• Operating System: Kali Linux( Version 2021)

Description

I was trying to install pyscopg2 which is a database driver for posgresql.
I did notice the package is available, as from the stack trace pip was able to download the package,
but it does fails whenever it tries to build the package as it has some missing dependencies like: libpq-fe.h.

Running the following command:
pip install psycopg2

fails with the following error stack trace:

Defaulting to user installation because normal site-packages is not writeable
Collecting psycopg2
  Using cached psycopg2-2.9.3.tar.gz (380 kB)
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-r88tgur4/psycopg2_14d63bb466404a78826008b2da45b021/setup.py'"'"'; __file__='"'"'/tmp/pip-install-r88tgur4/psycopg2_14d63bb466404a78826008b2da45b021/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-679zt9_t
       cwd: /tmp/pip-install-r88tgur4/psycopg2_14d63bb466404a78826008b2da45b021/
  Complete output (38 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/psycopg2
  copying lib/errors.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/pool.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/__init__.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/sql.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/_json.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/tz.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/extras.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/_ipaddress.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/_range.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/extensions.py -> build/lib.linux-x86_64-3.8/psycopg2
  copying lib/errorcodes.py -> build/lib.linux-x86_64-3.8/psycopg2
  running build_ext
  building 'psycopg2._psycopg' extension
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/psycopg
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSYCOPG_VERSION=2.9.3 (dt dec pq3 ext lo64) -DPSYCOPG_DEBUG=1 -DPG_VERSION_NUM=140000 -DHAVE_LO64=1 -DPSYCOPG_DEBUG=1 -I/usr/local/include/python3.8 -I. -I/usr/include/postgresql -I/usr/include/postgresql/14/server -I/usr/include/libxml2 -c psycopg/psycopgmodule.c -o build/temp.linux-x86_64-3.8/psycopg/psycopgmodule.o -Wdeclaration-after-statement
  In file included from psycopg/psycopgmodule.c:28:
  ./psycopg/psycopg.h:36:10: fatal error: libpq-fe.h: No such file or directory
     36 | #include <libpq-fe.h>
        |          ^~~~~~~~~~~~
  compilation terminated.
  
  It appears you are missing some prerequisite to build the package from source.
  
  You may install a binary package by installing 'psycopg2-binary' from PyPI.
  If you want to install psycopg2 from source, please install the packages
  required for the build and try again.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <https://www.psycopg.org/docs/install.html>).
  
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for psycopg2
  Running setup.py clean for psycopg2
Failed to build psycopg2
Installing collected packages: psycopg2
    Running setup.py install for psycopg2 ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-r88tgur4/psycopg2_14d63bb466404a78826008b2da45b021/setup.py'"'"'; __file__='"'"'/tmp/pip-install-r88tgur4/psycopg2_14d63bb466404a78826008b2da45b021/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-4iq8smwq/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/benji/.local/include/python3.8/psycopg2
         cwd: /tmp/pip-install-r88tgur4/psycopg2_14d63bb466404a78826008b2da45b021/
    Complete output (38 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/psycopg2
    copying lib/errors.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/pool.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/__init__.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/sql.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/_json.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/tz.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/extras.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/_ipaddress.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/_range.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/extensions.py -> build/lib.linux-x86_64-3.8/psycopg2
    copying lib/errorcodes.py -> build/lib.linux-x86_64-3.8/psycopg2
    running build_ext
    building 'psycopg2._psycopg' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/psycopg
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSYCOPG_VERSION=2.9.3 (dt dec pq3 ext lo64) -DPSYCOPG_DEBUG=1 -DPG_VERSION_NUM=140000 -DHAVE_LO64=1 -DPSYCOPG_DEBUG=1 -I/usr/local/include/python3.8 -I. -I/usr/include/postgresql -I/usr/include/postgresql/14/server -I/usr/include/libxml2 -c psycopg/psycopgmodule.c -o build/temp.linux-x86_64-3.8/psycopg/psycopgmodule.o -Wdeclaration-after-statement
    In file included from psycopg/psycopgmodule.c:28:
    ./psycopg/psycopg.h:36:10: fatal error: libpq-fe.h: No such file or directory
       36 | #include <libpq-fe.h>
          |          ^~~~~~~~~~~~
    compilation terminated.
    
    It appears you are missing some prerequisite to build the package from source.
    
    You may install a binary package by installing 'psycopg2-binary' from PyPI.
    If you want to install psycopg2 from source, please install the packages
    required for the build and try again.
    
    For further information please check the 'doc/src/install.rst' file (also at
    <https://www.psycopg.org/docs/install.html>).
    
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-r88tgur4/psycopg2_14d63bb466404a78826008b2da45b021/setup.py'"'"'; __file__='"'"'/tmp/pip-install-r88tgur4/psycopg2_14d63bb466404a78826008b2da45b021/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-4iq8smwq/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/benji/.local/include/python3.8/psycopg2 Check the logs for full command output.```

[isedwards]

They provide a very useful error message:

    It appears you are missing some prerequisite to build the package from source.
    
    You may install a binary package by installing 'psycopg2-binary' from PyPI.
    If you want to install psycopg2 from source, please install the packages
    required for the build and try again.
    
    For further information please check the 'doc/src/install.rst' file (also at
    <https://www.psycopg.org/docs/install.html>).

We know:

1. The prerequisite were not present to build the package from source
2. A binary package is available
3. To install from source we need to fulfil the build dependencies

Looking first at the link they provide (https://www.psycopg.org/docs/install.html), and specifically at the psycopg-vs-psycopg-binary section, they state:

• "The psycopg2-binary package is meant for beginners to start playing with Python and PostgreSQL"
• "For production use you are advised to use the source distribution"

(they give further information on the reasons why - these make sense since they have built the binary version using a stack that is particular to their build environment which may have subtle differences to other environments that the binary package may be used in)

The psycopg2-binary docs give a similar warning: The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources.

I'm a bit reluctant to even use psycopg2-binary in development of any application for production use due to the tenth principle of 12 factor app development (Dev/prod parity) which states: "Keep development, staging, and production as similar as possible".

I think we may want to use psycopg2-binary for OpenCDMS Reference Implementation (for users who are experimenting and not directly using the code in production systems), but in general we need to make sure that psycopg2 can be built in each environment that we're supporting in production.

See the prerequisites in the psycopg2 docs for a broad overview and also Option 2 in this StackOverflow discussion for some guidance.