Fatal Error: 'python/google/protobuf/proto_api.h' Not Found During Compilation

[tadteo]

When trying to compile the project, I encountered a fatal error related to a missing header file python/google/protobuf/proto_api.h. This is causing the build to fail.

Steps to Reproduce:

Clone the repository
Navigate to /home/user/gz_ws/src/gz-python

mkdir build
cmake ..
make

Expected Behavior:

The code should compile successfully.

Actual Behavior:

The build fails with the following error:

In file included from /home/user/gz_ws/src/gz-python/build/_deps/pybind11_protobuf-src/pybind11_protobuf/native_proto_caster.h:21,
                 from /home/user/gz_ws/src/gz-python/src/pybind11/gz/msgs/_gz_msgs_extras_pybind11.cc:22:
/home/user/gz_ws/src/gz-python/build/_deps/pybind11_protobuf-src/pybind11_protobuf/check_unknown_fields.h:7:10: fatal error: python/google/protobuf/proto_api.h: No such file or directory
    7 | #include "python/google/protobuf/proto_api.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/extras.dir/build.make:76: CMakeFiles/extras.dir/src/pybind11/gz/msgs/_gz_msgs_extras_pybind11.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1243: CMakeFiles/extras.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

System Information:

OS: Ubuntu 22.04
Compiler: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

[pedrofontanatf]

Hi @tadteo , any updates on this? I'm experiencing the same error

[kyranf]

It must need to be an older version of something here to work, because things don't appear to build anymore due to pybind11 not having the header file needed by the google protocol buffer package missing files now. Things used to build, within the last 12 months, but not anymore.

I would be keen to know what combination of versions of the libraries involved are required to compile this!

relevant reading:
protocolbuffers/protobuf#9464

pybind/pybind11_protobuf#60

[srmainwaring]

This is related to an upstream issue: pybind/pybind11_protobuf#127. It was partly introduced by the change in: pybind/pybind11_protobuf#128, which uses the system version of Protobuf first if available (which does not include the proto_api.h header).

Gazebo Harmonic now include Python bindings for gz-transport using the binding code from this repo, but avoiding using pybind11_protobuf. There are a couple of options: 1. archive this repo in it's current state, or 2. update this repo to build again and maintain it as an alternative implementation for transport. To do the latter requires some changes to avoid conflicts with the binding libraries now exposed in Gazebo Harmonic.

• Use the Python bindings generated from the protobuf files in gz-msgs. This requires the PYTHONPATH to include the gz-msgs install location. This is done automatically using the projects env hooks when sourcing the Gazebo project.
• Move the gz.msgs.extras and gz.transport libraries into a different module namespaces to not conflict with the Gazebo libraries. There does not seem to be an easy way to make the two projects co-exist with the same module name because the .__init__.py file in Gazebo checks for an unversioned namespace (i.e. gz.msgs) and raises and error if found. It then aliases the unversioned module name to the versioned one.
• Update examples and documentation to use the renamed modules gz.python.msgs_extras and gz.python.transport.

[Mizux]

Since this project FetchContent Protobuf and pybind11_protobuf, you can patch Protobuf to make the proto_api.h available.

see: https://github.com/google/or-tools/blob/870edf6f7bff6b8ff0d267d936be7e331c5b8c2d/patches/protobuf-v26.1.patch#L59-L68
and https://github.com/google/or-tools/blob/870edf6f7bff6b8ff0d267d936be7e331c5b8c2d/cmake/dependencies/CMakeLists.txt#L107-L110

note: this header is public available in bazel based build (https://github.com/protocolbuffers/protobuf/blob/2124e2687d7c7cced2f4d9e759701204aa113b4b/python/build_targets.bzl#L434-L441), Protobuf Maintainers want to push for replacing it by the UPB protobuf implem (but it is not staffed and stall for 2 years already...).