No pdb files generated in release mode

[aboelens]

In the process of getting openssl to build with the MSVC toolchain I ran into a problem where there are no pdb files produced in release mode:

microsoft/vcpkg#41796 (comment)

Would you agree with this statement from one of the maintainers I have been communicating with that this should be enabled in the toolchain?

"/DEBUG /OPT:REF is expected to come from the cmake toolchain.
From the gmp port (error logs), I know this value:

set(VCPKG_DETECTED_CMAKE_SHARED_LINKER_FLAGS_RELEASE "-machine:x64 -nologo -DEBUG -INCREMENTAL:NO -OPT:REF -OPT:ICF")
So -OPT:REF -DEBUG is already there."

[MarkSchofield]

CMake "Release" builds typically don't have PDB files, "RelWithDebInfo" builds do. That's configured by (I think) the logic in Modules/Platform/Windows-MSVC.cmake, which sets (for example) CMAKE_EXE_LINKER_FLAGS_<CONFIG>_INIT. So the default behavior comes from CMake, not a toolchain.

Since VCPkg typically just builds two configurations 'dbg' and rel', then it's not unreasonable for the 'rel' builds to have PDB's incase they're consumed from 'RelWithDebInfo' builds, but I'd have thought that that would be something that VCPkg would handle in, say, vcpkg_cmake_configure.

TBH, I'm not sure what you're trying to do. If you're just building openssl using 'WindowsToolchain', I'm not sure why you'd file a vcpkg issue...? If you're building openssl as a VCPkg port, then VCPkg hooks-up it's own toolchain that (IIRC) depends on environment variables that vcpkg.exe sets (much like a Visual Studio command prompt), so you shouldn't be using WindowsToolchain?

[aboelens]

I am trying to build the vcpkg port of openssl using the VCPKG_CHAINLOAD_TOOLCHAIN_FILE option in the triplet file. That way both my code and its dependencies are build with the same SDK and compiler versions.

[MarkSchofield]

Rather than switching Vcpkg to use WindowsToolchain, can you use properties that WindowsToolchain has set to influence the behavior of the Vcpkg toolchain...? That seems like a slightly less intrusive approach.

[aboelens]

I like the idea, but that is generally difficult. To provide consistent builds, vcpkg sets up a clean build environment for every port by default.

[MarkSchofield]

Sorry - I've completely neglected this. Just taking a look now. Writing things down here, for comments and notes...

So, I'm starting from the documentation on VCPKG_CHAINLOAD_TOOLCHAIN_FILE in triplet files. By setting VCPKG_CHAINLOAD_TOOLCHAIN_FILE, the specified toolchain "will override all other compiler detection logic". So the 'chainload' part is a bit of a misnomer. The toolchain is expected to map the other triplet variables - for example VCPKG_CXX_FLAGS - to CMake constructs, based on the precedent from the 'default' toolchain files that ship with vcpkg - for example scripts/toolchains/windows.cmake. The documentation suggests including an existing toolchain:

To create a custom toolchain file, it's best to start by including an existing toolchain from ${VCPKG_ROOT}/scripts/toolchains and extend it.

The existing scripts/toolchains/windows.cmake sets:

set(CMAKE_MODULE_LINKER_FLAGS_RELEASE "/nologo /DEBUG /INCREMENTAL:NO /OPT:REF /OPT:ICF ${VCPKG_LINKER_FLAGS} ${VCPKG_LINKER_FLAGS_RELEASE}" CACHE STRING "")
set(CMAKE_SHARED_LINKER_FLAGS_RELEASE "/nologo /DEBUG /INCREMENTAL:NO /OPT:REF /OPT:ICF ${VCPKG_LINKER_FLAGS} ${VCPKG_LINKER_FLAGS_RELEASE}" CACHE STRING "")
set(CMAKE_EXE_LINKER_FLAGS_RELEASE "/nologo /DEBUG /INCREMENTAL:NO /OPT:REF /OPT:ICF ${VCPKG_LINKER_FLAGS} ${VCPKG_LINKER_FLAGS_RELEASE}" CACHE STRING "")

Notice that the options pass "/DEBUG" to generate debug information.

So, to answer your question:

Would you agree with this statement from one of the maintainers I have been communicating with that this should be enabled in the toolchain?

Well, I don't think it should be enabled in a generic toolchain, but it appears as though Vcpkg has the expectation that a toolchain that it uses should set it. So it should probably be set. I'd rather not do that in, say, Windows.MSVC.toolchain.cmake (since that's a 'generic toolchain'), but we could introduce, say Windows.Vcpkg.MSVC.toolchain.cmake that is (effectively):

list(APPEND CMAKE_TRY_COMPILE_PLATFORM_VARIABLES _VCPKG_ROOT_DIR)

include(${_VCPKG_ROOT_DIR}/scripts/toolchains/windows.cmake)
include(${CMAKE_CURRENT_LIST_DIR}/Windows.MSVC.toolchain.cmake)

Note: I couldn't use ${VCPKG_ROOT}/scripts/toolchains as the documentation recommends, because it isn't set - only _VCPKG_ROOT_DIR and Z_VCPKG_ROOT_DIR are.

And that gets us a step further - .pdb files are generated on the 'Release' builds. I'm using the 'sqlite3' port to verify (since it's a bit easier than the openssl port), and I see .pdb files generated in the buildtrees\sqlite3\x64-windows-rel folder. But I still see warnings:

CMake Warning at scripts/cmake/vcpkg_copy_pdbs.cmake:48 (message):
  Could not find a matching pdb file for:

      W:/scratch/Build/vcpkg/packages/sqlite3_x64-windows/bin/sqlite3.dll
      W:/scratch/Build/vcpkg/packages/sqlite3_x64-windows/debug/bin/sqlite3.dll

And that appears to be down to vcpkg_copy_pdbs.cmake making assumptions. In vcpkg_copy_pdbs.cmake it looks for '.dll' files and then runs 'dumpbin' to try and identify the path to the .pdb files:

            execute_process(COMMAND dumpbin /PDBPATH "${dll}"
                            COMMAND findstr PDB
                OUTPUT_VARIABLE pdb_line
                ERROR_QUIET
                RESULT_VARIABLE error_code
            )

The script is just using 'dumpbin' for the command to run. When a VCPKG_CHAINLOAD_TOOLCHAIN_FILE value isn't specified, then VCPkg will set environment variables (including the PATH) so that 'dumpbin' is found. But when VCPKG_CHAINLOAD_TOOLCHAIN_FILE is specified, then dumpbin won't be found. And if 'dumpbin' isn't found, symbols aren't copied and warnings are logged.

Problems:

1. vcpkg_copy_pdbs.cmake shouldn't be running 'dumpbin' directly. It should be calling find_program to find 'dumpbin', so that the various methods that CMake has for finding tooling can be leveraged.
2. 'Windows.MSVC.toolchain.cmake' already is (as of Include the VS toolset and Win32 SDK binary paths in the CMAKE_PROGRAM_PATH #119) setting CMAKE_PROGRAM_PATH so that Windows SDK and Visual Studio tooling can be found through calls to find_program. But those updates to CMAKE_PROGRAM_PATH made by the toolchain aren't present when vcpkg_copy_pdbs.cmake is invoked.

Even with (1) fixed, I can't find a way to propagate find_program-context into the vcpkg_copy_pdbs.cmake state.

To some extent, these problems appear to be why the openssl port doesn't find nmake.exe - in that case the openssl port is using vcpkg_build_nmake - which correctly uses find_program(NMAKE nmake REQUIRED) - but that would work with nmake in the path. If VCPKG_CHAINLOAD_TOOLCHAIN_FILE is set, and the environment not updated, I don't see how a specified toolchain would be able to configure CMake so that find_program is successful.