Leverage cuda-python for GPU detection

[h-vetinari]

After the 3.0.0, I tried to redo conda-forge/scs-feedstock#21, but the problems with running the test suite remain. In particular, the GPU builds segfault when there's no GPU hardware (as happens in the conda-forge CI).

Very recently, the new python-wrappers for cuda from NVIDIA reached general availability, and this would presumably be an excellent tool to use to determine dynamically whether the GPU can actually be used.

@bodono, what do you think about adding a check (possibly conditional on its availability) that the GPU tests are only run in the drivers & GPU can be found?

[bodono]

I think it's easier to do this in the C code rather than python. I have created this PR which should make SCS fail cleanly if there is no gpu availble: https://github.com/cvxgrp/scs/pull/181/files

Are you able to patch this in and test?

[h-vetinari]

Are you able to patch this in and test?

Sorry for the delayed response. I applied the patch in conda-forge/scs-feedstock#21, but the test suite still segfaults (both on linux & on windows)...

[bodono]

Are we sure it's to do with the gpu? Just looking at this:

export PREFIX=/home/conda/feedstock_root/build_artifacts/scs-split_1636715761516/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh
export SRC_DIR=/home/conda/feedstock_root/build_artifacts/scs-split_1636715761516/test_tmp
import: 'scs'
import: '_scs_direct'
import: '_scs_indirect'
import: '_scs_direct'
import: '_scs_indirect'
import: 'scs'
+ pytest test/ -v
============================= test session starts ==============================
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- $PREFIX/bin/python
cachedir: .pytest_cache
rootdir: $SRC_DIR
Fatal Python error: Segmentation fault

I don't see import _scs_gpu.

[h-vetinari]

I don't see import _scs_gpu.

That's because it's not part of the test recipe

    test:
      imports:
        - scs
        - _scs_direct
        - _scs_indirect
      requires:
        - pytest
      source_files:
        - test/
      commands:
        - pytest test/ -v

However, I'm sure it has to do with the GPU build/code paths somehow, because the test suite for the CPU version passes.

[bodono]

This is very strange, I don't understand how just building the gpu version could break like this. Is it all platforms (linux, mac, windows)?

[bodono]

For context, each of the _scs_direct, _scs_indirect, _scs_gpu are totally independent packages.

[h-vetinari]

This is very strange, I don't understand how just building the gpu version could break like this. Is it all platforms (linux, mac, windows)?

There are not GPU builds for mac in conda-forge, but for linux & windows, the GPU builds are broken when trying to run the test suite (the imports work fine), while everything runs through for the CPU builds.

To a degree (which this issue is about), this is to be expected, because the Azure CI that conda-forge uses does not have actual GPUs. So at runtime, if a GPU-enabled package tries to access a GPU that is not there, things fail. Hence the desire to add device detection so that GPU builds don't crash if there's no physical hardware.

[bodono]

That's what I'm confused by, the _scs_direct and _scs_indirect packages should never access the gpu since they are completely independent binaries, even if they were built at the same time as _scs_gpu. These segfaults are very strange. I added the check that will fail cleanly if trying to run on gpu and none is available. However, even before that change when there was no GPU available it would also fail reasonably (ie without a segfault).

Previous behavior with no GPU:

------------------------------------------------------------------
               SCS v3.0.0 - Splitting Conic Solver
        (c) Brendan O'Donoghue, Stanford University, 2012
------------------------------------------------------------------
problem:  variables n: 2, constraints m: 4
cones:    l: linear vars: 4
settings: eps_abs: 1.0e-06, eps_rel: 1.0e-06, eps_infeas: 1.0e-09
          alpha: 1.50, scale: 1.00e-01, adaptive_scale: 1
          max_iters: 100000, normalize: 1, warm_start: 0
          acceleration_lookback: 10, acceleration_interval: 10
lin-sys:  sparse-indirect GPU
          nnz(A): 4, nnz(P): 2
 ** On entry to cusparseCreateCsr() parameter number 5 (csrRowOffsets) had an illegal value: null pointer

 ** On entry to cusparseCreateCsr() parameter number 5 (csrRowOffsets) had an illegal value: null pointer

 ** On entry to cusparseCreateDnVec() parameter number 3 (values) had an illegal value: null pointer

 ** On entry to cusparseCreateDnVec() parameter number 3 (values) had an illegal value: null pointer

 ** On entry to cusparseCreateDnVec() parameter number 3 (values) had an illegal value: null pointer

 ** On entry to cusparseCreateCsr() parameter number 6 (csrColInd) had an illegal value: null pointer

linsys/gpu/indirect/private.c:357:scs_init_lin_sys_work
ERROR_CUDA (*): no CUDA-capable device is detected
ERROR: init_lin_sys_work failure
Failure:could not initialize work
**********************************************************

New behavior with no gpu:

------------------------------------------------------------------
               SCS v3.0.0 - Splitting Conic Solver
        (c) Brendan O'Donoghue, Stanford University, 2012
------------------------------------------------------------------
problem:  variables n: 2, constraints m: 4
cones:    l: linear vars: 4
settings: eps_abs: 1.0e-06, eps_rel: 1.0e-06, eps_infeas: 1.0e-09
          alpha: 1.50, scale: 1.00e-01, adaptive_scale: 1
          max_iters: 100000, normalize: 1, warm_start: 0
          acceleration_lookback: 10, acceleration_interval: 10
lin-sys:  sparse-indirect GPU
          nnz(A): 4, nnz(P): 2
cudaError: 100 (100 indicates no device)
ERROR: init_lin_sys_work failure
Failure:could not initialize work
**********************************************************

[h-vetinari]

That's what I'm confused by, the _scs_direct and _scs_indirect packages should never access the gpu

Do you mean the package imports here? As I said above, the imports work (for the GPU builds, even on an agent without a GPU), but the test suite fails.

In any case, great to hear that the failure should now be more gracious! I'm guessing these changes haven't made it to the repo(s) yet?

[bodono]

By test suite do you mean running out/run_tests_gpu_indirect is what is failing?

It should never seg fault with or without a gpu (even before the latest change to make the failing more graceful), it's very strange and it makes me thing something else weird is going on.

[h-vetinari]

With testsuite I mean running the equivalent** of pytest -v test/

** slight adaptation because the test-folder is not packaged in the same way as the package itself, but for basically all intents and purposes it should be the same as running the tests in the source tree.

[bodono]

Ok I understand now, that does `import _scs_gpu'. Still, there shouldn't be a seg fault even without a gpu so I'm not sure what's going on here.

[h-vetinari]

Ok I understand now, that does `import _scs_gpu'

OK cool, glad we're on the same page now

Still, there shouldn't be a seg fault even without a gpu so I'm not sure what's going on here.

I still have artefact persistence switched on in conda-forge/scs-feedstock#21. You could try again to download an appropriate artefact, unpack it, and then install it into an environment. If we can get past the resolver errors this time, then you could have a closer look at what's happening... 🙃

[h-vetinari]

[...] and then install it into an environment

to recall;

• use conda info to see which cuda version is detected on your system (anything higher than 11.2 works with the 11.2 artefact)
• download the artefact with appropriate platform / cuda version / python version (e.g. 3.8)
• unpack the artefact until you get to the first folder that contains channeldata.json
• conda create -n test_env -c "path/to/said/folder" -c conda-forge python=3.8 scs
• conda activate test_env
• etc.

[h-vetinari]

@bodono, I've tried again for 3.1.0, and import _scs_gpu still segfaults hard on both linux and windows in the absence of a GPU.

Could we give it another shot with you installing one of the artefacts? I think the setup has hopefully improved enough now that you should be able to install it (the last CI run on that PR has green CI because I switched off the failing test suite so that the artefacts are more easily installable) - the instructions in the previous comment remain correct.