Installing Error

[Doruk-Dilmen]

Hi,

I tried to install Tao Toolkit from this link: https://docs.nvidia.com/tao/tao-toolkit/text/tao_toolkit_quick_start_guide.html

When I tried this command, I get error:

Error like this:

I should use Tao Toolkit but if I couldn't, only Retail dataset and pretrained model can be enough .I just want to use retail pretrained model. I need this model because my school project depends this dataset. Can you help me? I need retail dataset and pretrained model. Can you share with me?

[iamamiramine]

First install nvidia-container-toolkit following the instructions from nvidia docs

Then update the setup/quickstart_launcher.sh script starting from info "Checking nvidia-docker2 installation" (line 114)
change the following script:

    info "Checking nvidia-docker2 installation"
    if command -v nvidia-docker >/dev/null; then
        nvidia_docker_version=$(nvidia-docker version | grep "NVIDIA Docker:" | cut -d " " -f 3)
        [[ $nvidia_docker_version = 2* ]] || warning "nvidia-docker2 is required, current version is $nvidia_docker_version"
    else
        error "nvidia-docker not found."
    fi

to

    info "Checking nvidia-docker2 installation"
    if command -v docker >/dev/null; then
        if docker info --format '{{json .Runtimes}}' | grep -q nvidia; then
            echo "NVIDIA Container Toolkit is properly installed."
        else
            warning "NVIDIA Container Toolkit is not detected in Docker runtimes."
        fi
        
        # Check Docker version
        docker_version=$(docker version --format '{{.Server.Version}}')
        echo "Docker version: $docker_version"
        
        # Check NVIDIA Container Toolkit version
        nvidia_toolkit_version=$(dpkg -l | grep nvidia-container-toolkit | awk '{print $3}')
        if [ -n "$nvidia_toolkit_version" ]; then
            echo "NVIDIA Container Toolkit version: $nvidia_toolkit_version"
        else
            error "NVIDIA Container Toolkit not found."
        fi
    else
        error "Docker not found."
    fi

[HappySamuel]

Hi @iamamiramine

I am also facing the same issue. I tried your solution, it makes the error msg gone. But i still facing another issue when performing this command.

!tao model centerpose train \
          -e $SPECS_DIR/train.yaml \
          results_dir=$RESULTS_DIR/

The issue is regarding No supported gpu backend found shown as below:

  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_pytorch/core/decorators/workflow.py", line 69, in _func
    raise e
  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_pytorch/core/decorators/workflow.py", line 48, in _func
    runner(cfg, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_pytorch/cv/centerpose/scripts/train.py", line 84, in main
    run_experiment(experiment_config=cfg,
  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_pytorch/cv/centerpose/scripts/train.py", line 56, in run_experiment
    trainer = Trainer(logger=ptl_loggers,
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/argparse.py", line 70, in insert_env_defaults
    return fn(self, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 400, in __init__
    self._accelerator_connector = _AcceleratorConnector(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 147, in __init__
    self._accelerator_flag = self._choose_gpu_accelerator_backend()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py", line 367, in _choose_gpu_accelerator_backend
    raise MisconfigurationException("No supported gpu backend found!")
lightning_fabric.utilities.exceptions.MisconfigurationException: No supported gpu backend found!

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
[2025-01-17 07:50:47,256 - TAO Toolkit - root - WARNING] Telemetry data couldn't be sent, but the command ran successfully.
[2025-01-17 07:50:47,256 - TAO Toolkit - root - WARNING] [Error]: Unknown Error
[2025-01-17 07:50:47,256 - TAO Toolkit - root - WARNING] Execution status: FAIL
2025-01-17 15:50:48,113 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.

What shall i do to resolve this ?

[iamamiramine]

Did you properly install NVIDIA Container Toolkit? Also are your NVIDIA drivers properly setup? You can always make sure by running nvidia-smi to check your NVIDIA drivers. You can also check if CUDA is available using the python command torch.cuda.is_available()

If all the above is properly setup, you can run the bash command bash setup/quickstart_launcher.sh --upgrade to troubleshoot.

Let me know if you performed any of these checks.

Installation guide: TAO Toolkit Quick Start Guide

[HappySamuel]

Hi @iamamiramine

Yes, i have nvidia drivers installed and working on the host, and it's showing as below:

$ nvidia-smi
Sat Jan 18 09:24:34 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2050        Off | 00000000:01:00.0 Off |                  N/A |
| N/A   43C    P5               4W /  40W |    559MiB /  4096MiB |     14%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1386      G   /usr/lib/xorg/Xorg                          223MiB |
|    0   N/A  N/A      1595      G   /usr/bin/gnome-shell                         94MiB |
|    0   N/A  N/A      5583      G   ...29,262144 --variations-seed-version      235MiB |
+---------------------------------------------------------------------------------------+

However, when run the docker container with command nvidia-smi, it shown Failed to initialize NVML: Unknown Error
Anyway to troubleshoot this ?

I looked into the link you sent me, but i can't download the file. Not sure why it is. I checked my internet, everything is working fine. Do you have this file that can be shared with me ?
wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/tao/tao-getting-started/versions/5.3.0/zip -O getting_started_v5.3.0.zip

[iamamiramine]

I believe NVIDIA Container Toolkit is not properly setup.
Can you install the toolkit following this guide Installing the NVIDIA Container Toolkit and let me know if it works.

Regarding the tao-getting-started you can clone it from the original repository tao_tutorials

[HappySamuel]

Shall i using follow v5.3.0 or v5.5.0 ? As i saw that v5.3.0 is indicating Ubuntu20.04, while v5.5.0 is indicating Ubuntu22.04. And my host is currently Ubuntu20.04

btw, i have tried again and again on the reinstalling Nvidia container toolkit. But the result still the same showing Failed to initialize NVML: Unknown Error when running nvidia-smi inside the docker container.

[iamamiramine]

If so then it is preferable you install the v5.3.0

Could you please show me what container you are using? What steps are you following to create the docker container and when and how are you accessing the container?

[HappySamuel]

Currently the issue Failed to initialize NVML: Unknown Error when running nvidia-smi inside the docker container already resolved. It can be solved by adding --privileged to the docker run command.

[HappySamuel]

However i faced another issue when running centerpose train, and have no idea why is this happening.

Train results will be saved at: /workspace/results/train
No pretrained configuration specified for convnext_base_in22k model. Using a default. Please add a config to the model pretrained_cfg registry or pass explicitly.
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Loaded pretrained weights from /workspace/results/pretrained_models/pretrained_fan_classification_nvimagenet_vfan_small_hybrid_nvimagenet/fan_small_hybrid_nvimagenet.pth
_IncompatibleKeys(missing_keys=['out_norm0.weight', 'out_norm0.bias', 'out_norm1.weight', 'out_norm1.bias', 'out_norm2.weight', 'out_norm2.bias', 'out_norm3.weight', 'out_norm3.bias', 'learnable_downsample.weight', 'learnable_downsample.bias'], unexpected_keys=['norm.weight', 'norm.bias', 'head.fc.weight', 'head.fc.bias'])Initializing train toilet data.
Loaded train 23 samples
Initializing test toilet data.
Loaded test 7 samples
/usr/local/lib/python3.10/dist-packages/pytorch_lightning/callbacks/model_checkpoint.py:652: Checkpoint directory /workspace/results/train exists and is not empty.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]/usr/local/lib/python3.10/dist-packages/torch/optim/lr_scheduler.py:28: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate.
  warnings.warn("The verbose parameter is deprecated. Please use get_last_lr() "

  | Name              | Type             | Params
-------------------------------------------------------
0 | loss              | ObjectPoseLoss   | 0     
1 | model             | CenterPose_FAN   | 33.5 M
2 | hm_decoder        | HeatmapDecoder   | 0     
3 | transform_outputs | TransformOutputs | 0     
4 | merge_output      | MergeOutput      | 0     
5 | pnp_process       | PnPProcess       | 0     
-------------------------------------------------------
33.5 M    Trainable params
0         Non-trainable params
33.5 M    Total params
133.854   Total estimated model params size (MB)

Sanity Checking: |          | 0/? [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/data_connector.py:441: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=15` in the `DataLoader` to improveError executing job with overrides: []Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_pytorch/core/decorators/workflow.py", line 69, in _func
    raise e
  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_pytorch/core/decorators/workflow.py", line 48, in _func
    runner(cfg, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_pytorch/cv/centerpose/scripts/train.py", line 84, in main
    run_experiment(experiment_config=cfg,
  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_pytorch/cv/centerpose/scripts/train.py", line 70, in run_experiment
    trainer.fit(pt_model, dm, ckpt_path=resume_ckpt)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit
    call._call_and_handle_interrupt(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 986, in _run
    results = self._run_stage()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1030, in _run_stage
    self._run_sanity_check()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1059, in _run_sanity_check
    val_loop.run()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/utilities.py", line 182, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/evaluation_loop.py", line 128, in run
    batch, batch_idx, dataloader_idx = next(data_fetcher)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/fetchers.py", line 133, in __next__
    batch = super().__next__()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/fetchers.py", line 60, in __next__
    batch = next(self.iterator)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/combined_loader.py", line 341, in __next__
    out = next(self._iterator)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/combined_loader.py", line 142, in __next__
out = next(self.iterators[0])  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 631, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 675, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_pytorch/cv/centerpose/dataloader/data_feeder.py", line 362, in __getitem__
pts[j, :2] = affine_transform(pts[j, :2], trans_output_rot)IndexError: index 7 is out of bounds for axis 0 with size 7

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

[shash-ank]

Hello even though I am inside the TAO toolkit tensorflow I am still having the issue of decrypting a model with tao

 docker run -it --rm -v /home/models:/workspace/model_dir -p 8888:8888 --runtime=nvidia nvcr.io/nvidia/tao/tao-toolkit:4.0.1-tf1.15.5 /bin/bash

==============================
=== TAO Toolkit TensorFlow ===
==============================

NVIDIA Release 4.0.1-TensorFlow (build )
TAO Toolkit Version 4.0.1

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the TAO Toolkit End User License Agreement.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/tao-toolkit-software-license-agreement

NOTE: The SHMEM allocation limit is set to the default of 64MB.  This may be
   insufficient for TAO Toolkit.  NVIDIA recommends the use of the following flags:
   docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 ...

root@5a:/workspace# ls
EULA.pdf  NVIDIA_Deep_Learning_Container_License.pdf  README.md  model_dir
root@5a:/workspace# cd model_dir/
root@5a:/workspace/model_dir# ls
gazenet_vdeployable_v1.0
root@5a:/workspace/model_dir# cd gazenet_vdeployable_v1.0/
root@5a:/workspace/model_dir/gazenet_vdeployable_v1.0# ls
model.etlt
root@5a:/workspace/model_dir/gazenet_vdeployable_v1.0#

whenever I run

root@5a:/# tao deploy gazenet gen_trt_engine \  -m /workspace/model_dir/gazenet_deployable_v1.0/model.etlt 
  -k nvidia_tlt 
  --data_type fp16 
  --engine_file /workspace/model_dir/gazenet_deployable_v1.0/gazenet_engine.engine

bash: tao: command not found

I can't understand why this is happening