Upload and download files

[onnozweers]

• Implemented --upload and --download
• Uploads and downloads are done via a WebDAV door. Ada usually only knows about the dCache API door. Ada has several methods to find out which WebDAV door to use, and will try each method until a door is found.
• If no (usable) WebDAV door can be found, the WebDAV door may be provided as a prefix of the remote file (https://webdav.example.com/remotepath/remotefile)
• The target can be a full name (/targetdir/file) or only a directory name (/targetdir). Ada will check this. If the target is a dir, the target filename wil be the same as the source filename.
• Before downloading, Ada checks whether the file is online. If the file is on tape but not online, refuse to download (because it would take forever) and instead, provide the command that can be used to stage (restore) the file from tape.
• With --verify-checksum, compare the checksum of the transferred file with the source file. During upload, the local file's MD5 checksum is sent as a header in the transfer, so that dCache can do the verification. The MD5 checksum is stored in dCache's database, along with the default (Adler32) checksum. In case of MD5 mismatch, the uploaded (corrupt) file is cleaned up.
• If the target file exists, quit. Except when --verify-checksum is specified: in that case, compare the checksums of source and target.
• Refuse to follow redirects from HTTPS to HTTP (some dCache WebDAV doors might be configured to do so), because it would be unsafe. Except when --allow-insecure-redirects is used.

[onnozweers]

Some examples:

Uploading

% username=onno

% ada/ada --netrc --upload ~/test.txt /users/$username/disk/test.txt
Uploading...

Uploading but target exists

% ada/ada --netrc --upload ~/test.txt /users/$username/disk/test.txt                
ERROR: target '/users/onno/disk/test.txt' is an existing file. Ada doesn't support overwriting a file.

% ada/ada --netrc --upload ~/test.txt /users/$username/disk/        
Target '/users/onno/disk/test.txt' already exists. Ada does not support overwriting.

Uploading with checksum verification

% ada/ada --netrc --upload ~/test.txt /users/$username/disk/test2.txt --verify-checksum
Uploading with checksum verification...

Uploading with checksum verification has the side effect that the uploaded file has both an Adler32 (default in dCache) and an MD5 checksum.

% ada/ada --netrc --checksum /users/$username/disk/test2.txt
/users/onno/disk/test2.txt  ADLER32=062801cb  MD5_TYPE=d8e8fca2dc0f896fd7cb4cb0031ba249

Uploading with checksum verification and target exists

% ada/ada --netrc --upload ~/test.txt /users/$username/disk/test2.txt --verify-checksum
Target (remote) file '/users/onno/disk/test.txt' already exists. Verifying checksum...
Checksum OK.

Downloading

% ada/ada --netrc --download /users/$username/disk/test.txt ~/testfile
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100     5  100     5    0     0     67      0 --:--:-- --:--:-- --:--:--    67

Downloading but local file exists

% ada/ada --netrc --download /users/$username/disk/test.txt ~/testfile
ERROR: Target file '/Users/zweer001/testfile' already exists.

Downloading with checksum verification

First a checksum failure because the target file exists but it's a different file. Then removing it and trying again.

# Local target exists but it's a different file: triggers a checksum error
% ada/ada --netrc --download /users/onno/disk/test.txt ~/testfile --verify-checksum
Target (local) file '/Users/mylocalhomedir/testfile' already exists. Verifying checksum...
ERROR: Checksum mismatch!
Remote checksum: 062801cb
Local checksum : 5120053d

# Clean up local file
% rm ~/testfile

# Trying again
% ada/ada --netrc --download /users/onno/disk/test.txt ~/testfile --verify-checksum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100     5  100     5    0     0     35      0 --:--:-- --:--:-- --:--:--    35
Checksum OK.

# And again. Now the target exists. The checksum check confirms it's the same file.
% ada/ada --netrc --download /users/onno/disk/test.txt ~/testfile --verify-checksum
Target (local) file '/Users/mylocalhomedir/testfile' already exists. Verifying checksum...
Checksum OK.

Downloading, but remote file is not online

% ada/ada --netrc --longlist /users/$username/tape/test-badger14-12         
/users/onno/tape/test-badger14-12  12  2025-03-13 14:07 UTC  tape  NEARLINE

% ada/ada --netrc --download /users/$username/tape/test-badger14-12 ~/test-download-tapefile          
ERROR: file '/users/onno/tape/test-badger14-12' exists, but it is on tape, and not online. You need to stage it first before you can download it. You can do that with this command:

   ada/ada --stage /users/onno/tape/test-badger14-12 --netrc  

Please note, that staging works most efficiently when done in bulk.