From c19bf497b8848c8b170379b61f2427e414a84f86 Mon Sep 17 00:00:00 2001
From: alessio94 <alessiopizzini@gmail.com>
Date: Wed, 11 Feb 2026 15:17:46 +0100
Subject: [PATCH 1/5] Add tutorial for configuring Rucio with Docker Compose

This tutorial provides a comprehensive guide for setting up a Rucio test environment using Docker Compose, detailing prerequisites, environment management, configuration, and data management scripts for effective data replication and multi-hop transfers.
---
 docs/operator/setting_up_demo.md | 204 +++++++++++++++++++++++++++++++
 1 file changed, 204 insertions(+)

diff --git a/docs/operator/setting_up_demo.md b/docs/operator/setting_up_demo.md
index 3934261940c..dbe978a666e 100644
--- a/docs/operator/setting_up_demo.md
+++ b/docs/operator/setting_up_demo.md
@@ -423,3 +423,207 @@ $ rucio list-file-replicas test:mynewdataset
 | test  | file4 | 10.486 MB | 65786e49 | XRD3: root://xrd3:1096//rucio/... |
 +-------+-------+-----------+----------+-----------------------------------+
 ```
+
+# Configuring S3 Storage and Multi-Hop Transfers in Rucio
+
+
+
+This tutorial walks you through setting up a complete Rucio test environment using Docker Compose. The playground includes 17 containers with XRootD servers, MinIO S3 storage, FTS (File Transfer Service), and a full Rucio stack for testing data replication and multi-hop transfers.
+
+The original scripts and a Jupyter notebook version of the tutorial are available on [rucio-playground](/_resources/rucio-playground).
+
+
+## Prerequisites
+
+Before starting, ensure you have the following installed:
+
+- **Docker** (with Docker Compose support)
+- **Python 3** (for Rucio clients)
+- **Basic understanding** of Rucio concepts (RSEs, replication rules, datasets)
+
+## Tutorial Scripts Overview
+
+This tutorial uses a series of numbered scripts that progressively set up and configure your Rucio playground environment. Each script builds on the previous one to create a fully functional test infrastructure.
+
+### Environment Management Scripts
+
+#### qt-999-down.sh - Clean Up Environment
+
+Stops and removes all Docker containers, volumes, and resources from the Rucio playground. Use this to start fresh or clean up after testing.
+
+```bash
+./qt-999-down.sh
+```
+
+**What it does:**
+- Stops all running containers
+- Removes containers and networks
+- Prunes orphaned containers
+- Attempts to clean up unused volumes
+
+:::warning[Volume Persistence Issue]
+The `docker volume prune -f` command only removes **unused** volumes, not the ones that might still be referenced. **PostgreSQL database volumes persist** between runs, keeping old schema objects.
+
+To completely clean up, you may need to manually remove specific volumes:
+
+```bash
+docker volume rm dev_vol-ruciodb-data1 2>/dev/null
+docker volume rm dev_vol-ftsdb-mysql1 2>/dev/null
+```
+:::
+
+#### qt-000-up.sh - Start the Environment
+
+Launches the complete Docker Compose stack with all storage services and the Rucio infrastructure.
+
+```bash
+./qt-000-up.sh
+```
+
+**What it does:**
+- Generates MinIO SSL certificates
+- Starts 17 containers including:
+  - XRootD servers (XRD1, XRD2, XRD3)
+  - MinIO instances (MINIO1, MINIO2)
+  - File Transfer Service (FTS)
+  - Rucio server and daemons
+  - PostgreSQL database
+
+
+### Configuration Scripts
+
+#### qt-001-initialize.sh - Initialize Rucio
+
+Runs the Rucio initialization script that sets up the basic Rucio infrastructure.
+
+```bash
+./qt-001-initialize.sh
+```
+
+**What it does:**
+- Creates test scopes
+- Creates initial datasets
+- Registers XRootD RSEs (XRD1, XRD2, XRD3)
+- Sets up basic RSE attributes
+
+#### qt-002-xrd3-http-protocol.sh - Add HTTPS Protocol
+
+Adds HTTPS protocol support to the XRD3 RSE, enabling it to communicate with S3 backends for multi-hop transfers.
+
+```bash
+./qt-002-xrd3-http-protocol.sh
+```
+
+**What it does:**
+- Configures HTTPS protocol on XRD3
+- Enables S3 compatibility for multi-hop routing
+- Sets protocol priorities for WAN and LAN operations
+
+#### qt-003-minio-buckets.sh - Create MinIO Buckets
+
+Creates the necessary storage buckets on MinIO instances.
+
+```bash
+./qt-003-minio-buckets.sh
+```
+
+**What it does:**
+- Creates the `rucio` bucket on MinIO instances
+- Sets appropriate bucket policies
+- Verifies bucket creation
+
+#### qt-004-minio-rses.sh - Configure MinIO RSEs
+
+Registers MinIO instances as Rucio Storage Elements with S3 protocol configuration.
+
+```bash
+./qt-004-minio-rses.sh
+```
+
+**What it does:**
+- Registers MINIO1 and MINIO2 as Rucio RSEs
+- Configures S3 protocol endpoints
+- Sets up storage credentials
+- Configures FTS endpoints for each MinIO RSE
+- Establishes network distances between MinIO RSEs and XRD3 for multi-hop routing
+
+:::tip[Understanding RSE Distance]
+RSE distance determines routing for data transfers. A distance of 1 means direct transfer is possible. Setting distances between MINIO RSEs and XRD3 enables multi-hop transfers through XRD3.
+:::
+
+#### qt-005-minio-fts-creds.sh - Configure FTS Credentials
+
+Registers S3 storage credentials in the File Transfer Service (FTS).
+
+```bash
+./qt-005-minio-fts-creds.sh
+```
+
+**What it does:**
+- Registers S3 storage credentials for both MinIO instances in FTS
+- Allows FTS to authenticate and perform third-party copy operations
+- Configures GFAL2 S3 settings for proper S3 protocol handling
+
+:::info[What is FTS?]
+FTS (File Transfer Service) is responsible for executing actual data transfers between storage endpoints. It handles authentication, retries, checksums, and provides monitoring for transfers.
+:::
+
+### Data Management Scripts
+
+#### qt-006-simple-upload.sh - Upload Test Data
+
+Creates and uploads initial test files to MinIO instances.
+
+```bash
+./qt-006-simple-upload.sh
+```
+
+**What it does:**
+- Generates two random test files
+- Uploads one file to MINIO1
+- Uploads one file to MINIO2
+- Adds files to test datasets for replication testing
+
+#### qt-007-replication-rules.sh - Create Replication Rules
+
+Creates replication rules to trigger data transfers between storage endpoints.
+
+```bash
+./qt-007-replication-rules.sh
+```
+
+**What it does:**
+- Creates a replication rule to copy dataset to XRD1
+- Creates a replication rule to copy dataset to MINIO2
+- Triggers the Rucio rule evaluation workflow
+
+
+#### qt-008-replicate-loop.sh - Execute Replication Cycle
+
+Runs a single iteration of the Rucio replication cycle to process pending transfers.
+
+```bash
+./qt-008-replicate-loop.sh
+```
+
+**What it does:**
+- Executes `rucio-judge-evaluator --run-once`
+- Evaluates pending replication rules
+- Submits transfer requests to FTS
+- Updates rule states based on transfer results
+
+:::info[Running Continuously]
+In a production environment, these daemons run continuously. This script executes a single cycle for testing purposes. You may need to run it multiple times or check daemon logs to see transfers complete.
+:::
+
+
+### Multi-Hop Transfer Flow
+
+The playground demonstrates multi-hop transfers where data flows from MinIO through XRD3 to other destinations:
+
+1. User creates replication rule
+2. Judge Evaluator processes rule
+3. Conveyor submits transfer to FTS
+4. FTS orchestrates multi-hop transfer:
+   - MINIO1/2 → XRD3 (S3 to HTTPS)
+   - XRD3 → XRD1/2 (XRootD protocol)

From aea3543357f9a68cab76c22495f135d5267c6527 Mon Sep 17 00:00:00 2001
From: alessio94 <alessiopizzini@gmail.com>
Date: Wed, 11 Feb 2026 15:27:07 +0100
Subject: [PATCH 2/5] Add script to start Rucio playground environment

---
 docs/operator/_resources/rucio-playground/qt-000-up.txt | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 docs/operator/_resources/rucio-playground/qt-000-up.txt

diff --git a/docs/operator/_resources/rucio-playground/qt-000-up.txt b/docs/operator/_resources/rucio-playground/qt-000-up.txt
new file mode 100644
index 00000000000..9523b7e1e6b
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/qt-000-up.txt
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+cd `dirname $0`
+
+etc/certs/generate_minio12.sh
+
+docker compose --file etc/docker/dev/docker-compose-qt.yml --profile storage up -d

From 93afe4630b6bd65a8d9fc677015133e4f56d88ce Mon Sep 17 00:00:00 2001
From: alessio94 <alessiopizzini@gmail.com>
Date: Wed, 11 Feb 2026 15:37:25 +0100
Subject: [PATCH 3/5] More files for rucio playground

---
 .../rucio-playground/qt-001-initialize.txt    | 11 ++++
 .../qt-002-xrd3-http-protocol.txt             |  9 +++
 .../rucio-playground/qt-003-minio-buckets.txt | 17 ++++++
 .../rucio-playground/qt-004-minio-rses.txt    | 57 ++++++++++++++++++
 .../qt-005-minio-fts-creds.txt                | 60 +++++++++++++++++++
 .../rucio-playground/qt-006-simple-upload.txt | 16 +++++
 .../qt-007-replication-rules.txt              |  6 ++
 .../rucio-playground/qt-008-replicte-loop.txt | 15 +++++
 .../rucio-playground/qt-999-down.txt          |  7 +++
 .../_resources/rucio-playground/qt-enter.txt  | 13 ++++
 .../_resources/rucio-playground/qt-replay.txt |  4 ++
 11 files changed, 215 insertions(+)
 create mode 100644 docs/operator/_resources/rucio-playground/qt-001-initialize.txt
 create mode 100644 docs/operator/_resources/rucio-playground/qt-002-xrd3-http-protocol.txt
 create mode 100644 docs/operator/_resources/rucio-playground/qt-003-minio-buckets.txt
 create mode 100644 docs/operator/_resources/rucio-playground/qt-004-minio-rses.txt
 create mode 100644 docs/operator/_resources/rucio-playground/qt-005-minio-fts-creds.txt
 create mode 100644 docs/operator/_resources/rucio-playground/qt-006-simple-upload.txt
 create mode 100644 docs/operator/_resources/rucio-playground/qt-007-replication-rules.txt
 create mode 100644 docs/operator/_resources/rucio-playground/qt-008-replicte-loop.txt
 create mode 100644 docs/operator/_resources/rucio-playground/qt-999-down.txt
 create mode 100644 docs/operator/_resources/rucio-playground/qt-enter.txt
 create mode 100644 docs/operator/_resources/rucio-playground/qt-replay.txt

diff --git a/docs/operator/_resources/rucio-playground/qt-001-initialize.txt b/docs/operator/_resources/rucio-playground/qt-001-initialize.txt
new file mode 100644
index 00000000000..e9ee4d67b85
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/qt-001-initialize.txt
@@ -0,0 +1,11 @@
+#!/bin/bash
+
+# rucio list-file-replicas test:dataset1
+# rucio list-file-replicas test:dataset2
+# rucio list-file-replicas test:dataset3
+# rucio list-file-replicas test:container
+
+# Step zero, get a compliant proxy. The key must NOT be group/other readable
+# (KEY=$(mktemp); cat /opt/rucio/etc/userkey.pem > "$KEY"; voms-proxy-init -valid 9999:00 -cert /opt/rucio/etc/usercert.pem -key "$KEY"; rm -f "$KEY")
+
+docker exec -i dev-rucio-1 /bin/bash tools/run_tests.sh -ir
diff --git a/docs/operator/_resources/rucio-playground/qt-002-xrd3-http-protocol.txt b/docs/operator/_resources/rucio-playground/qt-002-xrd3-http-protocol.txt
new file mode 100644
index 00000000000..60054a91094
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/qt-002-xrd3-http-protocol.txt
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# XRD3 is enabled for multihop transfers
+# so add HTTP protocol to be able to talk to S3
+
+docker exec -i dev-rucio-1 /bin/bash <<END
+rucio rse protocol add XRD3 --host xrd3 --scheme https --prefix //rucio --port 1096 --impl rucio.rse.protocols.gfal.Default --domain-json '{"wan": {"read": 2, "write": 2, "delete": 2, "third_party_copy_read": 2, "third_party_copy_write": 2}, "lan": {"read": 2, "write": 2, "delete": 2}}'
+
+END
diff --git a/docs/operator/_resources/rucio-playground/qt-003-minio-buckets.txt b/docs/operator/_resources/rucio-playground/qt-003-minio-buckets.txt
new file mode 100644
index 00000000000..27c0fd3b0e6
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/qt-003-minio-buckets.txt
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+docker exec -i dev-minio-1 /bin/bash <<END
+export MC_INSECURE=true
+mc alias set local https://localhost:9001 admin password
+mc admin info local
+mc mb local/rucio
+mc ls local/
+END
+
+docker exec -i dev-minio-2 /bin/bash <<END
+export MC_INSECURE=true
+mc alias set local https://localhost:9002 admin password
+mc admin info local
+mc mb local/rucio
+mc ls local/
+END
diff --git a/docs/operator/_resources/rucio-playground/qt-004-minio-rses.txt b/docs/operator/_resources/rucio-playground/qt-004-minio-rses.txt
new file mode 100644
index 00000000000..9341094a6f6
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/qt-004-minio-rses.txt
@@ -0,0 +1,57 @@
+#!/bin/bash
+
+#
+# https://rucio.github.io/documentation/operator/s3_rse_config/
+#
+
+# set up RSEs
+docker exec -i dev-rucio-1 /bin/bash <<END
+rucio rse add MINIO1
+rucio rse protocol add MINIO1 --host minio1 --port 9001 --scheme https --prefix /rucio/ --impl rucio.rse.protocols.gfal.NoRename --domain-json '{"lan": {"read": 1, "write": 1, "delete": 1}, "wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}}'
+rucio rse attribute add MINIO1 --key sign_url --value s3
+rucio rse attribute add MINIO1 --key s3_url_style --value path
+rucio rse attribute add MINIO1 --key verify_checksum --value False
+rucio rse attribute add MINIO1 --key skip_upload_stat --value True
+rucio rse attribute add MINIO1 --key strict_copy --value True
+rucio rse attribute add MINIO1 --key fts --value https://fts:8446
+rucio account limit add root --rse MINIO1 --bytes infinity
+
+rucio rse add MINIO2
+rucio rse protocol add MINIO2 --host minio2 --port 9002 --scheme https --prefix /rucio/ --impl rucio.rse.protocols.gfal.NoRename --domain-json '{"lan": {"read": 1, "write": 1, "delete": 1}, "wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}}'
+rucio rse attribute add MINIO2 --key sign_url --value s3
+rucio rse attribute add MINIO2 --key s3_url_style --value path
+rucio rse attribute add MINIO2 --key verify_checksum --value False
+rucio rse attribute add MINIO2 --key skip_upload_stat --value True
+rucio rse attribute add MINIO2 --key strict_copy --value True
+rucio rse attribute add MINIO2 --key fts --value https://fts:8446
+rucio account limit add root --rse MINIO2 --bytes infinity
+
+# XRD3 has HTTP enabled, link it up to our mesh
+rucio rse distance add XRD3 MINIO1 --distance 1
+rucio rse distance add XRD3 MINIO2 --distance 1
+
+rucio rse distance add MINIO1 XRD3 --distance 1
+rucio rse distance add MINIO2 XRD3 --distance 1
+END
+
+# set up credentials for RSEs
+docker exec -i dev-rucio-1 /bin/bash <<'END'
+ID1=$(rucio rse show MINIO1 | grep '^  id:' | awk '{print$2}')
+ID2=$(rucio rse show MINIO2 | grep '^  id:' | awk '{print$2}')
+cat >/opt/rucio/etc/rse-accounts.cfg <<JSON
+{
+  "$ID1": {
+    "access_key": "admin",
+    "secret_key": "password",
+    "signature_version": "s3v4",
+    "region": "us-east-1"
+  },
+  "$ID2": {
+    "access_key": "admin",
+    "secret_key": "password",
+    "signature_version": "s3v4",
+    "region": "us-east-1"
+  }
+}
+JSON
+END
diff --git a/docs/operator/_resources/rucio-playground/qt-005-minio-fts-creds.txt b/docs/operator/_resources/rucio-playground/qt-005-minio-fts-creds.txt
new file mode 100644
index 00000000000..d9b4f53b570
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/qt-005-minio-fts-creds.txt
@@ -0,0 +1,60 @@
+#!/bin/bash
+
+#
+# https://docs.egi.eu/users/tutorials/adhoc/data-transfer-object-storage/
+# https://fts3-docs.web.cern.ch/fts3-docs/docs/s3_support.html
+#
+
+# set up credentials for MINIO RSEs in FTS3
+docker exec -i dev-fts-1 /bin/bash <<'END'
+curl \
+  --cert /etc/grid-security/hostcert.pem \
+  --key /etc/grid-security/hostkey.pem \
+  --capath /etc/grid-security/certificates \
+  https://fts:8446/config/cloud_storage \
+  -H "Content-Type: application/json" \
+  -X POST \
+  -d '{"storage_name":"S3:minio1"}'
+
+curl \
+  --cert /etc/grid-security/hostcert.pem \
+  --key /etc/grid-security/hostkey.pem \
+  --capath /etc/grid-security/certificates \
+  https://fts:8446/config/cloud_storage \
+  -H "Content-Type: application/json" \
+  -X POST \
+  -d '{"user_dn":"/CN=Rucio User","storage_name":"S3:minio1","access_token":"admin","access_token_secret":"password"}'
+
+
+curl \
+  --cert /etc/grid-security/hostcert.pem \
+  --key /etc/grid-security/hostkey.pem \
+  --capath /etc/grid-security/certificates \
+  https://fts:8446/config/cloud_storage \
+  -H "Content-Type: application/json" \
+  -X POST \
+  -d '{"storage_name":"S3:minio2"}'
+
+curl \
+  --cert /etc/grid-security/hostcert.pem \
+  --key /etc/grid-security/hostkey.pem \
+  --capath /etc/grid-security/certificates \
+  https://fts:8446/config/cloud_storage \
+  -H "Content-Type: application/json" \
+  -X POST \
+  -d '{"user_dn":"/CN=Rucio User","storage_name":"S3:minio2","access_token":"admin","access_token_secret":"password"}'
+
+cat >/etc/gfal2.d/s3.conf <<INI
+[S3:MINIO1]
+ACCESS_KEY=admin
+SECRET_KEY=password
+REGION=us-east-1
+ALTERNATE=true
+
+[S3:MINIO2]
+ACCESS_KEY=admin
+SECRET_KEY=password
+REGION=us-east-1
+ALTERNATE=true
+INI
+END
diff --git a/docs/operator/_resources/rucio-playground/qt-006-simple-upload.txt b/docs/operator/_resources/rucio-playground/qt-006-simple-upload.txt
new file mode 100644
index 00000000000..8af530e6428
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/qt-006-simple-upload.txt
@@ -0,0 +1,16 @@
+#!/bin/bash
+
+#
+# https://github.com/rucio/k8s-tutorial#create-initial-transfer-testing-data
+#
+
+docker exec -i dev-rucio-1 /bin/bash <<END
+dd if=/dev/urandom of=file5 bs=10M count=1
+dd if=/dev/urandom of=file6 bs=10M count=1
+
+rucio upload --rse MINIO1 --scope test file5
+rucio upload --rse MINIO2 --scope test file6
+
+rucio did add --type dataset test:dataset9
+rucio did content add -to test:dataset9 test:file5 test:file6
+END
diff --git a/docs/operator/_resources/rucio-playground/qt-007-replication-rules.txt b/docs/operator/_resources/rucio-playground/qt-007-replication-rules.txt
new file mode 100644
index 00000000000..4f2c406f805
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/qt-007-replication-rules.txt
@@ -0,0 +1,6 @@
+#!/bin/bash
+
+docker exec -i dev-rucio-1 /bin/bash <<END
+rucio rule add test:dataset9 --copies 1 --rses XRD1
+rucio rule add test:dataset2 --copies 1 --rses MINIO2
+END
diff --git a/docs/operator/_resources/rucio-playground/qt-008-replicte-loop.txt b/docs/operator/_resources/rucio-playground/qt-008-replicte-loop.txt
new file mode 100644
index 00000000000..7b0da943e1e
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/qt-008-replicte-loop.txt
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+#
+# https://github.com/rucio/k8s-tutorial#create-initial-transfer-testing-data
+#
+
+docker exec -i dev-rucio-1 /bin/bash <<END
+rucio rule list --account root
+rucio-judge-evaluator --run-once
+rucio-conveyor-submitter --run-once
+rucio-conveyor-poller --run-once  --older-than 0
+rucio-conveyor-finisher --run-once
+
+rucio rule list --account root
+END
diff --git a/docs/operator/_resources/rucio-playground/qt-999-down.txt b/docs/operator/_resources/rucio-playground/qt-999-down.txt
new file mode 100644
index 00000000000..f45a3fb974a
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/qt-999-down.txt
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+cd `dirname $0`
+
+docker compose --file etc/docker/dev/docker-compose-qt.yml --profile storage down
+docker container prune -f
+docker volume prune -f
diff --git a/docs/operator/_resources/rucio-playground/qt-enter.txt b/docs/operator/_resources/rucio-playground/qt-enter.txt
new file mode 100644
index 00000000000..2ed91df875e
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/qt-enter.txt
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+echo "Run: tools/run_tests.sh -ir"
+# rucio list-file-replicas test:dataset1
+# rucio list-file-replicas test:dataset2
+# rucio list-file-replicas test:dataset3
+# rucio list-file-replicas test:container
+
+# Step zero, get a compliant proxy. The key must NOT be group/other readable
+# (KEY=$(mktemp); cat /opt/rucio/etc/userkey.pem > "$KEY"; voms-proxy-init -valid 9999:00 -cert /opt/rucio/etc/usercert.pem -key "$KEY"; rm -f "$KEY")
+
+
+docker exec -it dev-rucio-1 /bin/bash
diff --git a/docs/operator/_resources/rucio-playground/qt-replay.txt b/docs/operator/_resources/rucio-playground/qt-replay.txt
new file mode 100644
index 00000000000..141f7da842a
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/qt-replay.txt
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+cd `dirname $0`
+for i in {0..8} ; do ./qt-00${i}-* ; done

From 6440726089eada345ee587f73a1673e18e28205f Mon Sep 17 00:00:00 2001
From: alessio94 <alessiopizzini@gmail.com>
Date: Wed, 11 Feb 2026 15:39:04 +0100
Subject: [PATCH 4/5] Added Jupyter notebook file

---
 .../rucio-playground-tutorial.ipynb           | 410 ++++++++++++++++++
 1 file changed, 410 insertions(+)
 create mode 100644 docs/operator/_resources/rucio-playground/rucio-playground-tutorial.ipynb

diff --git a/docs/operator/_resources/rucio-playground/rucio-playground-tutorial.ipynb b/docs/operator/_resources/rucio-playground/rucio-playground-tutorial.ipynb
new file mode 100644
index 00000000000..9a2859a4937
--- /dev/null
+++ b/docs/operator/_resources/rucio-playground/rucio-playground-tutorial.ipynb
@@ -0,0 +1,410 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "# Rucio Playground Tutorial\n\nThis notebook walks you through setting up a complete **Rucio** test environment using Docker Compose. The playground includes 17 containers with XRootD servers, MinIO S3 storage, FTS (File Transfer Service), and a full Rucio stack for testing data replication and multi-hop transfers.\n\nRucio is a scientific data management framework used to organize, manage, and access large volumes of scientific data across distributed storage systems. It was originally developed for the ATLAS experiment at CERN and is now used across many scientific communities.\n\n## What you will learn\n\n1. **Environment setup** — generate certificates, start 17 Docker containers\n2. **Storage registration** — configure XRootD and MinIO S3 as Rucio Storage Elements (RSEs)\n3. **Credential management** — wire up S3 credentials for Rucio, FTS, and GFAL2\n4. **Data upload** — create files, upload to RSEs, organize into datasets\n5. **Replication policies** — declare where data should live using Rucio rules\n6. **Automated transfers** — run the judge/conveyor pipeline to execute multi-hop transfers\n\n## Prerequisites\n\n- **Docker** (with Docker Compose support) installed locally\n- The **Rucio dev environment** cloned (provides `etc/docker/dev/docker-compose-qt.yml`)\n- This notebook should be run **from the root of the repository** so that relative paths resolve correctly\n\n> **Note:** All Rucio CLI commands run inside Docker containers — no local Rucio or Python installation is required.\n\n## References\n\n- [Rucio Documentation](https://rucio.cern.ch/documentation)\n- [S3 RSE Configuration](https://rucio.github.io/documentation/operator/s3_rse_config/)\n- [FTS3 S3 Support](https://fts3-docs.web.cern.ch/fts3-docs/docs/s3_support.html)\n- [EGI Data Transfer Tutorial](https://docs.egi.eu/users/tutorials/adhoc/data-transfer-object-storage/)\n- [Rucio K8s Tutorial](https://github.com/rucio/k8s-tutorial)\n- [Rucio Docker Dev Environment](https://github.com/rucio/rucio/tree/master/etc/docker/dev)"
+  },
+  {
+   "cell_type": "markdown",
+   "source": "---\n## Environment Overview\n\nThe Docker Compose stack spins up 17 containers. The key ones you will interact with:\n\n| Container | Service | Role |\n|-----------|---------|------|\n| `dev-rucio-1` | Rucio server | CLI commands, daemons, uploads |\n| `dev-minio-1` | MinIO (port 9001) | S3 storage instance 1 |\n| `dev-minio-2` | MinIO (port 9002) | S3 storage instance 2 |\n| `dev-fts-1` | FTS3 | File transfer service |\n| XRD1, XRD2, XRD3 | XRootD servers | Grid storage (XRD3 is the multihop intermediary) |\n\n**Credentials** used throughout (demo only): `admin` / `password`",
+   "metadata": {}
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## Step 0 — Start the Environment\n",
+    "\n",
+    "Generate TLS certificates for the MinIO instances and start the full Docker Compose stack (Rucio server, MinIO storage nodes, FTS3 transfer service, XRootD servers)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "etc/certs/generate_minio12.sh"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "docker compose --file etc/docker/dev/docker-compose-qt.yml --profile storage up -d"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## Step 1 — Initialize Rucio\n",
+    "\n",
+    "Run Rucio's built-in initialization and test suite inside the main container. This bootstraps the database schema, creates the default account (`root`), and registers the initial set of RSEs (Replica Storage Elements) and scopes.\n",
+    "\n",
+    "> **Note:** This may take a few minutes as it runs the full init + test harness (`tools/run_tests.sh -ir`)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "docker exec -i dev-rucio-1 /bin/bash tools/run_tests.sh -ir"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## Step 2 — Add HTTPS Protocol to XRD3 RSE\n",
+    "\n",
+    "XRD3 is an XRootD storage element that is enabled for **multihop transfers**. We add an HTTPS protocol endpoint so it can act as an intermediary when transferring data to/from S3 storage (which speaks HTTPS, not XRootD native).\n",
+    "\n",
+    "The `gfal.Default` implementation is used, with both LAN and WAN domains configured for read, write, delete, and third-party-copy operations."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "docker exec -i dev-rucio-1 /bin/bash <<END\n",
+    "rucio rse protocol add XRD3 --host xrd3 --scheme https --prefix //rucio --port 1096 --impl rucio.rse.protocols.gfal.Default --domain-json '{\"wan\": {\"read\": 2, \"write\": 2, \"delete\": 2, \"third_party_copy_read\": 2, \"third_party_copy_write\": 2}, \"lan\": {\"read\": 2, \"write\": 2, \"delete\": 2}}'\n",
+    "END"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## Step 3 — Create MinIO S3 Buckets\n",
+    "\n",
+    "Create a `rucio` bucket on each MinIO instance. MinIO provides S3-compatible object storage.\n",
+    "\n",
+    "- **dev-minio-1** listens on port 9001\n",
+    "- **dev-minio-2** listens on port 9002\n",
+    "\n",
+    "We use the MinIO Client (`mc`) inside each container to set up the alias and create the bucket. Credentials: `admin` / `password` (demo only)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "docker exec -i dev-minio-1 /bin/bash <<END\n",
+    "export MC_INSECURE=true\n",
+    "mc alias set local https://localhost:9001 admin password\n",
+    "mc admin info local\n",
+    "mc mb local/rucio\n",
+    "mc ls local/\n",
+    "END"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "docker exec -i dev-minio-2 /bin/bash <<END\n",
+    "export MC_INSECURE=true\n",
+    "mc alias set local https://localhost:9002 admin password\n",
+    "mc admin info local\n",
+    "mc mb local/rucio\n",
+    "mc ls local/\n",
+    "END"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "---\n## Step 4 — Register MinIO RSEs and Configure S3 Attributes\n\nRegister **MINIO1** and **MINIO2** as Rucio RSEs (Replica Storage Elements) with:\n- S3-compatible protocol (`gfal.NoRename` — S3 does not support rename operations)\n- Signed URL support (`sign_url = s3`)\n- Path-style S3 URLs (`s3_url_style = path`)\n- FTS endpoint for third-party copy transfers\n- Infinite storage quota for the `root` account\n\nWe also define **distances** between XRD3 and the MinIO RSEs (bidirectional, distance=1) to enable the transfer mesh.\n\n> **Understanding RSE Distance:** RSE distance determines routing for data transfers. A distance of 1 means direct transfer is possible. Setting distances between MINIO RSEs and XRD3 enables multi-hop transfers through XRD3.\n\nFinally, the RSE S3 credentials (access key / secret key) are written to `/opt/rucio/etc/rse-accounts.cfg` keyed by RSE ID.\n\n> **Ref:** [S3 RSE Configuration Guide](https://rucio.github.io/documentation/operator/s3_rse_config/)"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "# Register RSEs, set attributes, define distances\n",
+    "docker exec -i dev-rucio-1 /bin/bash <<END\n",
+    "rucio rse add MINIO1\n",
+    "rucio rse protocol add MINIO1 --host minio1 --port 9001 --scheme https --prefix /rucio/ --impl rucio.rse.protocols.gfal.NoRename --domain-json '{\"lan\": {\"read\": 1, \"write\": 1, \"delete\": 1}, \"wan\": {\"read\": 1, \"write\": 1, \"delete\": 1, \"third_party_copy_read\": 1, \"third_party_copy_write\": 1}}'\n",
+    "rucio rse attribute add MINIO1 --key sign_url --value s3\n",
+    "rucio rse attribute add MINIO1 --key s3_url_style --value path\n",
+    "rucio rse attribute add MINIO1 --key verify_checksum --value False\n",
+    "rucio rse attribute add MINIO1 --key skip_upload_stat --value True\n",
+    "rucio rse attribute add MINIO1 --key strict_copy --value True\n",
+    "rucio rse attribute add MINIO1 --key fts --value https://fts:8446\n",
+    "rucio account limit add root --rse MINIO1 --bytes infinity\n",
+    "\n",
+    "rucio rse add MINIO2\n",
+    "rucio rse protocol add MINIO2 --host minio2 --port 9002 --scheme https --prefix /rucio/ --impl rucio.rse.protocols.gfal.NoRename --domain-json '{\"lan\": {\"read\": 1, \"write\": 1, \"delete\": 1}, \"wan\": {\"read\": 1, \"write\": 1, \"delete\": 1, \"third_party_copy_read\": 1, \"third_party_copy_write\": 1}}'\n",
+    "rucio rse attribute add MINIO2 --key sign_url --value s3\n",
+    "rucio rse attribute add MINIO2 --key s3_url_style --value path\n",
+    "rucio rse attribute add MINIO2 --key verify_checksum --value False\n",
+    "rucio rse attribute add MINIO2 --key skip_upload_stat --value True\n",
+    "rucio rse attribute add MINIO2 --key strict_copy --value True\n",
+    "rucio rse attribute add MINIO2 --key fts --value https://fts:8446\n",
+    "rucio account limit add root --rse MINIO2 --bytes infinity\n",
+    "\n",
+    "# XRD3 has HTTP enabled, link it up to our mesh\n",
+    "rucio rse distance add XRD3 MINIO1 --distance 1\n",
+    "rucio rse distance add XRD3 MINIO2 --distance 1\n",
+    "\n",
+    "rucio rse distance add MINIO1 XRD3 --distance 1\n",
+    "rucio rse distance add MINIO2 XRD3 --distance 1\n",
+    "END"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "# Write S3 credentials keyed by RSE ID\n",
+    "docker exec -i dev-rucio-1 /bin/bash <<'END'\n",
+    "ID1=$(rucio rse show MINIO1 | grep '^  id:' | awk '{print$2}')\n",
+    "ID2=$(rucio rse show MINIO2 | grep '^  id:' | awk '{print$2}')\n",
+    "cat >/opt/rucio/etc/rse-accounts.cfg <<JSON\n",
+    "{\n",
+    "  \"$ID1\": {\n",
+    "    \"access_key\": \"admin\",\n",
+    "    \"secret_key\": \"password\",\n",
+    "    \"signature_version\": \"s3v4\",\n",
+    "    \"region\": \"us-east-1\"\n",
+    "  },\n",
+    "  \"$ID2\": {\n",
+    "    \"access_key\": \"admin\",\n",
+    "    \"secret_key\": \"password\",\n",
+    "    \"signature_version\": \"s3v4\",\n",
+    "    \"region\": \"us-east-1\"\n",
+    "  }\n",
+    "}\n",
+    "JSON\n",
+    "END"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "---\n## Step 5 — Configure FTS3 Cloud Storage Credentials\n\nRegister the MinIO S3 endpoints as cloud storage in the **FTS3** transfer service and supply the access credentials. This allows FTS3 to perform third-party-copy transfers between S3 endpoints.\n\n> **What is FTS?** FTS (File Transfer Service) is the middleware responsible for executing actual data transfers between storage endpoints. It handles authentication, retries, checksums, and provides monitoring. Rucio delegates the physical movement of data to FTS.\n\nWe also write a `gfal2` S3 configuration file so the transfer agent knows how to authenticate.\n\n> **Ref:** [FTS3 S3 Support](https://fts3-docs.web.cern.ch/fts3-docs/docs/s3_support.html) · [EGI Data Transfer Tutorial](https://docs.egi.eu/users/tutorials/adhoc/data-transfer-object-storage/)"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "docker exec -i dev-fts-1 /bin/bash <<'END'\n",
+    "# Register cloud storage endpoints in FTS3\n",
+    "curl \\\n",
+    "  --cert /etc/grid-security/hostcert.pem \\\n",
+    "  --key /etc/grid-security/hostkey.pem \\\n",
+    "  --capath /etc/grid-security/certificates \\\n",
+    "  https://fts:8446/config/cloud_storage \\\n",
+    "  -H \"Content-Type: application/json\" \\\n",
+    "  -X POST \\\n",
+    "  -d '{\"storage_name\":\"S3:minio1\"}'\n",
+    "\n",
+    "curl \\\n",
+    "  --cert /etc/grid-security/hostcert.pem \\\n",
+    "  --key /etc/grid-security/hostkey.pem \\\n",
+    "  --capath /etc/grid-security/certificates \\\n",
+    "  https://fts:8446/config/cloud_storage \\\n",
+    "  -H \"Content-Type: application/json\" \\\n",
+    "  -X POST \\\n",
+    "  -d '{\"user_dn\":\"/CN=Rucio User\",\"storage_name\":\"S3:minio1\",\"access_token\":\"admin\",\"access_token_secret\":\"password\"}'\n",
+    "\n",
+    "\n",
+    "curl \\\n",
+    "  --cert /etc/grid-security/hostcert.pem \\\n",
+    "  --key /etc/grid-security/hostkey.pem \\\n",
+    "  --capath /etc/grid-security/certificates \\\n",
+    "  https://fts:8446/config/cloud_storage \\\n",
+    "  -H \"Content-Type: application/json\" \\\n",
+    "  -X POST \\\n",
+    "  -d '{\"storage_name\":\"S3:minio2\"}'\n",
+    "\n",
+    "curl \\\n",
+    "  --cert /etc/grid-security/hostcert.pem \\\n",
+    "  --key /etc/grid-security/hostkey.pem \\\n",
+    "  --capath /etc/grid-security/certificates \\\n",
+    "  https://fts:8446/config/cloud_storage \\\n",
+    "  -H \"Content-Type: application/json\" \\\n",
+    "  -X POST \\\n",
+    "  -d '{\"user_dn\":\"/CN=Rucio User\",\"storage_name\":\"S3:minio2\",\"access_token\":\"admin\",\"access_token_secret\":\"password\"}'\n",
+    "\n",
+    "# Write gfal2 S3 configuration\n",
+    "cat >/etc/gfal2.d/s3.conf <<INI\n",
+    "[S3:MINIO1]\n",
+    "ACCESS_KEY=admin\n",
+    "SECRET_KEY=password\n",
+    "REGION=us-east-1\n",
+    "ALTERNATE=true\n",
+    "\n",
+    "[S3:MINIO2]\n",
+    "ACCESS_KEY=admin\n",
+    "SECRET_KEY=password\n",
+    "REGION=us-east-1\n",
+    "ALTERNATE=true\n",
+    "INI\n",
+    "END"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## Step 6 — Create Test Files and Upload to RSEs\n",
+    "\n",
+    "Generate two 10 MB random files and upload them to different RSEs:\n",
+    "- `file5` → **MINIO1**\n",
+    "- `file6` → **MINIO2**\n",
+    "\n",
+    "Then create a Rucio **dataset** (`test:dataset9`) and attach both files to it. Datasets are logical containers that group related files (DIDs) together.\n",
+    "\n",
+    "> **Ref:** [K8s Tutorial — Create transfer testing data](https://github.com/rucio/k8s-tutorial#create-initial-transfer-testing-data)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "docker exec -i dev-rucio-1 /bin/bash <<END\n",
+    "dd if=/dev/urandom of=file5 bs=10M count=1\n",
+    "dd if=/dev/urandom of=file6 bs=10M count=1\n",
+    "\n",
+    "rucio upload --rse MINIO1 --scope test file5\n",
+    "rucio upload --rse MINIO2 --scope test file6\n",
+    "\n",
+    "rucio did add --type dataset test:dataset9\n",
+    "rucio did content add -to test:dataset9 test:file5 test:file6\n",
+    "END"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## Step 7 — Create Replication Rules\n",
+    "\n",
+    "Replication rules tell Rucio **where** data should exist. Rucio's rule engine will then figure out *how* to get it there.\n",
+    "\n",
+    "We create two rules:\n",
+    "- `test:dataset9` → 1 copy on **XRD1** (will trigger S3→XRootD transfers via multihop through XRD3)\n",
+    "- `test:dataset2` → 1 copy on **MINIO2** (standard test dataset replication)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "docker exec -i dev-rucio-1 /bin/bash <<END\n",
+    "rucio rule add test:dataset9 --copies 1 --rses XRD1\n",
+    "rucio rule add test:dataset2 --copies 1 --rses MINIO2\n",
+    "END"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "---\n## Step 8 — Execute the Transfer Pipeline\n\nRun the Rucio **conveyor pipeline** to actually execute the transfers created by the rules above. In production, these run as long-lived daemons. Here we run each stage once:\n\n1. **`rucio-judge-evaluator`** — Evaluates replication rules and creates transfer requests\n2. **`rucio-conveyor-submitter`** — Submits transfer requests to FTS3\n3. **`rucio-conveyor-poller`** — Polls FTS3 for transfer completion status\n4. **`rucio-conveyor-finisher`** — Finalizes completed transfers and updates the replica catalog\n\nWe list the rules before and after to observe the state change.\n\n> **Tip:** You may need to **run this cell multiple times**. Transfers take time and the poller/finisher need to catch up with FTS3. Re-run until you see all rule states change to `OK`."
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "docker exec -i dev-rucio-1 /bin/bash <<END\n",
+    "rucio rule list --account root\n",
+    "rucio-judge-evaluator --run-once\n",
+    "rucio-conveyor-submitter --run-once\n",
+    "rucio-conveyor-poller --run-once  --older-than 0\n",
+    "rucio-conveyor-finisher --run-once\n",
+    "\n",
+    "rucio rule list --account root\n",
+    "END"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "---\n## Inspect & Debug\n\nBefore tearing down, you can open an interactive shell inside the Rucio container to inspect the state of your transfers, replicas, and rules."
+  },
+  {
+   "cell_type": "code",
+   "source": "%%bash\n# Check rule states, replica locations, and dataset contents\ndocker exec -i dev-rucio-1 /bin/bash <<END\necho \"=== Rules ===\"\nrucio rule list --account root\n\necho \"\"\necho \"=== Replicas for dataset9 ===\"\nrucio list-file-replicas test:dataset9\n\necho \"\"\necho \"=== Replicas for dataset2 ===\"\nrucio list-file-replicas test:dataset2\n\necho \"\"\necho \"=== RSE info ===\"\nrucio rse show MINIO1\nrucio rse show MINIO2\nEND",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "source": "---\n## Teardown\n\nStop all containers, prune stopped containers and unused volumes to free disk space.\n\n> **Warning:** This destroys all state. You will need to re-run from Step 0 to start again.\n>\n> **Volume Persistence:** `docker volume prune -f` only removes *unused* volumes. PostgreSQL database volumes may persist between runs. To fully clean up:\n> ```\n> docker volume rm dev_vol-ruciodb-data1 2>/dev/null\n> docker volume rm dev_vol-ftsdb-mysql1 2>/dev/null\n> ```",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%bash\n",
+    "docker compose --file etc/docker/dev/docker-compose-qt.yml --profile storage down\n",
+    "docker container prune -f\n",
+    "docker volume prune -f"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "source": "---\n## Appendix: Multi-Hop Transfer Flow\n\nThe playground demonstrates multi-hop transfers where data flows from MinIO S3 through XRD3 to XRootD destinations:\n\n```\nUser creates replication rule\n        │\n        ▼\nJudge Evaluator processes rule\n        │\n        ▼\nConveyor submits transfer to FTS\n        │\n        ▼\nFTS orchestrates multi-hop transfer:\n   MINIO1/2 ──(S3→HTTPS)──▶ XRD3 ──(XRootD)──▶ XRD1/2\n```\n\nXRD3 acts as the **multihop intermediary** because it speaks both HTTPS (needed for S3) and native XRootD protocol. Without the HTTPS protocol added in Step 2, Rucio would have no path between the S3 and XRootD worlds.",
+   "metadata": {}
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.10.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
\ No newline at end of file

From 30fce702ed08a99f02deecb3a8bb9b72518b47e3 Mon Sep 17 00:00:00 2001
From: "pre-commit-ci[bot]"
 <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date: Wed, 11 Feb 2026 14:56:34 +0000
Subject: [PATCH 5/5] [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci
---
 .../_resources/rucio-playground/rucio-playground-tutorial.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/operator/_resources/rucio-playground/rucio-playground-tutorial.ipynb b/docs/operator/_resources/rucio-playground/rucio-playground-tutorial.ipynb
index 9a2859a4937..ba354509b63 100644
--- a/docs/operator/_resources/rucio-playground/rucio-playground-tutorial.ipynb
+++ b/docs/operator/_resources/rucio-playground/rucio-playground-tutorial.ipynb
@@ -407,4 +407,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 4
-}
\ No newline at end of file
+}