[Fusion] Troubleshooting error codes #1007

christopher-hakkaart · 2026-01-12T03:52:52Z

Update headers and sections
Make into multiple pages
Start migration error messages

netlify · 2026-01-12T03:52:57Z

✅ Deploy Preview for seqera-docs ready!

Name	Link
🔨 Latest commit	`911dd2f`
🔍 Latest deploy log	https://app.netlify.com/projects/seqera-docs/deploys/6970ba3a739a250009a716d3
😎 Deploy Preview	https://deploy-preview-1007--seqera-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

…mpliance

fusion_docs/troubleshooting/error-codes-exit-messages.md

Updated last updated date and removed architectural limitations section. Signed-off-by: Justine Geffen <justinegeffen@users.noreply.github.com>

Signed-off-by: Justine Geffen <justinegeffen@users.noreply.github.com>

fusion_docs/troubleshooting/error-codes-exit-messages.md

cristianrcv · 2026-01-20T14:13:40Z

fusion_docs/troubleshooting/error-codes-exit-messages.md

+   - Storage backends return normalized cloud errors (with provider-agnostic categories) or internal errors (`ErrNotFound`, `ErrReadOnly`, etc.)
+   - The FUSE layer maps both cloud errors and internal errors to FUSE status codes (`ENOENT`, `EACCES`, `EREMOTEIO`, `EIO`)
+   - The kernel translates FUSE status to errno values for the application
+   - Fusion logs cloud errors with structured details (provider, error code, HTTP status, request ID)


Suggested change

- Fusion logs cloud errors with structured details (provider, error code, HTTP status, request ID)

- Fusion logs cloud errors with structured details (i.e., provider, error code, HTTP status, request ID)

cristianrcv · 2026-01-20T14:15:11Z

fusion_docs/troubleshooting/error-codes-exit-messages.md

+
+## FUSE status codes
+
+Fusion maps internal errors to standard FUSE status codes returned to the operating system. These are the [errno](https://man7.org/linux/man-pages/man3/errno.3.html) values applications receive when filesystem operations fail.


Suggested change

Fusion maps internal errors to standard FUSE status codes returned to the operating system. These are the [errno](https://man7.org/linux/man-pages/man3/errno.3.html) values applications receive when filesystem operations fail.

Fusion maps internal errors to standard FUSE status codes returned to the operating system. These are the [errno](https://man7.org/linux/man-pages/man3/errno.3.html) values that applications receive when filesystem operations fail.

fusion_docs/troubleshooting/error-codes-exit-messages.md

alberto-miranda · 2026-01-20T14:23:01Z

fusion_docs/troubleshooting/error-codes-exit-messages.md

+1. **Cloud > Storage Backend > FUSE Layer > Kernel > Application**
+
+   - Storage backends catch and normalize cloud errors (network timeouts, auth failures, rate limits) using the `clouderr` package
+   - Storage backends return normalized cloud errors (with provider-agnostic categories) or internal errors (`ErrNotFound`, `ErrReadOnly`, etc.)


Suggested change

- Storage backends return normalized cloud errors (with provider-agnostic categories) or internal errors (`ErrNotFound`, `ErrReadOnly`, etc.)

- The Storage backend may also return internal errors when other logic errors are encountered. These errors will always begin with an `Err` prefix (e.g. `ErrNotFound`, `ErrReadOnly`, `ErrListTruncated`, etc.)

I think this suggestion truncates the normalised errors. I'd say something like 'The storage back end returns normalized cloud errors (with provider-agnostic categories). It may also return other internal errors when other logic errors are encountered

alberto-miranda · 2026-01-20T14:25:27Z

fusion_docs/troubleshooting/error-codes-exit-messages.md

+
+   - Storage backends catch and normalize cloud errors (network timeouts, auth failures, rate limits) using the `clouderr` package
+   - Storage backends return normalized cloud errors (with provider-agnostic categories) or internal errors (`ErrNotFound`, `ErrReadOnly`, etc.)
+   - The FUSE layer maps both cloud errors and internal errors to FUSE status codes (`ENOENT`, `EACCES`, `EREMOTEIO`, `EIO`)


The FUSE layer maps both cloud errors and internal errors to errno status codes (ENOENT, EACCES, EREMOTEIO, EIO, etc.)

alberto-miranda · 2026-01-20T14:25:54Z

fusion_docs/troubleshooting/error-codes-exit-messages.md

+   - Storage backends catch and normalize cloud errors (network timeouts, auth failures, rate limits) using the `clouderr` package
+   - Storage backends return normalized cloud errors (with provider-agnostic categories) or internal errors (`ErrNotFound`, `ErrReadOnly`, etc.)
+   - The FUSE layer maps both cloud errors and internal errors to FUSE status codes (`ENOENT`, `EACCES`, `EREMOTEIO`, `EIO`)
+   - The kernel translates FUSE status to errno values for the application


The kernel communicates the errno values to the application

alberto-miranda · 2026-01-20T15:04:13Z

fusion_docs/troubleshooting/error-codes-exit-messages.md

+| Cloud error category | FUSE status | Examples |
+|---------------------|-------------|----------|
+| `Unauthenticated` | `fuse.EACCES` | No credentials provided |
+| `InvalidCredentials` | `fuse.EACCES` | Wrong, malformed, or expired credentials |
+| `Forbidden` | `fuse.EACCES` | Valid credentials, insufficient permissions |
+| `AccountError` | `fuse.EACCES` | Account disabled, suspended, or has billing issues |
+| `ResourceNotFound` | `fuse.ENOENT` | S3 "NoSuchKey", Azure "BlobNotFound", GCS 404 |
+| `ContainerNotFound` | `fuse.ENOENT` | S3 "NoSuchBucket", Azure "ContainerNotFound", GCS 404 with bucket error |
+| `RateLimited` | `fuse.EBUSY` | Request rate limits exceeded |
+| `Busy` | `fuse.EBUSY` | Service temporarily unavailable or overloaded |
+| `ResourceArchived` | `fuse.EBUSY` | Resource in archived/transitional state (for example, Glacier) |
+| `Conflict` | `fuse.EEXIST` | Resource already exists or precondition failed |
+| `InvalidArgument` | `fuse.EINVAL` | Malformed request or invalid parameters |
+| `QuotaExceeded` | `fuse.EREMOTEIO` | Storage quota or capacity limit reached |
+| `Unknown` (cloud errors) | `fuse.EREMOTEIO` | Unclassified cloud provider errors |


Same changes as I mentioned above:

FUSE status -> errno code

Remove the fuse. prefix.

alberto-miranda · 2026-01-20T15:04:24Z

fusion_docs/troubleshooting/error-codes-exit-messages.md

+| Internal error | FUSE status |
+|---------------|-------------|
+| Not found | `fuse.ENOENT` |
+| Read-only | `fuse.EROFS` |
+| Unsupported | `fuse.ENOSYS` |
+| Context cancelled | `fuse.EINTR` |
+| Context deadline exceeded | `fuse.ETIMEDOUT` |
+| Other errors | `fuse.EIO` |


alberto-miranda · 2026-01-20T15:07:13Z

fusion_docs/troubleshooting/error-codes-exit-messages.md

+## Cloud provider error categories
+
+Fusion normalizes errors from different cloud storage providers (S3, Azure Blob Storage, Google Cloud Storage) into consistent categories. When you see an `error_code` field in Fusion logs, it represents one of these categories:
+
+| Category | Description | Common provider codes |
+|----------|-------------|----------------------|
+| `ResourceNotFound` | Requested resource (object/file) does not exist | S3: "NoSuchKey", Azure: "BlobNotFound", GCS: HTTP 404 |
+| `ContainerNotFound` | Storage container (bucket) does not exist | S3: "NoSuchBucket", Azure: "ContainerNotFound", GCS: HTTP 404 with bucket error |
+| `Unauthenticated` | No credentials provided | S3: "MissingSecurityHeader", GCS: HTTP 401 with no credentials |
+| `InvalidCredentials` | Credentials provided but wrong, malformed, or expired | S3: "InvalidAccessKeyId", "ExpiredToken", Azure: "InvalidAuthenticationInfo", GCS: HTTP 401 with invalid credentials |
+| `Forbidden` | Valid credentials but insufficient permissions | S3: "AccessDenied", Azure: "AuthorizationPermissionMismatch", GCS: HTTP 403 |
+| `AccountError` | Account-level problems (disabled, suspended, billing issues) | S3: "AccountProblem", Azure: "AccountIsDisabled", GCS: HTTP 403 with specific messages |
+| `ResourceArchived` | Resource exists but is in archived/transitional state | S3: "InvalidObjectState" (Glacier), Azure: "BlobArchived" |
+| `RateLimited` | Request rate limits exceeded | S3: "SlowDown", Azure: "TooManyRequests", GCS: HTTP 429 |
+| `Busy` | Service temporarily unavailable or overloaded | S3: "ServiceUnavailable", "InternalError", Azure: "ServerBusy", GCS: HTTP 503 |
+| `Conflict` | Resource state conflict or precondition failure | S3: "BucketAlreadyExists", Azure: "BlobAlreadyExists", "ConditionNotMet", GCS: HTTP 409/412 |
+| `InvalidArgument` | Malformed request or invalid parameters | S3: "InvalidArgument", "InvalidRange", Azure: "InvalidQueryParameterValue", GCS: HTTP 400 |
+| `QuotaExceeded` | Storage quota or capacity limit reached | S3: "TooManyBuckets", Azure: "AccountLimitExceeded", GCS: HTTP 429 with quota message |
+| `Unknown` | Unclassified or unexpected error | Various |


I know we wrote most of this, but it seems to provide similar information to the table above: maybe a unified table would work better?

alberto-miranda · 2026-01-20T15:08:24Z

fusion_docs/troubleshooting/error-codes-exit-messages.md

+## Nextflow integration
+
+When running Nextflow with Fusion:
+
+- Exit code `0`: Task completed successfully
+- Exit code `127`: Retry logic activates (`.command.sh` not found)
+- Exit code `174`: Fusion I/O error—check logs for details
+
+### Check exit codes
+
+```bash
+fusion --foreground
+exit_code=$?
+
+case $exit_code in
+    0)
+        echo "Success"
+        ;;
+    127)
+        echo "Command not found - may retry"
+        ;;
+    174)
+        echo "Fusion I/O error - check Fusion logs"
+        ;;
+    *)
+        echo "Unknown exit code: $exit_code"
+        ;;
+esac
+```


I don't think this section is needed.

alberto-miranda · 2026-01-20T15:12:45Z

fusion_docs/troubleshooting/error-codes-exit-messages.md

+| `mounting filesystem` | Failed to mount FUSE filesystem |
+| `could not get current job attempt` | Failed to get job attempt from compute environment |
+
+## Understanding Fusion logs


This section is rather important both to support and users, thus I would put it after the "Triaging errors" section. As a matter of fact, most of the sections in the document are for reference purposes, and since we have a specific Reference document, it might be interesting to move them there and cross-reference them here as needed.

Agreed I would actually have this as the first section (though after triaging also works)

Co-authored-by: Alberto Miranda <alberto.miranda@protonmail.com> Signed-off-by: Justine Geffen <justinegeffen@users.noreply.github.com>

Co-authored-by: Cristian Ramon-Cortes <cristianrcv@users.noreply.github.com> Signed-off-by: Justine Geffen <justinegeffen@users.noreply.github.com>

MichaelTansiniSeqera · 2026-01-21T11:41:38Z

fusion_docs/troubleshooting/error-codes-exit-messages.md

+
+1. **Failures during startup/shutdown → Exit Code**
+
+   - Startup: Configuration errors, missing credentials, or mount failures terminate Fusion immediately


Either a comma after failures, or can we put will terminate Fusion immediately for clarity?

Testing

0a0280d

llewellyn-sl added 3 commits January 14, 2026 11:09

Add remaining user-facing error reference, lint for GD style guide co…

39aff27

…mpliance

Remove related docs links from page bottom

47e0a48

Delete first draft — changes incorporated

2263543

MichaelTansiniSeqera reviewed Jan 16, 2026

View reviewed changes

fusion_docs/troubleshooting/error-codes-exit-messages.md Outdated Show resolved Hide resolved

MichaelTansiniSeqera reviewed Jan 16, 2026

View reviewed changes

fusion_docs/troubleshooting/error-codes-exit-messages.md Outdated Show resolved Hide resolved

justinegeffen reviewed Jan 19, 2026

View reviewed changes

fusion_docs/troubleshooting/error-codes-exit-messages.md Outdated Show resolved Hide resolved

justinegeffen requested a review from alberto-miranda January 19, 2026 08:13

justinegeffen added 1. Editor review Needs a language review 1. Dev/PM/SME Needs a review by a Dev/PM/SME labels Jan 19, 2026

MichaelTansiniSeqera reviewed Jan 19, 2026

View reviewed changes

fusion_docs/troubleshooting/error-codes-exit-messages.md Outdated Show resolved Hide resolved

justinegeffen marked this pull request as ready for review January 20, 2026 13:50

justinegeffen added 3 commits January 20, 2026 15:50

Merge branch 'master' into chris-fusion-troubleshooting

15002b1

Update error codes documentation and last updated date

c6a91f6

Updated last updated date and removed architectural limitations section. Signed-off-by: Justine Geffen <justinegeffen@users.noreply.github.com>

Apply suggestion from @justinegeffen

f087c26

Signed-off-by: Justine Geffen <justinegeffen@users.noreply.github.com>

cristianrcv approved these changes Jan 20, 2026

View reviewed changes

alberto-miranda approved these changes Jan 20, 2026

View reviewed changes

justinegeffen and others added 3 commits January 21, 2026 00:50

Update fusion_docs/troubleshooting/error-codes-exit-messages.md

b13d64c

Co-authored-by: Alberto Miranda <alberto.miranda@protonmail.com> Signed-off-by: Justine Geffen <justinegeffen@users.noreply.github.com>

Apply suggestion from @alberto-miranda

a221275

Co-authored-by: Alberto Miranda <alberto.miranda@protonmail.com> Signed-off-by: Justine Geffen <justinegeffen@users.noreply.github.com>

Update fusion_docs/troubleshooting/error-codes-exit-messages.md

911dd2f

Co-authored-by: Cristian Ramon-Cortes <cristianrcv@users.noreply.github.com> Signed-off-by: Justine Geffen <justinegeffen@users.noreply.github.com>

MichaelTansiniSeqera reviewed Jan 21, 2026

View reviewed changes

	- Fusion logs cloud errors with structured details (provider, error code, HTTP status, request ID)
	- Fusion logs cloud errors with structured details (i.e., provider, error code, HTTP status, request ID)


		## FUSE status codes

		Fusion maps internal errors to standard FUSE status codes returned to the operating system. These are the [errno](https://man7.org/linux/man-pages/man3/errno.3.html) values applications receive when filesystem operations fail.

	- Storage backends return normalized cloud errors (with provider-agnostic categories) or internal errors (`ErrNotFound`, `ErrReadOnly`, etc.)
	- The Storage backend may also return internal errors when other logic errors are encountered. These errors will always begin with an `Err` prefix (e.g. `ErrNotFound`, `ErrReadOnly`, `ErrListTruncated`, etc.)


		1. Failures during startup/shutdown → Exit Code

		- Startup: Configuration errors, missing credentials, or mount failures terminate Fusion immediately

[Fusion] Troubleshooting error codes #1007

Are you sure you want to change the base?

[Fusion] Troubleshooting error codes #1007

Conversation

christopher-hakkaart commented Jan 12, 2026

Uh oh!

netlify bot commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for seqera-docs ready!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

netlify bot commented Jan 12, 2026 •

edited

Loading