Skip to content

Conversation

@QuiltSimon
Copy link

@QuiltSimon QuiltSimon commented Dec 15, 2025

Summary

Adds two new troubleshooting knowledge base articles based on a customer support case involving:

  • DNS resolution failures in S3 proxy when using custom VPC DHCP options
  • JSON encoding errors masking underlying S3 permission issues

Articles Added

1. trouble-dns-resolution-s3-proxy.md

Title: DNS Resolution Issues with S3 Proxy in Private VPC Deployments

Covers:

  • Root cause: S3 proxy container reads DNS from /etc/resolv.conf, which may not include AWS DNS (169.254.169.253) when custom DHCP options are configured
  • Symptoms: S3 proxy fails to resolve internal registry hostname or AWS S3 endpoints
  • Fixes:
    • Add AWS DNS to VPC DHCP options
    • Configure DNS forwarding for AWS domains
  • Debugging steps with ECS Exec, CloudWatch logs, VPC attribute checks

Tags: dns, s3-proxy, ecs, network, private-vpc, awsvpc, troubleshooting

2. trouble-json-encoding-error-hiding-permissions.md

Title: JSON Encoding Error Masking Underlying Permission Issues

Covers:

  • Root cause: AWS S3 returns XML error responses (e.g., AccessDenied), but error handling attempts JSON parsing, masking the original error
  • Symptoms: Generic JSONDecodeError instead of permission error messages
  • Fixes:
    • IAM permission verification steps
    • Bucket policy and VPC endpoint policy examples
    • Cross-account access configuration
  • Debugging with CloudTrail and S3 access logs

Tags: permissions, iam, s3, error-handling, debugging, s3-proxy, troubleshooting

Customer Feature Requests (tracked separately)

This support case also identified two feature requests:

  1. Custom DNS servers parameter for ECS task definitions (not yet implemented)
  2. Improved error handling to preserve original AWS error messages (not yet implemented)

These will be tracked as separate GitHub issues in the appropriate repos.

Test Plan

  • Review article formatting in HubSpot preview
  • Verify all CLI commands are correct
  • Check that links to AWS documentation are valid

Note

Adds two troubleshooting KB articles on S3 proxy DNS resolution failures in private VPCs and JSON parsing errors that obscure S3 permission issues, with fixes and debugging steps.

  • Docs:
    • New KB trouble-dns-resolution-s3-proxy.md:
      • Causes: custom DHCP excluding AWS DNS; private DNS on VPC endpoints; Cloud Map resolution failures.
      • Fixes: add AWS DNS to DHCP options; DNS forwarding via Route 53 Resolver.
      • Debugging: ECS Exec to inspect /etc/resolv.conf; CloudWatch logs; VPC DNS attributes; DHCP options.
      • Notes: future parameter to pass custom DNS servers to ECS tasks.
    • New KB trouble-json-encoding-error-hiding-permissions.md:
      • Issue: XML S3 errors parsed as JSON → JSONDecodeError masking AccessDenied.
      • Guidance: verify IAM roles; direct S3 access tests; enable S3 access logging; check CloudTrail.
      • Policy examples: bucket policy, VPC endpoint policy, cross-account setup.
      • Debugging: registry/proxy logs; CLI reproduction; future improvement to preserve/parse AWS errors.

Written by Cursor Bugbot for commit 8af197a. This will update automatically on new commits. Configure here.

Add two new troubleshooting articles based on customer support case:

1. DNS Resolution Issues with S3 Proxy in Private VPC Deployments
   - Documents issue where custom DHCP options exclude AWS DNS
   - Provides workarounds (DHCP options, DNS forwarding)
   - Includes debugging steps for ECS Exec, CloudWatch logs

2. JSON Encoding Error Masking Underlying Permission Issues
   - Documents how XML error responses from S3 can mask permission errors
   - Provides IAM debugging steps using CloudTrail and S3 access logs
   - Includes example bucket policies and common fixes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants