Guide: Using Short DNS TTLs for Dynamic Backend DNS Changes During Load Tests

[cbaugus]

Problem

During long-running load tests (especially RampRps and DailyTraffic models), the load testing agents cache DNS resolutions through:

1. HTTP connection pooling (connections are reused without re-resolving DNS)
2. System-level DNS resolver caching (respects DNS TTL values)

This means when backend DNS records change (e.g., during a blue/green deployment, failover, or infrastructure migration), the load test agents may continue sending traffic to the old IP addresses until connections expire or DNS cache TTLs expire.

Solution: Configure Short DNS TTLs

By setting a short DNS Time-To-Live (TTL) on your backend DNS records, you can ensure that even if the load testing agents cache DNS results, they will refresh more frequently.

How DNS TTL Works

When a DNS query is made, the DNS server returns not just the IP address, but also a TTL value (in seconds) that tells clients how long they can cache this result. Common TTL values:

• Default/Production: 300-3600 seconds (5 minutes to 1 hour)
• Short TTL for testing: 30-60 seconds
• Very short TTL: 5-15 seconds (use sparingly due to increased DNS query load)

Implementation Steps

1. For AWS Route53:

aws route53 change-resource-record-sets --hosted-zone-id YOUR_ZONE_ID --change-batch '{
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "your-backend.example.com",
      "Type": "A",
      "TTL": 60,
      "ResourceRecords": [{"Value": "192.168.1.100"}]
    }
  }]
}'

2. For Cloudflare:

• Navigate to DNS settings for your domain
• Edit the A/AAAA record for your backend
• Set "TTL" to "1 minute" (60 seconds) or "Auto" with proxy disabled
• Save changes

3. For Google Cloud DNS:

gcloud dns record-sets update your-backend.example.com \
  --zone=YOUR_ZONE_NAME \
  --type=A \
  --ttl=60 \
  --rrdatas="192.168.1.100"

4. For your own DNS server (BIND):

your-backend.example.com. 60 IN A 192.168.1.100

The 60 is the TTL in seconds.

Best Practices

1. Pre-test preparation: Set short TTLs before starting your load test (ideally 2x the old TTL duration before the test)

2. TTL recommendations by test duration:

   • Short tests (<30 min): 60 seconds
   • Medium tests (1-4 hours): 120 seconds
   • Long tests (>4 hours): 300 seconds

3. Post-test: Return TTL to normal production values after testing to reduce DNS query load

4. Trade-offs:

   • ✅ Faster DNS change propagation
   • ✅ More responsive to backend changes
   • ❌ Increased DNS query load on your DNS infrastructure
   • ❌ May increase latency slightly due to more frequent DNS lookups

Verification

You can verify your DNS TTL is set correctly:

# Check current TTL
dig +nocmd +noall +answer your-backend.example.com

# Output will show:
# your-backend.example.com. 60 IN A 192.168.1.100
#                           ^^
#                           This is your TTL in seconds

Limitations with This Load Tester

Even with short DNS TTLs, there are still limitations due to connection pooling:

1. Persistent connections: If HTTP keep-alive connections remain open, they won't trigger new DNS lookups regardless of TTL
2. Connection pool: reqwest maintains a connection pool that reuses connections to the same IP

For guaranteed immediate DNS change pickup, consider:

• Using RESOLVE_TARGET_ADDR to manually control routing during tests
• Implementing connection recycling in a future enhancement
• Running multiple shorter test cycles instead of one long test

Related Configuration

This approach works alongside the existing RESOLVE_TARGET_ADDR option, which completely bypasses DNS:

# This hardcodes DNS resolution (no TTL applies)
-e RESOLVE_TARGET_ADDR="example.com:192.168.1.50:8080"

Additional Resources

• AWS Route53 TTL Documentation
• Cloudflare DNS TTL Best Practices
• DNS TTL Explained (RFC 1035)

---

Type: Documentation/Guide
Priority: Medium
Related to: Load testing, DNS resolution, backend infrastructure changes