Skip to content

Conversation

@yeazelm
Copy link
Contributor

@yeazelm yeazelm commented Jan 20, 2026

Description of changes:

The CONFIG_EXT4_DEBUG (kernel docs) and CONFIG_JBD2_DEBUG (kernel docs) options are focused on filesystem development. They can cause additional latency in some situations so disabling them helps to remove potential paths of latency. See testing for how the p9999 latency can be improved with these changes.

Testing done:
Before the changes, running a fio test from a container results in large jumps in latency for the 99.99 and 99.999 latency.

99.0000th=570
99.9000th=791
99.9900th=81265
99.9990th=112722
99.9999th=112722

Details
kubectl exec -it fio-test-pod -n kube-system -- fio --name=test --ioengine=psync --rw=write --bs=4k --numjobs=1 --iodepth=1 --filename=/testdata/testfile --direct=1 --fsync=1 --size
=1G --runtime=60 --percentile_list=99:99.9:99.99:99.999:99.9999
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
test: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=3444KiB/s][w=861 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=682: Thu Jan 15 21:35:16 2026
  write: IOPS=854, BW=3417KiB/s (3499kB/s)(200MiB/60001msec); 0 zone resets
    clat (usec): min=243, max=112372, avg=383.46, stdev=1421.76
     lat (usec): min=243, max=112372, avg=383.54, stdev=1421.76
    clat percentiles (usec):
     | 99.0000th=[   570], 99.9000th=[   791], 99.9900th=[ 81265],
     | 99.9990th=[112722], 99.9999th=[112722]
   bw (  KiB/s): min= 2824, max= 3848, per=100.00%, avg=3420.10, stdev=270.84, samples=119
   iops        : min=  706, max=  962, avg=855.03, stdev=67.71, samples=119
  lat (usec)   : 250=0.02%, 500=96.92%, 750=2.90%, 1000=0.09%
  lat (msec)   : 2=0.03%, 50=0.01%, 100=0.03%, 250=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=533, max=111348, avg=786.33, stdev=2033.61
    sync percentiles (usec):
     | 99.0000th=[  1090], 99.9000th=[  1827], 99.9900th=[ 94897],
     | 99.9990th=[111674], 99.9999th=[111674]
  cpu          : usr=0.19%, sys=0.89%, ctx=102559, majf=0, minf=13
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,51259,0,51259 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=3417KiB/s (3499kB/s), 3417KiB/s-3417KiB/s (3499kB/s-3499kB/s), io=200MiB (210MB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=0/153546, merge=0/102376, ticks=0/54286, in_queue=54285, util=90.30%

After the changes, we see significantly better results.

99.0000th=783
99.9000th=1975
99.9900th=5669
99.9990th=9372
99.9999th=9372

Details
kubectl exec -it fio-test-pod -n kube-system -- fio --name=test --ioengine=psync --rw=write --bs=4k --numjobs=1 --iodepth=1 --filename=/testdata/testfile --direct=1 --fsync=1 --size=1G --runtime=60 --percentile_list=99:99.9:99.99:99.999:99.9999
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=2438KiB/s][w=609 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=682: Fri Jan 16 20:43:21 2026
  write: IOPS=1004, BW=4016KiB/s (4112kB/s)(235MiB/60001msec); 0 zone resets
    clat (usec): min=388, max=9371, avg=522.63, stdev=130.84
     lat (usec): min=388, max=9371, avg=522.69, stdev=130.84
    clat percentiles (usec):
     | 99.0000th=[  783], 99.9000th=[ 1975], 99.9900th=[ 5669],
     | 99.9990th=[ 9372], 99.9999th=[ 9372]
   bw (  KiB/s): min= 1592, max= 7232, per=100.00%, avg=4030.32, stdev=2145.31, samples=119
   iops        : min=  398, max= 1808, avg=1007.58, stdev=536.33, samples=119
  lat (usec)   : 500=47.63%, 750=50.79%, 1000=1.33%
  lat (msec)   : 2=0.15%, 4=0.07%, 10=0.03%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=308, max=6165.3k, avg=472752.41, stdev=550331.68
    sync percentiles (usec):
     | 99.0000th=[ 1401], 99.9000th=[ 2180], 99.9900th=[ 5276],
     | 99.9990th=[ 6194], 99.9999th=[ 6194]
  cpu          : usr=0.19%, sys=0.57%, ctx=86398, majf=0, minf=13
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,60242,0,60241 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=4016KiB/s (4112kB/s), 4016KiB/s-4016KiB/s (4112kB/s-4112kB/s), io=235MiB (247MB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=0/112372, merge=0/50100, ticks=0/59070, in_queue=59070, util=98.35%

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

The CONFIG_EXT4_DEBUG and CONFIG_JBD2_DEBUG options are focused on
filesystem development. They can cause additional latency in some
situations so disabling them helps to remove potential paths of latency.

Signed-off-by: Matthew Yeazel <yeazelm@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants