Page allocation failure

[shinge]

After running the Linux around a day, there is a page allocation failure as below. During this period, I am forwarding a stream from UART to a remote server using nc. Please help me to fix it and my application needs to run 24/7.

E (86428606) wifi:null wpa_sta_ap_set_rsnxe
E (86437956) wifi:null wpa_sta_ap_set_rsnxe
[86554.227453] irq/5-600c0004.: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null)
[86554.228646] CPU: 0 PID: 29 Comm: irq/5-600c0004. Not tainted 6.8.0 #1
[86554.233113] Stack:
[86554.235243] > 00000100 00000000 3dbfbc00 423b8f69
[86554.239881] 423b8f69 3d87d11b 3dbfc000 3d8850a4
[86554.244551] 8217cfcf 3dbfbc50 00000000 3dbfc350
[86554.249458] 3dbfbc30 00000004 00000830 00060f00
[86554.254156] 8217d4c9 3dbfbc70 00000820 00000000
[86554.258700] > 3dbfbc8c 00000820 3dbfbca0 00000000
[86554.263303] 8217d525 3dbfbcf0 00000000 00000000
[86554.267990] > 00000000 000007c8 3d9574a0 00000110
[86554.272683] 3dbfbcd0 3dbfbcb0 0000000c 423b88b0
[86554.277369] 3dbfbc80 3d87d698 00000820 3d87d448
[86554.282043] 00000820 3dbfbcd0 3d86c0fc 00000000
[86554.286736] 3dbfbe90 3dbfbe50 00000000 00000000
[86554.291421] 3d87d698 3dbfa000 00000000 3d8c2a60
[86554.296094] 00000001 00000000 00000820 00000000
[86554.300809] 8217d56d 3dbfbd70 00000000 00000820
[86554.305482] > 0000000f 3d8c0002 3dbfc000 3d870204
[86554.310195] Call Trace:
[86554.312552] [<42372119>] 0x42372119
[86554.316149] [<4217cfcf>] 0x4217cfcf
[86554.319680] [<4217d4c9>] 0x4217d4c9
[86554.323216] [<4217d525>] 0x4217d525
[86554.326772] [<4217d56d>] 0x4217d56d
[86554.330325] [<4227be0d>] 0x4227be0d
[86554.333877] [<42268c02>] 0x42268c02
[86554.337449] [<4226b0b1>] 0x4226b0b1
[86554.341152] [<4225d3cc>] 0x4225d3cc
[86554.344572] [<42144d48>] 0x42144d48
[86554.348140] [<4213504e>] 0x4213504e
[86554.351673] [<421209b4>] 0x421209b4
[86554.356342] Mem-Info:
[86554.357545] active_anon:0 inactive_anon:0 isolated_anon:0
[86554.357545] active_file:1 inactive_file:1 isolated_file:0
[86554.357545] unevictable:15 dirty:0 writeback:0
[86554.357545] slab_reclaimable:178 slab_unreclaimable:1105
[86554.357545] mapped:0 shmem:0 pagetables:0
[86554.357545] sec_pagetables:0 bounce:0
[86554.357545] kernel_misc_reclaimable:0
[86554.357545] free:33 free_pcp:0 free_cma:0
[86554.394913] Node 0 active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:60kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:336kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? yes
[86554.419561] Normal free:132kB boost:0kB min:340kB low:424kB high:508kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:4kB unevictable:60kB writepending:0kB present:8192kB managed:7328kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[86554.445055] lowmem_reserve[]: 0 0
[86554.447426] Normal: 334kB (U) 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 132kB
[86554.459128] 17 total pagecache pages
[86554.461577] 2048 pages RAM
[86554.464143] 0 pages HighMem/MovableOnly
[86554.468877] 216 pages reserved
[86554.490664] syslogd invoked oom-killer: gfp_mask=0x140cc0(GFP_USER|__GFP_COMP), order=0, oom_score_adj=0
[86554.491709] CPU: 0 PID: 52 Comm: syslogd Not tainted 6.8.0 #1
[86554.495628] Stack:
[86554.497776] > 00000100 00000000 3de8f9d0 423b8f69
[86554.502400] 423b8f69 3d87d11b 3dbff180 3d8850a4
[86554.507072] 8236fe74 3de8fa20 0000005c 3dbff4d0
[86554.511750] 3de8fa00 00000004 00000000 00060800
[86554.516445] 8216e6a7 3de8fa40 3de8fb10 4237b088
[86554.521121] > 00140cc0 3de8fb1c 00000000 00000000
[86554.525824] 8216ea7e 3de8fa90 3de8fb10 423b62c8
[86554.530504] > 3de8fa3c 00000001 3de8e000 3d88551c
[86554.535186] 3d87d70c 00000000 00000001 00000001
[86554.539870] 00000000 00000020 3de8e000 00000003
[86554.544565] 0000000a 00060200 3dee6400 3d87024c
[86554.549256] 8217d391 3de8fae0 3de8fb10 7fffffff
[86554.553925] > 00000020 00000000 00000000 00000000
[86554.558621] 00000000 ff001078 00000000 00140cc0
[86554.563313] 00000000 3d87d698 001408c0 3d87d448
[86554.568000] 3de8fae0 3d8816c0 3dee6400 00000000
[86554.572611] Call Trace:
[86554.575054] [<42372119>] 0x42372119
[86554.578612] [<4236fe74>] 0x4236fe74
[86554.582319] [<4216e6a7>] 0x4216e6a7
[86554.585733] [<4216ea7e>] 0x4216ea7e
[86554.589309] [<4217d391>] 0x4217d391
[86554.592839] [<4217d8cc>] 0x4217d8cc
[86554.596400] [<4216b88d>] 0x4216b88d
[86554.599966] [<4216d2e4>] 0x4216d2e4
[86554.603515] [<4216d3a9>] 0x4216d3a9
[86554.607104] [<421897f7>] 0x421897f7
[86554.610624] [<42189f65>] 0x42189f65
[86554.614194] [<4218a222>] 0x4218a222
[86554.617788] [<4218a447>] 0x4218a447
[86554.621300] [<4218ae38>] 0x4218ae38
[86554.624860] [<4218b419>] 0x4218b419
[86554.628445] [<421810e7>] 0x421810e7
[86554.631978] [<42181172>] 0x42181172
[86554.635535] [<42181188>] 0x42181188
[86554.639131] [<421208e0>] 0x421208e0
[86554.642671] [<4212066a>] 0x4212066a
[86554.678594] Mem-Info:
[86554.679287] active_anon:0 inactive_anon:0 isolated_anon:0
[86554.679287] active_file:0 inactive_file:1 isolated_file:0
[86554.679287] unevictable:15 dirty:0 writeback:0
[86554.679287] slab_reclaimable:178 slab_unreclaimable:1106
[86554.679287] mapped:0 shmem:0 pagetables:0
[86554.679287] sec_pagetables:0 bounce:0
[86554.679287] kernel_misc_reclaimable:0
[86554.679287] free:33 free_pcp:0 free_cma:0
[86554.710744] Node 0 active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:60kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:336kB pagetables:0kB sec_pagetables:0kB all_unreclaimable? no
[86554.743198] Normal free:136kB boost:0kB min:340kB low:424kB high:508kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:60kB writepending:0kB present:8192kB managed:7328kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB

[jcmvbkbc]

I am forwarding a stream from UART to a remote server using nc.

I'm curious how big that stream was and what network protocols were used?

irq/5-600c0004.: page allocation failure: order:0, mode:0x820(GFP_ATOMIC)

This must be the wifi IPC driver trying to allocate skb for the packet coming from wifi. The choice at this point is either allocate skb dynamically as packets arrive or use a preallocated pool of skbs. I have implemented the former and this kind of error is the expected consequence. The latter wouldn't have this issue, but since there's no way to tell the firmware to pause reception it would lose packets in this situation instead. Also the skbs taken from the pool need to be returned back into it, and that means changes in the esp-hosted driver.

[shinge]

I tested the commit 8678b10

reference this issue. But, the page error still exist. The data size for a day is around 430MB per day. Will new tag xtensa-20240502 solve this issue?

[jcmvbkbc]

I tested the commit 8678b10

It is entirely unrelated to this issue. github is linking it to this issue because the commit description has the text #17 in it.

Will new tag xtensa-20240502 solve this issue?

I don't think so.