Skip to content

Conversation

@marmarek
Copy link
Member

@marmarek marmarek commented Jan 6, 2026

Not only in Fedora.
The issue happens on Debian recently, and the default timeout of 90s is
longer than default VM shutdown timeout of 60s - which means:

  1. slow user session stop may prevent other services stopping properly
    before shutdown timeout expires
  2. logs do not include stop timeout message that normally tells which
    processes remained running - such information would help debugging
    the issue

The timeout applied in Fedora is really short (500ms), make it longer
(5s) but apply everywhere.

@marmarek marmarek force-pushed the shutdown-timeout branch 2 times, most recently from dfbc213 to fca100a Compare January 7, 2026 02:19
Not only in Fedora.
The issue happens on Debian recently, and the default timeout of 90s is
longer than default VM shutdown timeout of 60s - which means:
1. slow user session stop may prevent other services stopping properly
   before shutdown timeout expires
2. logs do not include stop timeout message that normally tells which
   processes remained running - such information would help debugging
   the issue

The timeout applied in Fedora is really short (500ms), make it longer
(5s) but apply everywhere.
@codecov
Copy link

codecov bot commented Jan 7, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.89%. Comparing base (f0089ce) to head (ebf117a).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #629      +/-   ##
==========================================
- Coverage   70.30%   69.89%   -0.41%     
==========================================
  Files           3        3              
  Lines         495      495              
==========================================
- Hits          348      346       -2     
- Misses        147      149       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@qubesos-bot
Copy link

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2026010704-4.3&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025111104-4.3&flavor=update

  • system_tests_network_ipv6

    • VmIPv6Networking_debian-13-xfce: test_211_custom_ip_proxy (failure)
      ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 1 != 0
  • system_tests_qwt_win10_seamless@hw13

  • system_tests_gui_tools

    • qubesmanager_manager: unnamed test (unknown)
    • qubesmanager_manager: Failed (test died)
      # Test died: no candidate needle with tag(s) 'qube-manager-rclicked...
  • system_tests_guivm_gui_interactive

    • collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/p9vOK-\d+-/...

    • collect_logs: Failed (test died + timed out)
      # Test died: command 'curl --form upload=@journalctl.log --form upn...

    • collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/qNMdR-\d+-/...

  • system_tests_dispvm_perf@hw7

  • system_tests_gui_tools@hw7

    • qubesmanager_manager: unnamed test (unknown)
    • qubesmanager_manager: Failed (test died)
      # Test died: no candidate needle with tag(s) 'qube-manager-rclicked...

Failed tests

12 failures
  • system_tests_network_ipv6

    • VmIPv6Networking_debian-13-xfce: test_211_custom_ip_proxy (failure)
      ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 1 != 0
  • system_tests_qwt_win10_seamless@hw13

    • windows_clipboard_and_filecopy: unnamed test (unknown)
    • windows_clipboard_and_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'windows-Edge-address-...
  • system_tests_qwt_win11@hw13

    • windows_install: Failed (test died)
      # Test died: Install failed with code 1 at qubesos/tests/windows_in...
  • system_tests_gui_tools

    • qubesmanager_manager: unnamed test (unknown)
    • qubesmanager_manager: Failed (test died)
      # Test died: no candidate needle with tag(s) 'qube-manager-rclicked...
  • system_tests_guivm_gui_interactive

    • collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/p9vOK-\d+-/...

    • collect_logs: Failed (test died + timed out)
      # Test died: command 'curl --form upload=@journalctl.log --form upn...

    • collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/qNMdR-\d+-/...

  • system_tests_dispvm_perf@hw7

  • system_tests_gui_tools@hw7

    • qubesmanager_manager: unnamed test (unknown)
    • qubesmanager_manager: Failed (test died)
      # Test died: no candidate needle with tag(s) 'qube-manager-rclicked...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/158999#dependencies

21 fixed
  • system_tests_dispvm

    • system_tests: Fail (unknown)
      Tests qubes.tests.integ.dispvm failed (exit code 1), details report...

    • system_tests: Failed (test died)
      # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...

    • TC_20_DispVM_whonix-workstation-18: test_030_edit_file (failure)
      AssertionError: Timeout waiting for editor window

    • TC_20_DispVM_whonix-workstation-18: test_100_open_in_dispvm (failure)
      AssertionError: './open-file test.txt' failed with ./open-file test...

  • system_tests_qwt_win10@hw13

    • windows_install: wait_serial (wait serial expected)
      # wait_serial expected: qr/kauF4-\d+-/...

    • windows_install: Failed (test died + timed out)
      # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

  • system_tests_qwt_win10_seamless@hw13

  • system_tests_qwt_win11@hw13

    • windows_install: wait_serial (wait serial expected)
      # wait_serial expected: qr/iDVvW-\d+-/...
  • system_tests_extra

    • system_tests: Fail (unknown)
      Tests qubes.tests.extra failed (exit code 1), details reported sepa...

    • system_tests: Failed (test died)
      # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...

    • TC_00_QVCTest_debian-13-xfce: test_010_screenshare (failure + cleanup)
      AssertionError: 2.4614345149565264 not less than 2.0

    • TC_00_QVCTest_whonix-gateway-18: test_010_screenshare (failure)
      AssertionError: 1 != 0 : Timeout waiting for /dev/video0 in test-in...

    • TC_00_QVCTest_whonix-workstation-18: test_010_screenshare (failure)
      AssertionError: 1 != 0 : Timeout waiting for /dev/video0 in test-in...

  • system_tests_guivm_vnc_gui_interactive

    • gui_filecopy: unnamed test (unknown)
    • gui_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'disp-text-editor' mat...
  • system_tests_guivm_gui_interactive

    • gui_keyboard_layout: unnamed test (unknown)
    • gui_keyboard_layout: Failed (test died)
      # Test died: no candidate needle with tag(s) 'work-xterm, work-xter...
  • system_tests_dispvm_perf@hw7

    • TC_00_DispVMPerf_debian-13-xfce: test_411_dom0_dispvm_preload_gui_concurrent_api (failure)
      AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

    • TC_00_DispVMPerf_debian-13-xfce: test_900_reader (failure)
      AssertionError: '/usr/lib/qubes/tests/dispvm_perf_reader.py --templ...

    • TC_00_DispVMPerf_whonix-workstation-18: test_409_dom0_dispvm_preload_gui_api (failure)
      AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...

    • TC_00_DispVMPerf_whonix-workstation-18: test_900_reader (failure)
      AssertionError: '/usr/lib/qubes/tests/dispvm_perf_reader.py --templ...

Unstable tests

Details
  • system_tests_gui_interactive

    collect_logs/Failed (1/5 times with errors)
    • job 163926 # Test died: command 'curl --form upload=@journalctl.log --form upn...
    collect_logs/wait_serial (1/5 times with errors)
    • job 163926 # wait_serial expected: qr/yCLOp-\d+-/...
    collect_logs/wait_serial (1/5 times with errors)
    • job 163926 # wait_serial expected: qr/S0w8Z-\d+-/...
  • system_tests_gui_interactive@hw7

    collect_logs/Failed (1/5 times with errors)
    • job 163926 # Test died: command 'curl --form upload=@journalctl.log --form upn...
    collect_logs/wait_serial (1/5 times with errors)
    • job 163926 # wait_serial expected: qr/yCLOp-\d+-/...
    collect_logs/wait_serial (1/5 times with errors)
    • job 163926 # wait_serial expected: qr/S0w8Z-\d+-/...

Performance Tests

Performance degradation:

16 performance degradations
  • debian-13-xfce_exec: 8.42 🔻 ( previous job: 6.42, degradation: 131.14%)
  • dom0_root_seq1m_q8t1_read 3:read_bandwidth_kb: 128380.00 🔻 ( previous job: 365230.00, degradation: 35.15%)
  • dom0_root_seq1m_q1t1_read 3:read_bandwidth_kb: 59175.00 🔻 ( previous job: 110619.00, degradation: 53.49%)
  • dom0_root_seq1m_q1t1_write 3:write_bandwidth_kb: 59167.00 🔻 ( previous job: 130397.00, degradation: 45.37%)
  • dom0_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 34981.00 🔻 ( previous job: 39923.00, degradation: 87.62%)
  • dom0_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 569.00 🔻 ( previous job: 3264.00, degradation: 17.43%)
  • dom0_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 10073.00 🔻 ( previous job: 12000.00, degradation: 83.94%)
  • dom0_varlibqubes_rnd4k_q32t1_write 3:write_bandwidth_kb: 8585.00 🔻 ( previous job: 10236.00, degradation: 83.87%)
  • fedora-42-xfce_root_seq1m_q8t1_write 3:write_bandwidth_kb: 66232.00 🔻 ( previous job: 137220.00, degradation: 48.27%)
  • fedora-42-xfce_root_seq1m_q1t1_read 3:read_bandwidth_kb: 274065.00 🔻 ( previous job: 355690.00, degradation: 77.05%)
  • fedora-42-xfce_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 70608.00 🔻 ( previous job: 85360.00, degradation: 82.72%)
  • fedora-42-xfce_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 1509.00 🔻 ( previous job: 3964.00, degradation: 38.07%)
  • fedora-42-xfce_private_seq1m_q8t1_write 3:write_bandwidth_kb: 74376.00 🔻 ( previous job: 108858.00, degradation: 68.32%)
  • fedora-42-xfce_private_rnd4k_q32t1_write 3:write_bandwidth_kb: 1677.00 🔻 ( previous job: 1909.00, degradation: 87.85%)
  • fedora-42-xfce_volatile_seq1m_q1t1_read 3:read_bandwidth_kb: 228100.00 🔻 ( previous job: 349176.00, degradation: 65.33%)
  • fedora-42-xfce_dom0-vm-api (mean:0.037): 0.45 🔻 ( previous job: 0.39, degradation: 114.03%)

Remaining performance tests:

94 tests
  • debian-13-xfce_exec-root: 27.47 🟢 ( previous job: 27.48, improvement: 99.96%)
  • debian-13-xfce_socket: 8.32 🟢 ( previous job: 8.68, improvement: 95.85%)
  • debian-13-xfce_socket-root: 7.85 🟢 ( previous job: 8.40, improvement: 93.46%)
  • debian-13-xfce_exec-data-simplex: 65.35 🟢 ( previous job: 67.70, improvement: 96.53%)
  • debian-13-xfce_exec-data-duplex: 69.94 🔻 ( previous job: 69.20, degradation: 101.06%)
  • debian-13-xfce_exec-data-duplex-root: 81.88 🟢 ( previous job: 86.64, improvement: 94.50%)
  • debian-13-xfce_socket-data-duplex: 134.40 🟢 ( previous job: 136.31, improvement: 98.60%)
  • fedora-42-xfce_exec: 9.12 🟢 ( previous job: 9.23, improvement: 98.82%)
  • fedora-42-xfce_exec-root: 58.87 🟢 ( previous job: 59.89, improvement: 98.29%)
  • fedora-42-xfce_socket: 7.95 🟢 ( previous job: 8.23, improvement: 96.56%)
  • fedora-42-xfce_socket-root: 8.47 🔻 ( previous job: 8.31, degradation: 101.97%)
  • fedora-42-xfce_exec-data-simplex: 62.54 🟢 ( previous job: 66.18, improvement: 94.51%)
  • fedora-42-xfce_exec-data-duplex: 70.68 🟢 ( previous job: 71.51, improvement: 98.85%)
  • fedora-42-xfce_exec-data-duplex-root: 100.30 🔻 ( previous job: 96.52, degradation: 103.92%)
  • fedora-42-xfce_socket-data-duplex: 135.16 🟢 ( previous job: 137.78, improvement: 98.10%)
  • whonix-gateway-18_exec: 8.13 🟢 ( previous job: 8.52, improvement: 95.49%)
  • whonix-gateway-18_exec-root: 119.80 🟢 ( previous job: 129.05, improvement: 92.83%)
  • whonix-gateway-18_socket: 8.24 🟢 ( previous job: 8.26, improvement: 99.76%)
  • whonix-gateway-18_socket-root: 7.92 🔻 ( previous job: 7.76, degradation: 102.18%)
  • whonix-gateway-18_exec-data-simplex: 68.36 🟢 ( previous job: 73.13, improvement: 93.49%)
  • whonix-gateway-18_exec-data-duplex: 67.84 🟢 ( previous job: 71.98, improvement: 94.25%)
  • whonix-gateway-18_exec-data-duplex-root: 146.54 🟢 ( previous job: 146.68, improvement: 99.91%)
  • whonix-gateway-18_socket-data-duplex: 146.32 🔻 ( previous job: 142.71, degradation: 102.53%)
  • whonix-workstation-18_exec: 8.67 🟢 ( previous job: 8.85, improvement: 97.97%)
  • whonix-workstation-18_exec-root: 141.31 🟢 ( previous job: 149.84, improvement: 94.31%)
  • whonix-workstation-18_socket: 8.58 🟢 ( previous job: 9.12, improvement: 94.12%)
  • whonix-workstation-18_socket-root: 8.02 🟢 ( previous job: 8.75, improvement: 91.75%)
  • whonix-workstation-18_exec-data-simplex: 69.36 🔻 ( previous job: 67.02, degradation: 103.49%)
  • whonix-workstation-18_exec-data-duplex: 71.91 🔻 ( previous job: 69.25, degradation: 103.84%)
  • whonix-workstation-18_exec-data-duplex-root: 146.42 🔻 ( previous job: 146.32, degradation: 100.07%)
  • whonix-workstation-18_socket-data-duplex: 134.90 🟢 ( previous job: 139.15, improvement: 96.94%)
  • dom0_root_seq1m_q8t1_write 3:write_bandwidth_kb: 281951.00 🟢 ( previous job: 231269.00, improvement: 121.91%)
  • dom0_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 4825.00 🟢 ( previous job: 2099.00, improvement: 229.87%)
  • dom0_varlibqubes_seq1m_q8t1_read 3:read_bandwidth_kb: 480777.00 🔻 ( previous job: 491827.00, degradation: 97.75%)
  • dom0_varlibqubes_seq1m_q8t1_write 3:write_bandwidth_kb: 250615.00 🟢 ( previous job: 103611.00, improvement: 241.88%)
  • dom0_varlibqubes_seq1m_q1t1_read 3:read_bandwidth_kb: 439102.00 🟢 ( previous job: 432938.00, improvement: 101.42%)
  • dom0_varlibqubes_seq1m_q1t1_write 3:write_bandwidth_kb: 185513.00 🟢 ( previous job: 143368.00, improvement: 129.40%)
  • dom0_varlibqubes_rnd4k_q32t1_read 3:read_bandwidth_kb: 106115.00 🟢 ( previous job: 103977.00, improvement: 102.06%)
  • dom0_varlibqubes_rnd4k_q1t1_read 3:read_bandwidth_kb: 7686.00 🔻 ( previous job: 8300.00, degradation: 92.60%)
  • dom0_varlibqubes_rnd4k_q1t1_write 3:write_bandwidth_kb: 3912.00 🔻 ( previous job: 3936.00, degradation: 99.39%)
  • fedora-42-xfce_root_seq1m_q8t1_read 3:read_bandwidth_kb: 383391.00 🟢 ( previous job: 346866.00, improvement: 110.53%)
  • fedora-42-xfce_root_seq1m_q1t1_write 3:write_bandwidth_kb: 30006.00 🟢 ( previous job: 26931.00, improvement: 111.42%)
  • fedora-42-xfce_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 8681.00 🔻 ( previous job: 8945.00, degradation: 97.05%)
  • fedora-42-xfce_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 1363.00 🟢 ( previous job: 457.00, improvement: 298.25%)
  • fedora-42-xfce_private_seq1m_q8t1_read 3:read_bandwidth_kb: 382273.00 🟢 ( previous job: 371835.00, improvement: 102.81%)
  • fedora-42-xfce_private_seq1m_q1t1_read 3:read_bandwidth_kb: 322242.00 🔻 ( previous job: 351871.00, degradation: 91.58%)
  • fedora-42-xfce_private_seq1m_q1t1_write 3:write_bandwidth_kb: 61918.00 🟢 ( previous job: 49472.00, improvement: 125.16%)
  • fedora-42-xfce_private_rnd4k_q32t1_read 3:read_bandwidth_kb: 90703.00 🟢 ( previous job: 75326.00, improvement: 120.41%)
  • fedora-42-xfce_private_rnd4k_q1t1_read 3:read_bandwidth_kb: 8400.00 🟢 ( previous job: 8365.00, improvement: 100.42%)
  • fedora-42-xfce_private_rnd4k_q1t1_write 3:write_bandwidth_kb: 1304.00 🟢 ( previous job: 517.00, improvement: 252.22%)
  • fedora-42-xfce_volatile_seq1m_q8t1_read 3:read_bandwidth_kb: 353532.00 🔻 ( previous job: 361328.00, degradation: 97.84%)
  • fedora-42-xfce_volatile_seq1m_q8t1_write 3:write_bandwidth_kb: 134618.00 🔻 ( previous job: 140939.00, degradation: 95.52%)
  • fedora-42-xfce_volatile_seq1m_q1t1_write 3:write_bandwidth_kb: 48226.00 🟢 ( previous job: 46842.00, improvement: 102.95%)
  • fedora-42-xfce_volatile_rnd4k_q32t1_read 3:read_bandwidth_kb: 82825.00 🟢 ( previous job: 79581.00, improvement: 104.08%)
  • fedora-42-xfce_volatile_rnd4k_q32t1_write 3:write_bandwidth_kb: 2853.00 🟢 ( previous job: 2110.00, improvement: 135.21%)
  • fedora-42-xfce_volatile_rnd4k_q1t1_read 3:read_bandwidth_kb: 7316.00 🔻 ( previous job: 7708.00, degradation: 94.91%)
  • fedora-42-xfce_volatile_rnd4k_q1t1_write 3:write_bandwidth_kb: 2148.00 🟢 ( previous job: 861.00, improvement: 249.48%)
  • debian-13-xfce_dom0-dispvm-api (mean:6.85): 82.20 🟢 ( previous job: 84.68, improvement: 97.07%)
  • debian-13-xfce_dom0-dispvm-gui-api (mean:7.569): 90.83 🟢 ( previous job: 98.62, improvement: 92.09%)
  • debian-13-xfce_dom0-dispvm-preload-2-api (mean:3.836): 46.03
  • debian-13-xfce_dom0-dispvm-preload-2-delay-0-api (mean:3.556): 42.67
  • debian-13-xfce_dom0-dispvm-preload-2-delay-minus-1d2-api (mean:4.119): 49.43
  • debian-13-xfce_dom0-dispvm-preload-4-api (mean:3.209): 38.51
  • debian-13-xfce_dom0-dispvm-preload-4-delay-0-api (mean:3.669): 44.03
  • debian-13-xfce_dom0-dispvm-preload-4-delay-minus-1d2-api (mean:3.471): 41.65
  • debian-13-xfce_dom0-dispvm-preload-4-gui-api (mean:3.414): 40.97
  • debian-13-xfce_dom0-dispvm-preload-6-gui-api (mean:3.458): 41.50
  • debian-13-xfce_dom0-vm-api (mean:0.035): 0.41 🟢 ( previous job: 0.59, improvement: 70.22%)
  • debian-13-xfce_dom0-vm-gui-api (mean:0.037): 0.45 🟢 ( previous job: 0.58, improvement: 77.55%)
  • fedora-42-xfce_dom0-dispvm-api (mean:7.227): 86.72 🟢 ( previous job: 89.70, improvement: 96.68%)
  • fedora-42-xfce_dom0-dispvm-gui-api (mean:8.48): 101.76 🟢 ( previous job: 107.00, improvement: 95.11%)
  • fedora-42-xfce_dom0-dispvm-preload-2-api (mean:4.107): 49.28
  • fedora-42-xfce_dom0-dispvm-preload-2-delay-0-api (mean:3.946): 47.36
  • fedora-42-xfce_dom0-dispvm-preload-2-delay-minus-1d2-api (mean:4.176): 50.11
  • fedora-42-xfce_dom0-dispvm-preload-4-api (mean:3.743): 44.92
  • fedora-42-xfce_dom0-dispvm-preload-4-delay-0-api (mean:3.893): 46.72
  • fedora-42-xfce_dom0-dispvm-preload-4-delay-minus-1d2-api (mean:3.57): 42.84
  • fedora-42-xfce_dom0-dispvm-preload-2-gui-api (mean:5.179): 62.15
  • fedora-42-xfce_dom0-dispvm-preload-4-gui-api (mean:3.89): 46.68
  • fedora-42-xfce_dom0-dispvm-preload-6-gui-api (mean:3.446): 41.35
  • fedora-42-xfce_dom0-vm-gui-api (mean:0.042): 0.50 🟢 ( previous job: 0.54, improvement: 93.15%)
  • whonix-workstation-18_dom0-dispvm-api (mean:9.291): 111.50 🟢 ( previous job: 117.52, improvement: 94.87%)
  • whonix-workstation-18_dom0-dispvm-gui-api (mean:10.199): 122.39 🟢 ( previous job: 130.38, improvement: 93.87%)
  • whonix-workstation-18_dom0-dispvm-preload-2-api (mean:5.38): 64.56
  • whonix-workstation-18_dom0-dispvm-preload-2-delay-0-api (mean:5.263): 63.16
  • whonix-workstation-18_dom0-dispvm-preload-2-delay-minus-1d2-api (mean:5.596): 67.16
  • whonix-workstation-18_dom0-dispvm-preload-4-api (mean:4.584): 55.01
  • whonix-workstation-18_dom0-dispvm-preload-4-delay-0-api (mean:4.446): 53.35
  • whonix-workstation-18_dom0-dispvm-preload-4-delay-minus-1d2-api (mean:4.797): 57.57
  • whonix-workstation-18_dom0-dispvm-preload-2-gui-api (mean:6.631): 79.57
  • whonix-workstation-18_dom0-dispvm-preload-4-gui-api (mean:5.053): 60.63
  • whonix-workstation-18_dom0-dispvm-preload-6-gui-api (mean:4.801): 57.61
  • whonix-workstation-18_dom0-vm-api (mean:0.036): 0.44 🟢 ( previous job: 0.60, improvement: 72.52%)
  • whonix-workstation-18_dom0-vm-gui-api (mean:0.04): 0.48 🔻 ( previous job: 0.45, degradation: 107.80%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants