redbot_cli crashes

[JannisBush]

I used redbot for a while on mac with python 3.9 and had no issues.
Today, I tried it out an an Ubuntu 18 machine (python 3.9) and it only works in roughly every third try.
The other runs crash with two different errors:

1. Error:

Traceback (most recent call last):
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 153, in handle_writable
    sent = self.socket.send(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data/spec_test/redbot/bin/redbot_cli", line 74, in <module>
    main()
  File "/data/data/spec_test/redbot/bin/redbot_cli", line 66, in main
    thor.run()
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 116, in run
    self._run_fd_events()
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 357, in _run_fd_events
    self._fd_event(event, fileno)
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 190, in _fd_event
    self._fd_targets[fd].emit(event)
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/events.py", line 89, in emit
    ev(*args)
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 158, in handle_writable
    self._handle_close()
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 202, in _handle_close
    self._close()
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 208, in _close
    self.unregister_fd()
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 57, in unregister_fd
    self._loop.unregister_fd(self._fd)
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 339, in unregister_fd
    self._epoll.unregister(fd)
OSError: [Errno 9] Bad file descriptor

2. Error:

exception calling callback for <Future at 0x7ff97bc8d340 state=finished returned list>
Traceback (most recent call last):
  File "/home/jannis/.pyenv/versions/3.9.6/lib/python3.9/concurrent/futures/_base.py", line 329, in _invoke_callbacks
    callback(self)
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/dns/__init__.py", line 25, in done
    cb(ff.result())
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/http/client.py", line 201, in _handle_dns
    self._initiate_connection()
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/http/client.py", line 220, in _initiate_connection
    tcp_client.connect_dns(
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 349, in connect_dns
    self.handle_socket_error(socket.error(err, os.strerror(err)))
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 376, in handle_socket_error
    self.handle_conn_error(err_type, err_id, err_str)
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/tcp.py", line 389, in handle_conn_error
    self.unregister_fd()
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 57, in unregister_fd
    self._loop.unregister_fd(self._fd)
  File "/home/jannis/.cache/pypoetry/virtualenvs/spec-test-meXfqZJN-py3.9/lib/python3.9/site-packages/thor/loop.py", line 339, in unregister_fd
    self._epoll.unregister(fd)
FileNotFoundError: [Errno 2] No such file or directory

[mnot]

What kernel are you running?

[JannisBush]

I was on Linux 4.15.0-202-generic, but I updated to Linux 5.4.0-136-generic today and the issue persists.

[JannisBush]

I installed everything in a fresh operation system and get another similar issue:

>>> a = subprocess.check_output(["python", "bin/redbot_cli", "-a", "-o", "har", "http://example.org/"])
Traceback (most recent call last):
  File "/home/exp/spec_test/redbot/bin/redbot_cli", line 74, in <module>
    main()
  File "/home/exp/spec_test/redbot/bin/redbot_cli", line 66, in main
    thor.run()
  File "/home/exp/.cache/pypoetry/virtualenvs/spec-test-QULprG1B-py3.10/lib/python3.10/site-packages/thor/loop.py", line 116, in run
    self._run_fd_events()
  File "/home/exp/.cache/pypoetry/virtualenvs/spec-test-QULprG1B-py3.10/lib/python3.10/site-packages/thor/loop.py", line 357, in _run_fd_events
    self._fd_event(event, fileno)
  File "/home/exp/.cache/pypoetry/virtualenvs/spec-test-QULprG1B-py3.10/lib/python3.10/site-packages/thor/loop.py", line 190, in _fd_event
    self._fd_targets[fd].emit(event)
  File "/home/exp/.cache/pypoetry/virtualenvs/spec-test-QULprG1B-py3.10/lib/python3.10/site-packages/thor/events.py", line 89, in emit
    ev(*args)
  File "/home/exp/.cache/pypoetry/virtualenvs/spec-test-QULprG1B-py3.10/lib/python3.10/site-packages/thor/tcp.py", line 135, in handle_readable
    data = self.socket.recv(self.read_bufsize)
OSError: [Errno 101] Network is unreachable
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', 'bin/redbot_cli', '-a', '-o', 'har', 'http://example.org/']' returned non-zero exit status 1.

[mnot]

Using a download of the latest source, try this:

make cli https://www.mnot.net/

[JannisBush]

I just realized the errors only occur when using http:// (then it fails almost always and only succeeds roughly every fifth run). When using https:// there is no problem and every call works.
The same is true when using make cli instead of the old command.

[mnot]

Hm. I'm not seeing that, using:

make cli http://www.mnot.net/

... on a 6.1.8 kernel (arch) using Python 3.10.9.

You're using 5.4.0 and 3.9.something.

I did a quick install of Ubuntu 18.04 LTS with kernel 4.15.0-202-generic and Python 3.9.16, and didn't see any issues.

Same on 20.04 LTS, with a 5.4.0-137-generic kernel and Python 3.9.16.

Is there something unusual about your kernel, network setup, etc.?

[JannisBush]

The new install is using Ubuntu 22.04 LTS with a 4.19 kernel and Python 3.10.6.

The first system is inside a VM, the second one inside an lxc container. Both run on the same server.
Not sure, if there is anything unusual about the network setup.

[cageyv]

@JannisBush do you have docker or any other container runtime?
Probably dockerized version of redbot-cli could help you

• https://github.com/cageyv/redbot/pkgs/container/redbot
• Docker: python:3-slim-bookworm, Docker Image CI: produce pre-builded docker images redbot#308

[mnot]

Just checking in - are you still seeing this?

[JannisBush]

I just checked on both machines (now updated to 6.1.38) and both still crash on HTTP with the respective error messages.

• OSError: [Errno 101] Network is unreachable with python3.10 and inside lxd container
• OSError: [Errno 9] Bad file descriptor with python3.9 and inside a VM

[mnot]

Can you try:

pip install thor # or eqivalent
python -m thor.http.client https://www.example.com/

?

[JannisBush]

Sure:

$ python -m thor.http.client http://www.example.com/
<frozen runpy>:128: RuntimeWarning: 'thor.http.client' found in sys.modules after import of package 'thor.http', but prior to execution of 'thor.http.client'; this may result in unpredictable behaviour
*** ERROR: DNS Error (Name or service not known)

Update:

python -m thor.http.client http://www.example.org/
<frozen runpy>:128: RuntimeWarning: 'thor.http.client' found in sys.modules after import of package 'thor.http', but prior to execution of 'thor.http.client'; this may result in unpredictable behaviour
exception calling callback for <Future at 0x7f32d78d3da0 state=finished returned list>
Traceback (most recent call last):
  File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/concurrent/futures/_base.py", line 340, in _invoke_callbacks
    callback(self)
  File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/dns/__init__.py", line 25, in done
    cb(ff.result())
  File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/http/client.py", line 202, in _handle_dns
    self._initiate_connection()
  File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/http/client.py", line 221, in _initiate_connection
    tcp_client.connect_dns(
  File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/tcp.py", line 349, in connect_dns
    self.handle_socket_error(socket.error(err, os.strerror(err)))
  File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/tcp.py", line 376, in handle_socket_error
    self.handle_conn_error(err_type, err_id, err_str)
  File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/tcp.py", line 389, in handle_conn_error
    self.unregister_fd()
  File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/loop.py", line 57, in unregister_fd
    self._loop.unregister_fd(self._fd)
  File "/home/jannis/.pyenv/versions/3.12.0/lib/python3.12/site-packages/thor/loop.py", line 339, in unregister_fd
    self._epoll.unregister(fd)
FileNotFoundError: [Errno 2] No such file or directory

Other container with python3.11:

python -m thor.http.client http://www.example.org/
<frozen runpy>:128: RuntimeWarning: 'thor.http.client' found in sys.modules after import of package 'thor.http', but prior to execution of 'thor.http.client'; this may result in unpredictable behaviour
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/http/client.py", line 660, in <module>
    test_client(sys.argv[1].encode("ascii"), sys.stdout.buffer.write, sys.stderr.write)
  File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/http/client.py", line 654, in test_client
    run()
  File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/loop.py", line 116, in run
    self._run_fd_events()
  File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/loop.py", line 357, in _run_fd_events
    self._fd_event(event, fileno)
  File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/loop.py", line 190, in _fd_event
    self._fd_targets[fd].emit(event)
  File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/events.py", line 89, in emit
    ev(*args)
  File "/home/ubuntu/miniconda3/lib/python3.11/site-packages/thor/tcp.py", line 153, in handle_writable
    sent = self.socket.send(data)
           ^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 101] Network is unreachable

I also updated to python3.12 and the original issue persists:

$ make cli http://www.mnot.net/
PYTHONPATH=.:./.venv/bin ./.venv/bin/python redbot/cli.py http://www.mnot.net/
Traceback (most recent call last):
  File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/tcp.py", line 153, in handle_writable
    sent = self.socket.send(data)
           ^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data/spec_test/redbot_new/redbot/cli.py", line 72, in <module>
    main()
  File "/data/data/spec_test/redbot_new/redbot/cli.py", line 64, in main
    thor.run()
  File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/loop.py", line 116, in run
    self._run_fd_events()
  File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/loop.py", line 357, in _run_fd_events
    self._fd_event(event, fileno)
  File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/loop.py", line 190, in _fd_event
    self._fd_targets[fd].emit(event)
  File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/events.py", line 89, in emit
    ev(*args)
  File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/tcp.py", line 158, in handle_writable
    self._handle_close()
  File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/tcp.py", line 202, in _handle_close
    self._close()
  File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/tcp.py", line 208, in _close
    self.unregister_fd()
  File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/loop.py", line 57, in unregister_fd
    self._loop.unregister_fd(self._fd)
  File "/data/data/spec_test/redbot_new/.venv/lib/python3.12/site-packages/thor/loop.py", line 339, in unregister_fd
    self._epoll.unregister(fd)
OSError: [Errno 9] Bad file descriptor
Makefile:56: recipe for target 'cli' failed
make: *** [cli] Error 1

[mnot]

Transferring to Thor.