Turris Omnia python3 packages

[mereacre]

Below is the list of default python3 packages installed on Turris Omnia:

python3 - 3.7.13-3.7-1
python3-asyncio - 3.7.13-3.7-1
python3-attrs - 19.3.0-3.7-1
python3-babel - 2.9.1-3.7-1
python3-base - 3.7.13-3.7-1
python3-cachelib - 0.1-3.7-1
python3-certifi - 2019.11.28-3.7-1
python3-cffi - 1.13.2-3.7-1
python3-cgi - 3.7.13-3.7-1
python3-cgitb - 3.7.13-3.7-1
python3-chardet - 3.0.4-3.7-2
python3-click - 7.0-3.7-1
python3-codecs - 3.7.13-3.7-1
python3-cryptography - 2.8-3.7-1
python3-ctypes - 3.7.13-3.7-1
python3-dbm - 3.7.13-3.7-1
python3-decimal - 3.7.13-3.7-1
python3-distro - 1.5.0-3.7-1
python3-distutils - 3.7.13-3.7-1
python3-email - 3.7.13-3.7-1
python3-flask - 1.1.2-3.7-1
python3-flask-babel - 2.0.0-3.7-1
python3-flask-seasurf - 0.3.0-3.7-1
python3-flup - 1.0.3-3.7-3
python3-gdbm - 3.7.13-3.7-1
python3-idna - 2.9-3.7-1
python3-ifaddr - 0.1.7-3.7-1
python3-importlib-metadata - 1.5.0-3.7-1
python3-itsdangerous - 1.1.0-3.7-1
python3-jinja2 - 2.10.1-3.7-1
python3-jsonschema - 3.2.0-3.7-2
python3-light - 3.7.13-3.7-1
python3-logging - 3.7.13-3.7-1
python3-lzma - 3.7.13-3.7-1
python3-markupsafe - 1.1.1-3.7-1
python3-more-itertools - 7.2.0-3.7-1
python3-multiprocessing - 3.7.13-3.7-1
python3-ncurses - 3.7.13-3.7-1
python3-openssl - 3.7.13-3.7-1
python3-paho-mqtt - 1.5.1-3.7-1
python3-pbkdf2 - 1.3-3.7-4
python3-pkg-resources - 47.1.0-1
python3-ply - 3.11-3.7-1
python3-prctl - 1.7-3.7-2
python3-pycparser - 2.20-3.7-1
python3-pydoc - 3.7.13-3.7-1
python3-pyrsistent - 0.15.3-3.7-1
python3-python - 1.0.0-1
python3-pytz - 2019.3-3.7-1
python3-requests - 2.23.0-3.7-1
python3-setuptools - 47.1.0-1
python3-simplejson - 3.16.0-3.7-2
python3-six - 1.13.0-3.7-1
python3-slugify - 3.0.3-3.7-1
python3-sqlite3 - 3.7.13-3.7-1
python3-text-unidecode - 1.2-3.7-1
python3-ubus - 0.1-3.7-2
python3-uci - 0.8.1-3.7-1
python3-unittest - 3.7.13-3.7-1
python3-urllib - 3.7.13-3.7-1
python3-urllib3 - 1.25.10-3.7-1
python3-websockets - 9.1-3.7-1
python3-werkzeug - 0.16.0-3.7-1
python3-xml - 3.7.13-3.7-1
python3-zeroconf - 0.28.0-3.7-1
python3-zipp - 0.6.0-3.7-2

Ideally for code running on Turris we should use the packages that come with the device. We could use other packages but it creates complication with the disk space, etc.

[cbrafter]

Thanks for this, if the scripts build the YAML to CSV it should be straightforward to read in Python using the built-in file IO, and sqlite3 will cover updating the database. I don't think we'll need to install any extra packages on the Turris.

[aloisklink]

python3-jsonschema - 3.2.0-3.7-2

Nice, the Turris even comes with the jsonschema package pre-installed. That's great.

I don't think we'll need to install any extra packages on the Turris.

100% agree. YAML parsing is probably the only extra package we need, but if we have json/csv D3 claims, that can be removed.

(although, PyYAML is pretty heavy since it's a C-compiled Python lib, but even then, it's probably only around 1 MB in size https://pypi.org/project/PyYAML/#files)

I'm guessing it's probably worth splitting up the Python code into 3 different packages, d3-importer, d3-convert, d3-lint (names not set it stone).
The d3-lint and d3-convert can be designed to run on CI/server and use almost any dependencies, while the d3-importer will be the one that must run on the Turris Omnia.