add support for using mbedtls instead of openssl

[keredson]

this library was recently updated to use libssh v0.10.6. libssh v0.10.0 added support for using mbedtls (instead of openssl). can we add a feature to enable building libssh with option WITH_MBEDTLS? (vendored-mbedtls perhaps?)

See:

• https://www.libssh.org/2022/08/26/libssh-0-10-0/
• https://github.com/xbmc/libssh/blob/master/README.mbedtls
• https://github.com/xbmc/libssh/blob/3cb0a1bf023cce7209290f998b8dc3029ed3df77/DefineOptions.cmake#L10C1-L11C1

[wez]

I don't personally have time to implement this, but I am open to reviewing a PR that does.

What's the goal for enabling this? In my experience, mbedtls is not widely used. It would be a shame to go to a lot of effort to implement and maintain something that "nobody" will use.

[keredson]

any ESP32 project would use it. esp-idf v5 came out in 2022, with a lot of new features and supported boards. (lots of risc-v) but dropped their fake openssl compatibility wrapper (which previously just wrapped mbedtls). so this project is won't build post esp-idf v4.

example:

$ cargo install espup
$ espup install
$ . $HOME/export-esp.sh
$ cargo generate esp-rs/esp-idf-template
⚠️   Favorite `esp-rs/esp-idf-template` not found in config, using it as a git repository: https://github.com/esp-rs/esp-idf-template.git
✔ 🤷   Which template should be expanded? · cargo
🤷   Project Name: test-ssh-idf5
🔧   Destination: /tmp/test-ssh-idf5 ...
🔧   project-name: test-ssh-idf5 ...
🔧   Generating template ...
✔ 🤷   Which MCU to target? · esp32
✔ 🤷   Configure advanced template options? · true
✔ 🤷   ESP-IDF version (master = UNSTABLE) · v5.1
✔ 🤷   Enable STD support? · true
✔ 🤷   Configure project to use Dev Containers (VS Code and GitHub Codespaces)? · false
✔ 🤷   Configure project to support Wokwi simulation with Wokwi VS Code extension? · false
✔ 🤷   Add CI files for GitHub Action? · false
🔧   Moving generated files into: `/tmp/test-ssh-idf5`...
🔧   Initializing a fresh Git repository
✨   Done! New project created /tmp/test-ssh-idf5
$ cargo build
   Compiling test-ssh-idf5 v0.1.0 (/tmp/test-ssh-idf5)
    Finished dev [optimized + debuginfo] target(s) in 2.77s

but when you add libssh-rs:

$ cargo add libssh-rs -F vendored-openssl
    Updating crates.io index
      Adding libssh-rs v0.3.2 to dependencies.
             Features:
             - vendored
             + vendored-openssl
    Updating crates.io index
$ cargo build
   Compiling openssl-sys v0.9.102
error: failed to run custom build command for `openssl-sys v0.9.102`

Caused by:
  process didn't exit successfully: `/tmp/test-ssh-idf5/target/debug/build/openssl-sys-682491a0cf30e5a8/build-script-main` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=XTENSA_ESP32_ESPIDF_OPENSSL_NO_VENDOR
  XTENSA_ESP32_ESPIDF_OPENSSL_NO_VENDOR unset
  cargo:rerun-if-env-changed=OPENSSL_NO_VENDOR
  OPENSSL_NO_VENDOR unset

  --- stderr
  thread 'main' panicked at /home/derek/.cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-src-300.2.3+3.2.1/src/lib.rs:366:18:
  don't know how to configure OpenSSL for xtensa-esp32-espidf
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[keredson]

i would think it would be as simple as adding cfg.define("WITH_MBEDTLS", Some("1")); to libssh-rs-sys/build.rs and conditioning all the openssl stuff, but clearly not. 😅

hacked up attempt here: main...keredson:libssh-rs:main

when building that test-ssh-idf5 above with dependency libssh-rs = { features = ["vendored"], path="/home/derek/projects/libssh-rs/libssh-rs" }

[keredson]

i discovered if you add this to .cargo/config.toml:

[env]
PKG_CONFIG_ALLOW_CROSS = "1"

it will build!

$ cargo build
Compiling libssh-rs-sys v0.2.4
Compiling openssl-sys v0.9.102
Compiling libssh-rs v0.3.3
Finished dev [optimized + debuginfo] target(s) in 6.62s

but the second you try to use it:

use libssh_rs::{Session};
let mut session = Session::new().unwrap();

it won't link (.rlib file gives file format not recognized):

= note: [ldproxy] Running ldproxy
Error: Linker /tmp/test-linking/.embuild/espressif/tools/xtensa-esp32-elf/esp-12.2.0_20230208/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc failed: exit status: 1
STDERR OUTPUT:
/tmp/test-linking/.embuild/espressif/tools/xtensa-esp32-elf/esp-12.2.0_20230208/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/12.2.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/test-linking/target/xtensa-esp32-espidf/debug/deps/liblibssh_rs_sys-ba33b208cf49f79d.rlib: error adding symbols: file format not recognized

$ objdump -d /tmp/test-linking-idfv4/target/xtensa-esp32-espidf/debug/deps/liblibssh_rs_sys-ba33b208cf49f79d.rlib looks OK, but admittedly i'm out of my depth here.