Add vega_lite_convert support

[Makesesama]

Hey,
I want to add vega_lite and vega_lite_convert to my project:

{:vega_lite, "~> 0.1.11"},
{:vega_lite_convert, "~> 1.0"},

but i get an error, because vega_lite_convert needs vl-convert and the build tries to download rusty-v8. I tried to work around it myself by providing the rusty_v8 with the RUSTY_V8_ARCHIVE env variable, but i cannot get it to work.
Here is the error:

...
   Compiling dynasm v1.2.3
   Compiling fontdb v0.21.0
   Compiling nix v0.26.2
   Compiling lazy-regex v3.1.0
   Compiling value-trait v0.8.1
   Compiling rsa v0.9.6
error: failed to run custom build command for `v8 v0.105.1`

Caused by:
  process didn't exit successfully: `/build/ex_vl_convert/target/release/build/v8-bb5c10486f72b0c9/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=.gn
  cargo:rerun-if-changed=BUILD.gn
  cargo:rerun-if-changed=src/binding.cc
  cargo:rerun-if-env-changed=CCACHE
  cargo:rerun-if-env-changed=CLANG_BASE_PATH
  cargo:rerun-if-env-changed=CXXSTDLIB
  cargo:rerun-if-env-changed=DENO_TRYBUILD
  cargo:rerun-if-env-changed=DOCS_RS
  cargo:rerun-if-env-changed=GN
  cargo:rerun-if-env-changed=GN_ARGS
  cargo:rerun-if-env-changed=HOST
  cargo:rerun-if-env-changed=NINJA
  cargo:rerun-if-env-changed=OUT_DIR
  cargo:rerun-if-env-changed=RUSTY_V8_ARCHIVE
  cargo:rerun-if-env-changed=RUSTY_V8_MIRROR
  cargo:rerun-if-env-changed=RUSTY_V8_SRC_BINDING_PATH
  cargo:rerun-if-env-changed=SCCACHE
  cargo:rerun-if-env-changed=V8_FORCE_DEBUG
  cargo:rerun-if-env-changed=V8_FROM_SOURCE
  cargo:rerun-if-env-changed=PYTHON
  cargo:rerun-if-env-changed=DISABLE_CLANG
  cargo:rerun-if-env-changed=EXTRA_GN_ARGS
  cargo:rerun-if-env-changed=PRINT_GN_ARGS
  cargo:rerun-if-env-changed=CARGO_ENCODED_RUSTFLAGS
  cargo:rustc-link-lib=static=rusty_v8
  lockfile: "/build/ex_vl_convert/target/x86_64-unknown-linux-gnu/release/build/v8.fslock"
  cargo:rustc-env=RUSTY_V8_SRC_BINDING_PATH=/build/cargo-vendor-dir/v8-0.105.1/gen/src_binding_release_x86_64-unknown-linux-gnu.rs
  static lib URL: https://github.com/denoland/rusty_v8/releases/download/v0.105.1/librusty_v8_release_x86_64-unknown-linux-gnu.a.gz
  cargo:rustc-link-search=/build/ex_vl_convert/target/x86_64-unknown-linux-gnu/release/gn_out/obj
  Looking for download in '"/homeless-shelter/.cargo/.rusty_v8/https___github_com_denoland_rusty_v8_releases_download_v0_105_1_librusty_v8_release_x86_64_unknown_linux_gnu_a_gz"'
  Downloading (using Python) https://github.com/denoland/rusty_v8/releases/download/v0.105.1/librusty_v8_release_x86_64-unknown-linux-gnu.a.gz
  Python downloader failed, trying with curl.

  --- stderr
  thread 'main' panicked at /build/cargo-vendor-dir/v8-0.105.1/build.rs:546:10:
  called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Probably you have an idea to fix this. Thanks for the great libary!

Cheers,
Max

[camelpunch]

I've had a quick look at this. The derivation that's failing isn't vega_lite_convert, but vega_lite_convert-native, which is generated by deps nix for rustler_precompiled deps.

I tried setting env.RUSTY_V8_ARCHIVE using the fetcher copied from the deno derivation, but ran into a linking issue. You might have more luck. The deno derivation has to solve this problem, so the answer is somewhere in there I think.

Ideally, we'd be able to tell the system not to download at all, and to provide the source required for the build. Then we could use nix caching without relying on blobs.

[Makesesama]

You are right. For now i needed a quick fix and just changed the native compilation to this:

 native =
          (extendedPkgs.makeRustPlatform {
            inherit (fenix) cargo rustc;
          }).buildRustPackage
            {
              pname = "${old.packageName}-native";
              version = old.version;
              src = nativeDir;
              cargoLock = {
                lockFile = "${nativeDir}/Cargo.lock";
              };
              nativeBuildInputs = [
                extendedPkgs.cmake
                extendedPkgs.python3
                extendedPkgs.curl
                extendedPkgs.pkg-config
              ] ++ extendedPkgs.lib.lists.optional extendedPkgs.stdenv.isDarwin extendedPkgs.darwin.IOKit;

              RUSTY_V8_ARCHIVE = pkgs.fetchurl {
                url = "https://github.com/denoland/rusty_v8/releases/download/v0.105.1/librusty_v8_release_x86_64-unknown-linux-gnu.a.gz";
                sha256 = "sha256-f7aDA74Jn2h4rp9sACGHX4DBbN6yevgWCEKdfI1fJDU=";
              };

              preBuild = ''
                export CARGO_HOME=$(mktemp -d)
                mkdir -p $CARGO_HOME/.rusty_v8

                gunzip -c $RUSTY_V8_ARCHIVE > $CARGO_HOME/.rusty_v8/librusty_v8_release_x86_64-unknown-linux-gnu.a
              '';

              doCheck = false;
            };

which is not nice but works for now. I also tried to do this with an override only for the vega_lite_convert but did not have enough time to figure out how to make that work. I will come back and look into this problem if i have time.

[tjarratt]

I might have a go at putting a workaround for vega_lite_convert-native in here. Based on what @Makesesama shared this looks like a pretty straightforward workaround.

[ferrine]

+1 for it

[Makesesama]

I actually have a setup that works right now. You can do this:

#package.nix
  # Import rustler overrides
  rustlerOverrides = import ./rustler_overrides.nix { inherit pkgs lib; };

  mixNixDeps = import ./deps.nix {
    inherit lib pkgs;
    beamPackages = pkgs.beamPackages;
    overrides = final: prev: {
      # Override vega_lite_convert to use our Rusty_v8 workaround
      vega_lite_convert = prev.vega_lite_convert.override (
        rustlerOverrides.vegaLiteConvertOverride prev.vega_lite_convert
      );
    };
  };

Copy your deps.nix section and override the important part. For me this is.

#rustler_overrides.nix
{
  pkgs,
  lib,
}:

let
  rustyV8Utils = import ./rusty_v8.nix {
    inherit lib;
    inherit (pkgs) stdenv fetchurl;
  };
  # Fenix overlay for Rust toolchain management
  fenixOverlay = import "${
    fetchTarball {
      url = "https://github.com/nix-community/fenix/archive/056c9393c821a4df356df6ce7f14c722dc8717ec.tar.gz";
      sha256 = "sha256:1cdfh6nj81gjmn689snigidyq7w98gd8hkl5rvhly6xj7vyppmnd";
    }
  }/overlay.nix";

  extendedPkgs = pkgs.extend fenixOverlay;

  # Rusty V8 configurations for different versions
  rustyV8Configs = {
    "v0.105.1" = {
      version = "0.105.1";
      shas = {
        x86_64-linux = "sha256-f7aDA74Jn2h4rp9sACGHX4DBbN6yevgWCEKdfI1fJDU=";
        aarch64-linux = "sha256-vuEP7in+A/PrBXSunRq1SC77dOuMiScsKcA712AuNuk=";
        x86_64-darwin = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="; # TODO: Add correct hash
        aarch64-darwin = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="; # TODO: Add correct hash
      };
      filename = "librusty_v8_release_x86_64-unknown-linux-gnu.a";
    };
  };

  # Common Rusty V8 workaround builder
  buildRustlerWithRustyV8 =
    {
      toolchain ? null,
      rustyV8Version ? "v0.105.1",
      ...
    }:
    old:
    let
      fenix =
        if toolchain == null then
          extendedPkgs.fenix.stable
        else
          extendedPkgs.fenix.fromToolchainName toolchain;

      rustyV8Config = rustyV8Configs.${rustyV8Version};

      nativeDir = "${old.src}/native/${with builtins; head (attrNames (readDir "${old.src}/native"))}";

      native =
        (extendedPkgs.makeRustPlatform {
          inherit (fenix) cargo rustc;
        }).buildRustPackage
          {
            pname = "${old.packageName}-native";
            version = old.version;
            src = nativeDir;
            cargoLock = {
              lockFile = "${nativeDir}/Cargo.lock";
            };

            nativeBuildInputs = [
              extendedPkgs.cmake
              extendedPkgs.python3
              extendedPkgs.curl
              extendedPkgs.pkg-config
            ]
            ++ lib.lists.optional extendedPkgs.stdenv.isDarwin extendedPkgs.darwin.IOKit;

            RUSTY_V8_ARCHIVE = rustyV8Utils.fetchLibrustyV8 rustyV8Config;

            preBuild = ''
              export CARGO_HOME=$(mktemp -d)
              mkdir -p $CARGO_HOME/.rusty_v8

              # Extract and place the rusty-v8 archive
              gunzip -c $RUSTY_V8_ARCHIVE > $CARGO_HOME/.rusty_v8/${rustyV8Config.filename}
            '';

            doCheck = false;
          };
    in
    {
      nativeBuildInputs = [
        extendedPkgs.cargo
        extendedPkgs.python3
        extendedPkgs.curl
      ];

      env.RUSTLER_PRECOMPILED_FORCE_BUILD_ALL = "true";
      env.RUSTLER_PRECOMPILED_GLOBAL_CACHE_PATH = "unused-but-required";

      preConfigure = ''
        mkdir -p priv/native
        for lib in ${native}/lib/*
        do
          ln -s "$lib" "priv/native/$(basename "$lib")"
        done
      '';

      buildPhase = ''
        suggestion() {
          echo "***********************************************"
          echo "                 deps_nix                      "
          echo
          echo " Rust dependency build failed.                 "
          echo
          echo " If you saw network errors, you might need     "
          echo " to disable compilation on the appropriate     "
          echo " RustlerPrecompiled module in your             "
          echo " application config.                           "
          echo
          echo " We think you need this:                       "
          echo
          echo -n " "
          grep -Rl 'use RustlerPrecompiled' lib \
            | xargs grep 'defmodule' \
            | sed 's/defmodule \(.*\) do/config :${old.packageName}, \1, skip_compilation?: true/'
          echo "***********************************************"
          exit 1
        }
        trap suggestion ERR
        ${old.buildPhase}
      '';
    };

in
{
  inherit buildRustlerWithRustyV8 rustyV8Configs;

  # Pre-configured overrides for common packages
  vegaLiteConvertOverride =
    old:
    buildRustlerWithRustyV8 {
      rustyV8Version = "v0.105.1";
    } old;

  explorerOverride =
    old:
    buildRustlerWithRustyV8 {
      toolchain = {
        name = "nightly-2024-11-01";
        sha256 = "sha256-wq7bZ1/IlmmLkSa3GUJgK17dTWcKyf5A+ndS9yRwB88=";
      };
    } old;
}

#rusty_v8.nix
{
  lib,
  stdenv,
  fetchurl,
}:

{
  fetchLibrustyV8 =
    args:
    fetchurl {
      name = "librusty_v8-${args.version}";
      url = "https://github.com/denoland/rusty_v8/releases/download/v${args.version}/librusty_v8_release_${stdenv.hostPlatform.rust.rustcTarget}.a.gz";
      sha256 = args.shas.${stdenv.hostPlatform.system};
      meta = {
        inherit (args) version;
        sourceProvenance = with lib.sourceTypes; [ binaryNativeCode ];
      };
    };
}

I don't think this is ideal or even the best way you can do it. But it works for me.