Run Error

[idontgetoutmuch]

*Main> main
main
[2021-08-04 06:56:58][FFExecutorD.executeLoop][Info][MacBook-Pro][PID 51384][ThreadId 59][funflow-1.5.0-GuwgHTnEu2V6hYgOjVs63C:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:183:9] Initialising connection to coordinator.
[2021-08-04 06:56:59][FFExecutorD.executeLoop.process][Info][MacBook-Pro][PID 51384][ThreadId 59][processId:CreateProcess {cmdspec = RawCommand "perl" ["/Users/dom/rg-map/funflow-example/store/item-8bccabdf0a55dc8a3ac4e73781ca09d8e469a3b1744b6415e432e0ce64e7f53a/get-world-files","/Users/dom/rg-map/funflow-example/store/pending-6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0/"], cwd = Just "/Users/dom/rg-map/funflow-example/store/pending-6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0/", env = Just [("NIX_PATH","nixpkgs=https://github.com/mpickering/nixpkgs/archive/547eef61effa37f10c49caebc771c3fceb16f14a.tar.gz")], std_in = Inherit, std_out = UseHandle {handle: /Users/dom/rg-map/funflow-example/store/metadata/hash-6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0/stdout}, std_err = UseHandle {handle: /Users/dom/rg-map/funflow-example/store/metadata/hash-6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0/stderr}, close_fds = True, create_group = False, delegate_ctlc = False, detach_console = False, create_new_console = False, new_session = False, child_group = Nothing, child_user = Nothing, use_process_jobs = False}][task:6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0][funflow-1.5.0-GuwgHTnEu2V6hYgOjVs63C:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:127:9] Executing
env: nix-shell: No such file or directory
[2021-08-04 06:56:59][FFExecutorD.executeLoop][Warning][MacBook-Pro][PID 51384][ThreadId 59][task:6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0][funflow-1.5.0-GuwgHTnEu2V6hYgOjVs63C:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:211:19] Task failed
FAILED: External task failed to construct item 'ContentHash "6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0"'. Task info: KnownTask (Failed (ExecutionInfo {_eiExecutor = Executor "MacBook-Pro", _eiElapsed = TimeSpec {sec = 0, nsec = 42243571}}) 127) stdout: Just "/Users/dom/rg-map/funflow-example/store/metadata/hash-6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0/stdout" stderr: Just "/Users/dom/rg-map/funflow-example/store/metadata/hash-6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0/stderr" Task: ExternalTask {_etCommand = "perl", _etParams = [Param [ParamPath (IPItem (Item {itemHash = ContentHash "8bccabdf0a55dc8a3ac4e73781ca09d8e469a3b1744b6415e432e0ce64e7f53a"})),ParamText "get-world-files"],Param [ParamOut]], _etEnv = EnvExplicit [("NIX_PATH",Param [ParamText "nixpkgs=",ParamText "https://github.com/mpickering/nixpkgs/archive/547eef61effa37f10c49caebc771c3fceb16f14a.tar.gz"])], _etWriteToStdOut = NoOutputCapture}

[idontgetoutmuch]

Maybe this in Main.hs is the problem?

nixPackages = "https://github.com/mpickering/nixpkgs/archive/547eef61effa37f10c49caebc771c3fceb16f14a.tar.gz"

Apparently not as I just downloaded it.

[idontgetoutmuch]

It has done something

/Users/dom/rg-map/funflow-example/store/complete-a1b4c8446ee6e4a9b961208ac9cd446b7195759e79fd79a4d98028485d0623b0:
  total used in directory 8 available 1.2 TiB
  dr-xr-xr-x  3 dom  staff   96  4 Aug 07:56 .
  dr-xr-xr-x  9 dom  staff  288  4 Aug 07:56 ..
  -r--r--r--  2 dom  staff  199  4 Aug 07:56 get-world-files

[idontgetoutmuch]

dom@MacBook-Pro rg-map % nix-shell bash -p google-cloud-sdk

[nix-shell:~/rg-map]$ mkdir foo
mkdir foo

[nix-shell:~/rg-map]$ gsutil -m rsync -r gs://rg-maps-final-world-files foo
gsutil -m rsync -r gs://rg-maps-final-world-files foo
Building synchronization state...
If you experience problems with multiprocessing on MacOS, they might be related to https://bugs.python.org/issue33725. You can disable multiprocessing by editing your .boto config or by adding the following flag to your command: `-o "GSUtil:parallel_process_count=1"`. Note that multithreading is still available even if you disable multiprocessing.

Caught non-retryable exception while listing gs://rg-maps-final-world-files/: invalid_grant: Bad Request
CommandException: Caught non-retryable exception - aborting rsync

So maybe that's the problem?

[idontgetoutmuch]

[nix-shell:~/rg-map]$ gsutil -o "GSUtil:parallel_process_count=1" -m rsync -r gs://rg-maps-final-world-files foo
gsutil -o "GSUtil:parallel_process_count=1" -m rsync -r gs://rg-maps-final-world-files foo
Building synchronization state...
Caught non-retryable exception while listing gs://rg-maps-final-world-files/: invalid_grant: Bad Request
CommandException: Caught non-retryable exception - aborting rsync

Now I am stuck.

[idontgetoutmuch]

What worries me is that I can see

env: nix-shell: No such file or directory

And trying this seems to suggest we are falling at the first hurdle (since I see no "Stop"):

  worldFiles <- getWorldFiles  -< script_dir
  stepIO error -< "Stop"

[idontgetoutmuch]

@mpickering any thoughts on how I might debug this further?

[idontgetoutmuch]

So it really is env not finding nix-shell

#! /usr/bin/env -v nix-shell
#! nix-shell -i bash -p google-cloud-sdk


#env executing:	/Users/dom/rg-map/funflow-example/store/item-7560ef43a4b2d604980de1c3777887c933e85a304e200324ae2d250f61ee4658/get-world-files
#env    arg[0]=	'/Users/dom/rg-map/funflow-example/store/item-7560ef43a4b2d604980de1c3777887c933e85a304e200324ae2d250f61ee4658/get-world-files'
#env    arg[1]=	'/Users/dom/rg-map/funflow-example/store/pending-a5f29a168b8bcc60812109a255e6796ea19105ae537448f18e096e1305fb6f0c/'
#env executing:	nix-shell
#env    arg[0]=	'nix-shell'
#env    arg[1]=	'/Users/dom/rg-map/funflow-example/store/item-7560ef43a4b2d604980de1c3777887c933e85a304e200324ae2d250f61ee4658/get-world-files'
#env    arg[2]=	'/Users/dom/rg-map/funflow-example/store/pending-a5f29a168b8bcc60812109a255e6796ea19105ae537448f18e096e1305fb6f0c/'
env: nix-shell: No such file or directory

[mpickering]

Why are you trying to build this project? I haven't worked on it for a few years.

[idontgetoutmuch]

Two reasons: 1) I want to do something similar (not with maps but with economic data) and 2) my wife is a keen orienteer and really liked the maps overlaid on the world map.

BTW I noticed a Scottish orienteering map had got put in the North Sea just off the coast of Holland

[idontgetoutmuch]

I changed get-world-files to contain

#! /usr/bin/env -S-P/bin:/usr/local/bin:/usr/bin:${PATH} zsh -c printenv
#! printenv

and got

NIX_PATH=nixpkgs=https://github.com/mpickering/nixpkgs/archive/547eef61effa37f10c49caebc771c3fceb16f14a.tar.gz
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x2
HOME=/Users/dom
LOGNAME=root
SHLVL=0
PWD=/Users/dom/rg-map/funflow-example/store/pending-5a35104e2bcb9f64e25e45d6f378489aac9e1d0a7dad3c4c4704bbea221edf43
OLDPWD=/Users/dom/rg-map/funflow-example/store/pending-5a35104e2bcb9f64e25e45d6f378489aac9e1d0a7dad3c4c4704bbea221edf43
GPG_TTY=/dev/ttys004
SSH_AUTH_SOCK=/Users/dom/.gnupg/S.gpg-agent.ssh
_=/usr/bin/printenv

[idontgetoutmuch]

Now I can get further and find something is not setting keys correctly

main
[2021-08-05 08:04:50][FFExecutorD.executeLoop][Info][MacBook-Pro][PID 58172][ThreadId 574][funflow-1.5.0-GuwgHTnEu2V6hYgOjVs63C:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:183:9] Initialising connection to coordinator.
[2021-08-05 08:04:51][FFExecutorD.executeLoop.process][Info][MacBook-Pro][PID 58172][ThreadId 574][processId:CreateProcess {cmdspec = RawCommand "perl" ["/Users/dom/rg-map/funflow-example/store/item-3bc1b83d257ff4944450d43c87c7fa6246eaf528450fb22521b7e9e7c66ef582/get-world-files","/Users/dom/rg-map/funflow-example/store/pending-c5b18c3853e9a763f67b194cd913399d23f9fb161cb1bb5ca862a9a205034c31/"], cwd = Just "/Users/dom/rg-map/funflow-example/store/pending-c5b18c3853e9a763f67b194cd913399d23f9fb161cb1bb5ca862a9a205034c31/", env = Just [("NIX_PATH","nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-21.05.tar.gz")], std_in = Inherit, std_out = UseHandle {handle: /Users/dom/rg-map/funflow-example/store/metadata/hash-c5b18c3853e9a763f67b194cd913399d23f9fb161cb1bb5ca862a9a205034c31/stdout}, std_err = UseHandle {handle: /Users/dom/rg-map/funflow-example/store/metadata/hash-c5b18c3853e9a763f67b194cd913399d23f9fb161cb1bb5ca862a9a205034c31/stderr}, close_fds = True, create_group = False, delegate_ctlc = False, detach_console = False, create_new_console = False, new_session = False, child_group = Nothing, child_user = Nothing, use_process_jobs = False}][task:c5b18c3853e9a763f67b194cd913399d23f9fb161cb1bb5ca862a9a205034c31][funflow-1.5.0-GuwgHTnEu2V6hYgOjVs63C:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:127:9] Executing
warning: unable to download 'https://github.com/NixOS/nixpkgs/archive/nixos-21.05.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 343 ms
warning: unable to download 'https://github.com/NixOS/nixpkgs/archive/nixos-21.05.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 677 ms
warning: unable to download 'https://github.com/NixOS/nixpkgs/archive/nixos-21.05.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 1077 ms
warning: unable to download 'https://github.com/NixOS/nixpkgs/archive/nixos-21.05.tar.gz': SSL peer certificate or SSH remote key was not OK (60); retrying in 2522 ms
warning: warning: unable to download 'https://github.com/NixOS/nixpkgs/archive/nixos-21.05.tar.gz': SSL peer certificate or SSH remote key was not OK (60); using cached result
warning: unable to download 'https://cache.nixos.org/cb59cbj156yml4r1pbdpra8m67gz51q4.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 310 ms
warning: unable to download 'https://cache.nixos.org/cb59cbj156yml4r1pbdpra8m67gz51q4.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 518 ms
warning: unable to download 'https://cache.nixos.org/cb59cbj156yml4r1pbdpra8m67gz51q4.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 1077 ms
warning: unable to download 'https://cache.nixos.org/cb59cbj156yml4r1pbdpra8m67gz51q4.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 2121 ms
error: unable to download 'https://cache.nixos.org/cb59cbj156yml4r1pbdpra8m67gz51q4.narinfo': SSL peer certificate or SSH remote key was not OK (60)
[2021-08-05 08:05:03][FFExecutorD.executeLoop][Warning][MacBook-Pro][PID 58172][ThreadId 574][task:c5b18c3853e9a763f67b194cd913399d23f9fb161cb1bb5ca862a9a205034c31][funflow-1.5.0-GuwgHTnEu2V6hYgOjVs63C:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:211:19] Task failed
FAILED: External task failed to construct item 'ContentHash "c5b18c3853e9a763f67b194cd913399d23f9fb161cb1bb5ca862a9a205034c31"'. Task info: KnownTask (Failed (ExecutionInfo {_eiExecutor = Executor "MacBook-Pro", _eiElapsed = TimeSpec {sec = 12, nsec = 83812698}}) 1) stdout: Just "/Users/dom/rg-map/funflow-example/store/metadata/hash-c5b18c3853e9a763f67b194cd913399d23f9fb161cb1bb5ca862a9a205034c31/stdout" stderr: Just "/Users/dom/rg-map/funflow-example/store/metadata/hash-c5b18c3853e9a763f67b194cd913399d23f9fb161cb1bb5ca862a9a205034c31/stderr" Task: ExternalTask {_etCommand = "perl", _etParams = [Param [ParamPath (IPItem (Item {itemHash = ContentHash "3bc1b83d257ff4944450d43c87c7fa6246eaf528450fb22521b7e9e7c66ef582"})),ParamText "get-world-files"],Param [ParamOut]], _etEnv = EnvExplicit [("NIX_PATH",Param [ParamText "nixpkgs=",ParamText "https://github.com/NixOS/nixpkgs/archive/nixos-21.05.tar.gz"])], _etWriteToStdOut = NoOutputCapture}

[idontgetoutmuch]

I can solve this with

nixScriptX impure std script scripts params = proc (scriptDir, a) -> do
  env <- mergeFiles -< absScripts scriptDir
  external' props (\(s, args) -> ExternalTask
        { _etCommand = "perl"
        , _etParams = contentParam (s ^</> script) : params args
        , _etWriteToStdOut = std
        , _etEnv = EnvExplicit [ ("NIX_PATH",
                                  textParam "nixpkgs=" <> textParam nixPackages )
                               , ("NIX_SSL_CERT_FILE",
                                  textParam "/Users/dom/.nix-profile/etc/ssl/certs/ca-bundle.crt")
                               ] }) -< (env, a)
  where
    props = def { ep_impure = impure }
    absScripts sd = map (sd ^</>) (script : scripts)

[idontgetoutmuch]

Now I get another error but it's to do with getting the actual maps and sad though I am to not be able to build the map (which is very cool IMO), I don't actually need this and can try my own external scripts.

Building synchronization state...
If you experience problems with multiprocessing on MacOS, they might be related to https://bugs.python.org/issue33725. You can disable multiprocessing by editing your .boto config or by adding the following flag to your command: `-o "GSUtil:parallel_process_count=1"`. Note that multithreading is still available even if you disable multiprocessing.

Caught non-retryable exception while listing gs://rg-maps-final-world-files/: invalid_grant: Bad Request
CommandException: Caught non-retryable exception - aborting rsync
[2021-08-05 08:14:10][FFExecutorD.executeLoop][Warning][MacBook-Pro][PID 58172][ThreadId 594][task:2e5bc76061c8e13b2bf0d1e322c3645716374f110fc0c8e6fe2db39eaeded4e4][funflow-1.5.0-GuwgHTnEu2V6hYgOjVs63C:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:211:19] Task failed
FAILED: External task failed to construct item 'ContentHash "2e5bc76061c8e13b2bf0d1e322c3645716374f110fc0c8e6fe2db39eaeded4e4"'. Task info: KnownTask (Failed (ExecutionInfo {_eiExecutor = Executor "MacBook-Pro", _eiElapsed = TimeSpec {sec = 65, nsec = 933493570}}) 1) stdout: Just "/Users/dom/rg-map/funflow-example/store/metadata/hash-2e5bc76061c8e13b2bf0d1e322c3645716374f110fc0c8e6fe2db39eaeded4e4/stdout" stderr: Just "/Users/dom/rg-map/funflow-example/store/metadata/hash-2e5bc76061c8e13b2bf0d1e322c3645716374f110fc0c8e6fe2db39eaeded4e4/stderr" Task: ExternalTask {_etCommand = "perl", _etParams = [Param [ParamPath (IPItem (Item {itemHash = ContentHash "3bc1b83d257ff4944450d43c87c7fa6246eaf528450fb22521b7e9e7c66ef582"})),ParamText "get-world-files"],Param [ParamOut]], _etEnv = EnvExplicit [("NIX_PATH",Param [ParamText "nixpkgs=",ParamText "https://github.com/NixOS/nixpkgs/archive/nixos-21.05.tar.gz"]),("NIX_SSL_CERT_FILE",Param [ParamText "/Users/dom/.nix-profile/etc/ssl/certs/ca-bundle.crt"])], _etWriteToStdOut = NoOutputCapture}