Specifying a subset of nodes from a queue

[avcopan]

I have adapted a code that I develop for to work with HyperQueue and am very impressed -- it's an amazing tool. Thank you for your work on it!

On one of the clusters we run on, I am encountering a scenario¹ where it would be helpful to specify a list of nodes to run on. In a PBS submission script, I can do the following:

#!/usr/bin/env bash

#PBS -q <queue name>
#PBS -A <account name>
#PBS -l nodes=csed-0021+csed-0022
...

I have tried to pass the same flag to the hq alloc command as follows, but it doesn't seem to work:

hq alloc add pbs --time-limit 1h -- -q <queue name> -A <account name> -l nodes=csed-0021+csed-0022

I don't get an error, but the job just hangs and nothing gets submitted.

Is there a way to do this? If not, is there a possibility of adding this feature?

Footnotes

1. The motivation is that this is a cluster where, unfortunately, many users are running jobs directly over SSH, rather than submitting through PBS. This is causing jobs to get submitted on nodes that are already occupied, causing them to fail, for example, by exceeding memory. The general culture on this cluster is to simply find a set of unoccupied nodes and run on those, and other users know to check which nodes are occupied first. So it would be useful to be able to specify a subset of nodes from the queue to run on, in order to avoid these clashes. ↩

[spirali]

Can you please provide us logs from from autoalloc?

It should be in ~/.hq-server/hq-current/autoalloc/<queue-id>

[Kobzol]

Even better, in directory there should be the actual bash files that were submitted to PBS. These could help with debugging, e.g. you can try to submit them manually to PBS to see what happens.

[avcopan]

When running with the -l nodes=<...> flag, the ~/.hq-server/hq-current directory is empty except for a single file access.json with the following contents:

{
  "version": "v0.22.0",
  "server_uid": "8AwxDj",
  "client": {
    "host": <...>,
    "port": 44045,
    "secret_key": <...>
  },
  "worker": {
    "host": <...>,
    "port": 45215,
    "secret_key": <...>
  }
}

The only output I see from the Python API is the loading bar, which just hangs at 0%:

  0%|                                                | 0/54 [01:20<?, ?task/s, 0/1 job completed]

[avcopan]

When I omit the -l nodes=<...> flag, it works without issue and I can find the bash files. For example, ~/.hq-server/hq-current/autoalloc/1/001 has a script called hq-submit.sh that looks fine:

#!/bin/bash
#PBS -l select=1
#PBS -N hq-1-1
#PBS -o /home/copan/.hq-server/001/autoalloc/1/001/stdout
#PBS -e /home/copan/.hq-server/001/autoalloc/1/001/stderr
#PBS -l walltime=01:00:00
#PBS -q csed -A g-CSE

/gpfs/fs1/home/copan/bin/hq worker start --idle-timeout "5m" --manager "pbs" --server-dir "/home/copan/.hq-server/001" --cpus "2" --resource "mem=sum(19074)" --on-server-lost "finish-running" --time-limit "1h"

[avcopan]

Ah, this may be the problem: I tried taking the above script and adding the flag to it:

#!/bin/bash
#PBS -l select=1
#PBS -N hq-1-1-with-nodes-flag
#PBS -o /home/copan/.hq-server/001/autoalloc/1/001/stdout
#PBS -e /home/copan/.hq-server/001/autoalloc/1/001/stderr
#PBS -l walltime=01:00:00
#PBS -q csed -A g-CSE -l nodes=csed-0001

/gpfs/fs1/home/copan/bin/hq worker start --idle-timeout "5m" --manager "pbs" --server-dir "/home/copan/.hq-server/001" --cpus "2" --resource "mem=sum(19074)" --on-server-lost "finish-running" --time-limit "1h"

and I get the following error from qsub:

copan@copan~/.hq-server/hq-current/autoalloc/1/001 $ qsub hq-submit-modified.sh
qsub: Cannot be used with select or place: nodes

[avcopan]

So, it looks like the nodes resource specification is incompatible with select. If I remove this line from the script:

#PBS -l select=1

I can submit it to PBS and it works without issue.

[avcopan]

Following this suggestion, I am able to do select=1:host=csed-0008, although I'm not sure how I would specify that through autoalloc, nor how to allow submission to a set of multiple nodes.

[avcopan]

... alternatively, if there were a way to ignore certain nodes in the queue, that would be an option too.

[Kobzol]

Hmm, I wonder why you didn't get an error reported by HQ. Did the server logs not contain any errors about the allocation? 🤔

In terms of modifying the -l parameters, that is sadly not so simple. We could maybe create some placeholders, like $NUM_NODES, and you could specify -l$NUM_NODES:host=csed-0008, and if HQ saw that you use -l, it would not pass it on its own. But we would ideally like to avoid inspecting the PBS/Slurm flags, as they can be pretty different per each PBS installation, these things vary a lot.

A more general solution would be to allow users to write their own custom spawning script templates, where HQ would run the provided script and fill in some placeholders. We already had some use-cases for that, but it hasn't been implemented yet.

... alternatively, if there were a way to ignore certain nodes in the queue, that would be an option too.

If it is not possible to ignore them without using -l, then currently it's probably not easily doable.

[avcopan]

Hmm, I wonder why you didn't get an error reported by HQ. Did the server logs not contain any errors about the allocation? 🤔

Sorry, this was my mistake. I am executing the commands via Python subprocess and I forgot that I had silenced the output. The server does indeed give an error, I just wasn't seeing it 🤦‍♂️

Error: Received error: "Could not submit allocation: qsub execution failed\n\nCaused by:\n    Exit code: 6\n    Stderr: qsub: Cannot be used with select or place: nodes\n    Stdout: "

I need to figure out a better way to redirect the server logs to a file.

In terms of modifying the -l parameters, that is sadly not so simple. We could maybe create some placeholders, like $NUM_NODES, and you could specify -l$NUM_NODES:host=csed-0008, and if HQ saw that you use -l, it would not pass it on its own. But we would ideally like to avoid inspecting the PBS/Slurm flags, as they can be pretty different per each PBS installation, these things vary a lot.

A more general solution would be to allow users to write their own custom spawning script templates, where HQ would run the provided script and fill in some placeholders. We already had some use-cases for that, but it hasn't been implemented yet.

This seems like it would be the right approach for my use-case, since it's a fairly special case.

If it is not possible to ignore them without using -l, then currently it's probably not easily doable.

That makes sense. I see a few references to something like this online, but none of them have worked so far.

[avcopan]

@spirali @Kobzol Now that I've experimented with the PBS scripts generated by auto-alloc, I understand how to create workers manually, and I think I can create my own workaround -- a simple script that creates the appropriate workers on a specific list of nodes. So, from my perspective, the issue is resolved and you can feel free to close it.

Thanks so much for your help, and for your patience.

P.S. I apologize for not realizing that the error messages were silenced 🤦‍♂️ My code was starting the server like this, which I mistakenly thought would print errors to stdout.

    subprocess.Popen(
        ["hq", "server", "start"], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT
    )

[Kobzol]

No problem, glad you can make it work even without auto allocation. I'll leave the issue open, because the submit templates would be useful in general.

[avcopan]

FWIW the PBS syntax for this would be as follows:

#PBS -l select=1:ncpus=36:mpiprocs=36:host=csed-0034+1:ncpus=36:mpiprocs=36:host=csed-0035

With the ability to create workers manually, though, I don't need to do this through auto-alloc.

I love how flexible the HyperQueue interface is. It is simple enough to be easy to learn, but provides a lot of direct control. Awesome work!