Skip to content

Add acquisition automation modes and acqmode command #391

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cfremling wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add acquisition automation modes and acqmode command #391

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a89f02d
Add acquisition automation modes and acqmode command
cfremling Feb 1, 2026
a20b509
Fix auto-acq waitstate and fine-tune xterm handling
cfremling Feb 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Config/sequencerd.cfg.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,10 @@ ACQUIRE_RETRYS=5 # max number of retrys before acquisition f
ACQUIRE_OFFSET_THRESHOLD=0.5 # computed offset below this threshold (in arcsec) defines successful acquisition
ACQUIRE_MIN_REPEAT=2 # minimum number of sequential successful acquires
ACQUIRE_TCS_MAX_OFFSET=60 # the maximum allowable offset sent to the TCS, in arcsec
ACQ_AUTOMATIC_MODE=1 # 1=legacy, 2=semi-auto, 3=auto
ACQ_FINE_TUNE_CMD=ngps_acq # command to run after guiding for final fine tune
ACQ_FINE_TUNE_XTERM=0 # run fine tune in xterm (0/1)
ACQ_OFFSET_SETTLE=0 # seconds to wait after automatic offset

# Calibration Settings
# CAL_TARGET=(name caldoor calcover lampthar lampfear lampbluc lampredc lolamp hilamp mod1 mod2 ... mod6)
Expand Down
2 changes: 2 additions & 0 deletions common/sequencerd_commands.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
#ifndef SEQEUNCERD_COMMANDS_H
#define SEQEUNCERD_COMMANDS_H
const std::string SEQUENCERD_ABORT = "abort";
const std::string SEQUENCERD_ACQMODE = "acqmode";
const std::string SEQUENCERD_CONFIG = "config";
const std::string SEQUENCERD_DOTYPE = "do";
const std::string SEQUENCERD_EXIT = "exit";
Expand Down Expand Up @@ -47,6 +48,7 @@ const std::vector<std::string> SEQUENCERD_SYNTAX = {
SEQUENCERD_TCS+" ...",
"",
SEQUENCERD_ABORT,
SEQUENCERD_ACQMODE+" [ ? | <mode> ]",
SEQUENCERD_CONFIG,
SEQUENCERD_DOTYPE+" [ one | all ]",
SEQUENCERD_EXIT,
Expand Down
210 changes: 186 additions & 24 deletions sequencerd/sequence.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@

#include "sequence.h"

#include <signal.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>

namespace Sequencer {

constexpr long CAMERA_PROLOG_TIMEOUT = 6000; ///< timeout msec to send camera prolog command
Expand Down Expand Up @@ -560,40 +565,177 @@ namespace Sequencer {
* std::thread( &Sequencer::Sequence::dothread_acquisition, this ).detach();
***/

// If not a calibration target then introduce a pause for the user
// to make adjustments, send offsets, etc.
// If not a calibration target then handle acquisition automation
//
if ( !this->target.iscal ) {

// waiting for user signal (or cancel)
//
// The sequencer is effectively paused waiting for user input. This
// gives the user a chance to ensure the correct target is on the slit,
// select offset stars, etc.
//
{
ScopedState wait_state( wait_state_manager, Sequencer::SEQ_WAIT_USER );
std::stringstream mode_msg;
mode_msg << "NOTICE: acquisition automation mode " << this->acq_automatic_mode;
this->async.enqueue_and_log( function, mode_msg.str() );
}

this->async.enqueue_and_log( function, "NOTICE: waiting for USER to send \"continue\" signal" );
auto wait_for_user = [&](const std::string &notice) -> bool {
this->is_usercontinue.store(false);
ScopedState wait_state( wait_state_manager, Sequencer::SEQ_WAIT_USER );
this->async.enqueue_and_log( function, "NOTICE: "+notice );
while ( !this->cancel_flag.load() && !this->is_usercontinue.load() ) {
std::unique_lock<std::mutex> lock(cv_mutex);
this->cv.wait( lock, [this]() { return( this->is_usercontinue.load() || this->cancel_flag.load() ); } );
}
this->async.enqueue_and_log( function, "NOTICE: received "
+(this->cancel_flag.load() ? std::string("cancel") : std::string("continue"))
+" signal!" );
if ( this->cancel_flag.load() ) return false;
this->is_usercontinue.store(false);
return true;
};

auto wait_for_guiding = [&]() -> long {
ScopedState wait_state( wait_state_manager, Sequencer::SEQ_WAIT_GUIDE );
this->async.enqueue_and_log( function, "NOTICE: waiting for ACAM guiding" );
Comment on lines 593 to 595
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot Feb 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid clearing SEQ_WAIT_ACQUIRE in wait_for_guiding

In auto modes 2/3, wait_for_guiding creates a ScopedState for SEQ_WAIT_ACQUIRE, which clears that bit as soon as the helper returns (on guiding, timeout, or error). The acquisition thread already sets the same SEQ_WAIT_ACQUIRE bit in dothread_acquisition, so this new scoped state can clear the bit while acquisition is still running. Any clients or logic watching the wait-state (telemetry/UI or other waits) will see acquisition complete early and may proceed to expose even though acquisition is still active. Consider removing this ScopedState or using a separate state/refcounted mechanism for the “waiting for guiding” phase.

Useful? React with 👍 / 👎.

Copy link
Collaborator Author

@cfremling cfremling Feb 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add a new state to wait for inside the mode loops to not mess with the overall acquisition state

Copy link
Collaborator Author

@cfremling cfremling Feb 2, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Added SEQ_WAIT_GUIDE instead of SEQ_WAIT_ACQUIRE while waiting
    for guiding.
  • Added ACQ_FINE_TUNE_XTERM so fine-tune can run in its own xterm when
    enabled so it can be properly monitored.
  • Fine-tune process is now launched in its own process group and is terminated on seq abort . works even if seq abort sent when ngps_acq in progress

seq acqmode 1,2,3 TESTED with NGPS simulator mode

Commit: a20b509

auto start_time = std::chrono::steady_clock::now();
const bool use_timeout = ( this->acquisition_timeout > 0 );
const auto timeout = std::chrono::duration<double>( this->acquisition_timeout );
while ( !this->cancel_flag.load() ) {
std::string reply;
if ( this->acamd.command( ACAMD_ACQUIRE, reply ) != NO_ERROR ) {
logwrite( function, "ERROR reading ACAM acquire state" );
return ERROR;
}
if ( reply.find( "guiding" ) != std::string::npos ) return NO_ERROR;
if ( reply.find( "stopped" ) != std::string::npos ) return ERROR;
if ( use_timeout && std::chrono::steady_clock::now() > ( start_time + timeout ) ) return TIMEOUT;
std::this_thread::sleep_for( std::chrono::milliseconds(500) );
}
return ERROR;
};

while ( !this->cancel_flag.load() && !this->is_usercontinue.load() ) {
std::unique_lock<std::mutex> lock(cv_mutex);
this->cv.wait( lock, [this]() { return( this->is_usercontinue.load() || this->cancel_flag.load() ); } );
}
auto run_fine_tune = [&]() -> long {
if ( this->acq_fine_tune_cmd.empty() ) return NO_ERROR;
this->async.enqueue_and_log( function, "NOTICE: running fine tune command: "+this->acq_fine_tune_cmd );

this->async.enqueue_and_log( function, "NOTICE: received "
+(this->cancel_flag.load() ? std::string("cancel") : std::string("continue"))
+" signal!" );
} // end scope for wait_state = WAIT_USER
if ( this->acq_fine_tune_xterm ) {
this->async.enqueue_and_log( function, "NOTICE: launching fine tune in xterm" );
}

if ( this->cancel_flag.load() ) {
this->async.enqueue_and_log( function, "NOTICE: sequence cancelled" );
return;
pid_t pid = fork();
if ( pid == 0 ) {
// make a dedicated process group so we can signal the whole tree
setpgid( 0, 0 );
if ( this->acq_fine_tune_xterm ) {
execlp( "xterm", "xterm", "-T", "NGPS Fine Tune", "-e",
"sh", "-lc", this->acq_fine_tune_cmd.c_str(), (char*)nullptr );
// fall through if xterm is missing
}
execl( "/bin/sh", "sh", "-c", this->acq_fine_tune_cmd.c_str(), (char*)nullptr );
_exit(127);
}
if ( pid < 0 ) {
logwrite( function, "ERROR starting fine tune command: "+this->acq_fine_tune_cmd );
return ERROR;
}
// Ensure the child is its own process group (best effort).
setpgid( pid, pid );
this->fine_tune_pid.store( pid );

int status = 0;
while ( true ) {
pid_t result = waitpid( pid, &status, WNOHANG );
if ( result == pid ) break;
if ( result < 0 ) {
logwrite( function, "ERROR waiting on fine tune command" );
return ERROR;
}
if ( this->cancel_flag.load() ) {
this->async.enqueue_and_log( function, "NOTICE: abort requested; terminating fine tune" );
// terminate the whole fine-tune process group
kill( -pid, SIGTERM );
auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(5);
while ( std::chrono::steady_clock::now() < deadline ) {
result = waitpid( pid, &status, WNOHANG );
if ( result == pid ) break;
std::this_thread::sleep_for( std::chrono::milliseconds(100) );
}
if ( result != pid ) {
kill( -pid, SIGKILL );
waitpid( pid, &status, 0 );
}
this->fine_tune_pid.store( 0 );
return ERROR;
}
std::this_thread::sleep_for( std::chrono::milliseconds(100) );
}

this->fine_tune_pid.store( 0 );
if ( WIFEXITED( status ) && WEXITSTATUS( status ) == 0 ) {
this->async.enqueue_and_log( function, "NOTICE: fine tune complete" );
return NO_ERROR;
}

logwrite( function, "ERROR fine tune command failed: "+this->acq_fine_tune_cmd );
return ERROR;
};

if ( this->acq_automatic_mode == 1 ) {
if ( !wait_for_user( "waiting for USER to send \"continue\" signal" ) ) {
this->async.enqueue_and_log( function, "NOTICE: sequence cancelled" );
return;
}
}
else {
if ( this->acq_automatic_mode == 2 ) {
if ( !wait_for_user( "waiting for USER to send \"continue\" signal to start acquisition" ) ) {
this->async.enqueue_and_log( function, "NOTICE: sequence cancelled" );
return;
}
}

this->async.enqueue_and_log( function, "NOTICE: starting acquisition" );
std::thread( &Sequencer::Sequence::dothread_acquisition, this ).detach();

this->is_usercontinue.store(false);
long acqerr = wait_for_guiding();
if ( acqerr != NO_ERROR ) {
std::string reason = ( acqerr == TIMEOUT ? "timeout" : "error" );
this->async.enqueue_and_log( function, "WARNING: failed to reach guiding state ("+reason+"); falling back to manual continue" );
if ( !wait_for_user( "waiting for USER to send \"continue\" signal to expose (guiding failed)" ) ) {
this->async.enqueue_and_log( function, "NOTICE: sequence cancelled" );
return;
}
}
else {
bool fine_tune_ok = ( run_fine_tune() == NO_ERROR );
if ( !fine_tune_ok ) {
this->async.enqueue_and_log( function, "WARNING: fine tune failed; waiting for USER continue to expose" );
if ( !wait_for_user( "waiting for USER to send \"continue\" signal to expose (fine tune failed)" ) ) {
this->async.enqueue_and_log( function, "NOTICE: sequence cancelled" );
return;
}
}

this->async.enqueue_and_log( function, "NOTICE: received USER continue signal!" );
if ( fine_tune_ok ) {
if ( this->acq_automatic_mode == 2 ) {
if ( !wait_for_user( "waiting for USER to send \"continue\" signal to expose" ) ) {
this->async.enqueue_and_log( function, "NOTICE: sequence cancelled" );
return;
}
}
else if ( this->acq_automatic_mode == 3 ) {
if ( this->target.offset_ra != 0.0 || this->target.offset_dec != 0.0 ) {
this->async.enqueue_and_log( function, "NOTICE: applying target offset automatically" );
error |= this->target_offset();
if ( error != NO_ERROR ) {
this->thread_error_manager.set( THR_ACQUISITION );
return;
}
if ( this->acq_offset_settle > 0 ) {
this->async.enqueue_and_log( function, "NOTICE: waiting for offset settle time" );
std::this_thread::sleep_for( std::chrono::duration<double>( this->acq_offset_settle ) );
}
}
}
}
}
}

// Ensure slit offset is in "expose" position
//
Expand Down Expand Up @@ -2222,6 +2364,26 @@ namespace Sequencer {
this->cancel_flag.store(true);
this->cv.notify_all();

// terminate fine tune process if running
//
pid_t ftpid = this->fine_tune_pid.load();
if ( ftpid > 0 ) {
logwrite( function, "NOTICE: terminating fine tune process" );
kill( -ftpid, SIGTERM );
auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(5);
int status = 0;
while ( std::chrono::steady_clock::now() < deadline ) {
pid_t result = waitpid( ftpid, &status, WNOHANG );
if ( result == ftpid ) break;
std::this_thread::sleep_for( std::chrono::milliseconds(100) );
}
if ( waitpid( ftpid, &status, WNOHANG ) == 0 ) {
kill( -ftpid, SIGKILL );
waitpid( ftpid, &status, 0 );
}
this->fine_tune_pid.store( 0 );
}

// drop into do-one to prevent auto increment to next target
//
this->do_once.store(true);
Expand Down
12 changes: 12 additions & 0 deletions sequencerd/sequence.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
#include <vector>
#include <chrono>
#include <atomic>
#include <sys/types.h>
#include <map>
#include <cmath>
#include <mysqlx/xdevapi.h>
Expand Down Expand Up @@ -153,6 +154,7 @@ namespace Sequencer {
SEQ_WAIT_TCS, ///< set when waiting for tcs
// states
SEQ_WAIT_ACQUIRE, ///< set when waiting for acquire
SEQ_WAIT_GUIDE, ///< set when waiting for guiding state
SEQ_WAIT_EXPOSE, ///< set when waiting for camera exposure
SEQ_WAIT_READOUT, ///< set when waiting for camera readout
SEQ_WAIT_TCSOP, ///< set when waiting specifically for tcs operator
Expand All @@ -173,6 +175,7 @@ namespace Sequencer {
{SEQ_WAIT_TCS, "TCS"},
// states
{SEQ_WAIT_ACQUIRE, "ACQUIRE"},
{SEQ_WAIT_GUIDE, "GUIDE"},
{SEQ_WAIT_EXPOSE, "EXPOSE"},
{SEQ_WAIT_READOUT, "READOUT"},
{SEQ_WAIT_TCSOP, "TCSOP"},
Expand Down Expand Up @@ -286,6 +289,7 @@ namespace Sequencer {
std::atomic<bool> cancel_flag{false};
std::atomic<bool> is_ontarget{false}; ///< remotely set by the TCS operator to indicate that the target is ready
std::atomic<bool> is_usercontinue{false}; ///< remotely set by the user to continue
std::atomic<pid_t> fine_tune_pid{0}; ///< fine tune process pid (process group leader)

/** @brief safely runs function in a detached thread using lambda to catch exceptions
*/
Expand All @@ -312,6 +316,10 @@ namespace Sequencer {
arm_readout_flag(false),
acquisition_timeout(0),
acquisition_max_retrys(-1),
acq_automatic_mode(1),
acq_fine_tune_cmd("ngps_acq"),
acq_fine_tune_xterm(false),
acq_offset_settle(0),
tcs_offsetrate_ra(45),
tcs_offsetrate_dec(45),
tcs_settle_timeout(10),
Expand Down Expand Up @@ -369,6 +377,10 @@ namespace Sequencer {

double acquisition_timeout; ///< timeout for target acquisition (in sec) set by configuration parameter ACAM_ACQUIRE_TIMEOUT
int acquisition_max_retrys; ///< max number of acquisition loop attempts
int acq_automatic_mode; ///< acquisition automation mode (1=legacy, 2=semi-auto, 3=auto)
std::string acq_fine_tune_cmd; ///< fine-tune command to run after guiding
bool acq_fine_tune_xterm; ///< run fine-tune command in its own xterm
double acq_offset_settle; ///< seconds to wait after automatic offset
double tcs_offsetrate_ra; ///< TCS offset rate RA ("MRATE") in arcsec per second
double tcs_offsetrate_dec; ///< TCS offset rate DEC ("MRATE") in arcsec per second
double tcs_settle_timeout; ///< timeout for telescope to settle (in sec) set by configuration parameter TCS_SETTLE_TIMEOUT
Expand Down
Loading