Torq LB Integration with Pulse (timestored UI)

[KaelynSom]

Torq Gateway/LB functionality doesn't work asyncronously with Pulse. Therefore when a long running request hits the gateway it blocks the GW and causes delays in other processes.

Query 1 – Intentionally mimicking a 10 second delay before returning a result:
.gw.syncexec["{system "sleep 10";:([] time:enlist .z.p)}[]";`rdb]

Query 2 – Executes every second inside Pulse along side query 1
.gw.syncexec["([] time:enlist .z.p)";`hdb]

What we observe is that whenever query 1 executes for 10 seconds, query 2 cannot. If I run the same test from a KDB connection doing the two queries simultaneously then it works as expected with no contention. Code below to show:

@[gw; (.gw.syncexec; "([] time:enlist .z.p)"; hdb)]
@[gw; (.gw.syncexec; "{system \"sleep 20\";:([] time:enlist .z.p)}[]"; rdb)]

[skairulla]

Update - 12/08/2025

Replicating client issue

• Successful, delay is occurring.
• Desired outcome is for grid to return .z.p every second while grid 2 returns .z.p every 5+ seconds due to sleep command
• What actually happening is grid waits for the grid 2 sleep command to complete before returning .z.p

Root Cause:
This is an intentional design of Torq's gateway.q script in which .gw.syncexec[query;servertypes(list of symbols)] will parallelize the queries across the backend servers, but block at the front end (so only 1 client query can be processed at a time). The results will be razed and returned to the client.

The Pulse dashboard is communicating with the kdb+ backend (the Torq session) via a WebSocket or HTTP endpoint which is being done over a single connection handle.

When multiple queries are sent to the gateway under a single connection handle - queries are done sequentially.

When multiple queries are send to the gateway under individual connection handles - (like separate kdb+ client processes connecting to the gateway) - queries are done in parallel

Potential Solutions

1. Open up multiple connection handles to the gateway, where each handle will be used by a different .gw.syncexec query that polls the TorQ kdb+ tickerplant every x seconds. Done through Pulse dashboard - connect data settings.
2. Incorporate parallel processing/multi-threading by starting the torq environment with slaves
3. Modification to gateway.q (.gw.syncexec / .gw.syncexecjpre36 / .gw.asyncexecjpts).
4. Edits to the Pulse UI, to get queries to be sent asynchly.

[skairulla]

Update - 12/09/2025

This is a current limitation with the Pulse dashboard in its ability to send multiple queries asynchronously.
Any changes to TorQ are expected to just get blocked up in a sequence of queries from the front end (Pulse dashboard).

It would require modifications to Pulse UI's java based code which is something Ryan Hamilton from timestored.com would have to do. This is a known issue by timestored.com but unsure on when a fix would be implemented by them.

Advise to close this ticket out since this isn't a TorQ bug but a limitation to the current version of Pulse.

[jonathonmcmurray]

I think what's happening here is we're trying to defer the sync query here: https://github.com/DataIntellectTech/TorQ/blob/master/code/processes/gateway.q#L466 and it must be failing - this is probably dependent on how Pulse is sending the query (e.g. I think it's not possible to defer response to HTTP requests, at least in some versions of q). If that's the case, I expect there would be log from https://github.com/DataIntellectTech/TorQ/blob/master/code/processes/gateway.q#L468 being produced, is that the case? We can probably enhance the error trapping of the defer to capture the actual error message & output this as well, which might be useful.

Beyond this, would be good to understand how the queries are being sent from Pulse, i.e. what message handler they are coming in through etc.

I know that some UIs (e.g. kx dashboards, qstudio) wrap their queries in custom logic, which can interfere with TorQ gateway logic. Timestored have a page here about the issue with qStudio https://www.timestored.com/qstudio/help/issues/qstudio-deferred-response-error not sure if there is a similar option in Pulse?

[jonathonmcmurray]

This is a known issue by timestored.com but unsure on when a fix would be implemented by them.

Is this documented somewhere on their website?

[KaelynSom]

Hey Jonathon,

Can answer a few of these Qs to help with this.

Can see from the logs the GW does try to deffered response, it executes asyncexecjpt as is expected for KDB version 4.1, so doesn't price that log error message that you have attached (I assumed it would fail for that reason initially).

Re the wrapper, from speaking to Ryan at timestored it doesn't have the same wrapper as qStudio, to the best of his knowledge but he didn't rule it out. Can disable the wrapper on qQstudio in settings>preferences>connections and get the same issue.

Can attach sample logs of the Pulse Query hitting the GW:

Pulse Query - Sent from Pulse to execute on the GW instance

2025.11.28D10:17:35.181010051| |gateway|muni_gateway|INF|runquery|query to run:
queryid | 27040
time | 2025.11.28D10:17:35.180734617
clienth | 31i
query | ({[u;q]$[.pm.execas ~ key .pm.execas;value (.pm.execas; q; u);.pm.valp ~ key .pm.valp; .pm.valp q; value q]};ksomerville;"([] time:enlist .z.p)")
servertype | ,hdb
queryattributes| ()!()
join | ,/
postback | ()
timeout | 0Wn
submittime | 0Np
returntime | 0Np
error | 0b
attributequery | 0b
sync | 1b
required | ,hdb
available | ,hdb

Query that executes as expected async (sent from qStudio from a client process to GW):

2025.11.28D10:19:24.363217188| |gateway|muni_gateway|INF|runquery|query to run:
queryid | 27102
time | 2025.11.28D10:19:24.362998306
clienth | 15i
query | ({[u;q]$[.pm.execas ~ key .pm.execas;value (.pm.execas; q; u);.pm.valp ~ key .pm.valp; .pm.valp q; value q]};torquser;"{queryStart:.z.p;system "sleep 20";:([] step:startend;time:(queryStart;.z.p))}[]")
servertype | ,rdb
queryattributes| ()!()
join | ,/
postback | ()
timeout | 0Wn
submittime | 0Np
returntime | 0Np
error | 0b
attributequery | 0b
sync | 1b
required | ,rdb
available | ,rdb

Re the Ticket or documentation from Ryan, was just from speaking to him that he seemed aware this could be an issue early doors when I mentioned it, but was keen on me to investigate (as expected 😆).