CLI becomes unresponsive.

[LordDarkHelmet]

The CLI becomes unresponsive on a Dynode running on a VPS. The underlying daemon is also apparently unresponsive. This seemingly happens randomly. I have not found a coloration between the unresponsiveness and any given event. However, this only "seems" to occur on a VPS running more than one Dynode.

Steps to reproduce

1. Run multiple Dynodes on a single VPS
2. Wait until it becomes unresponsive. (it seems to be rare)
3. I created a custom script to capture the log in the event that the CLI becomes unresponsive.

What platform are you using (Linux, Windows, Mac)

Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-65-generic x86_64)

Description

See attached for the debug output. Note that each "Unknown error" report gives a 250 line dump of the log file. It runs every 5 min, so a lot of it repeats from error to error. The error code "143" is caused by the "timeout --preserve-status -k 45s 40s " that is used before each CLI command to detect freezing. Please note that the line that says "CLI Unknown Error 143: Shutting Down" is a typo, I was not attempting to shut down the Dynode, I should have typed in "dumping log".
dynamiccd_A.error.txt

As a side note, I also have a sequence daemon running on the same VPS. It was also unresponsive when Dynode A had the issue. The act of restarting the sequence daemon caused the dynamic CLI to become responsive and the script was then able to heal the issue.

I can improve the error reporting if there is something you want to catch. Crashes have been very rare for me, but I was able to capture this when I loaded up a single instance with multiple Dynodes. (yes each has its own IPv4 address)

[Duality-CDOO]

If this is due to running multiple instances of a Dynode on one VPS, then we do not directly support that, and won't be actively working on a fix.

[LordDarkHelmet]

Running a local server with more than one instance on it (given that each instance has been assigned a unique IPv4 address) is something that is commonly done. I suspect that it running on a VPS vs on a local server has no bearing on the reported issue. I believe that some of our community members currently providing Dynodes hosting solutions are running into this issue. The current resolution for them is to quickly restart a non-responsive unit in order to not timeout. This band-aid relives the symptom but does not fix the issue.

What additional information can I collect that will aid in finding and fixing this issue?

[Duality-CDOO]

Are you assigning each instance with its own custom ports to use?

[SvenKercher]

U have to isolate the clients, like docker , vm , ....

[LordDarkHelmet]

Each port is static as per the dynode rules. The RPC port is unique to each instance. Here is an example config file.

# This file was generated. YYYY-MM-DD_hh:mm:ss  Version: x.xx.xx
# Do not use special characters or spaces with username/password
rpcuser=<Randomly Generated and Unique to each file>
rpcpassword=<Randomly Generated and Unique to each file>
rpcport=<Procedural Generated and Unique to each file ex 33350, 33351, ...>
port=33300

# MINING:  These are your mining variables
# Gen can be 0 or 1. 1=mining, 0=No mining
gen=0
# genproclimit sets the number of processors you want to use -1 for unbounded (all of them)
genproclimit=-1

# DYNODE: 
# externalip is the IP address of this machine. (the remote Dynode's IP address) 
externalip=<Uniique IPv4 Address for this instance>
bind=<Uniique IPv4 Address for this instance>
# dynode can be 0 or 1. 1=dynode, 0=not a dynode
dynode=1
# Use your local or control wallet, run the command "Dynode genkey" to generate a unique dynodepairingkey for each remote Dynode
dynodepairingkey=<Procedural Generated and Unique to each file>

# if you can't connect, you may need to add a node, check the peers list on an explorer and add some nodes here
# addnode=<IP Address>:<PORT>

# End of generated file

[LordDarkHelmet]

Just had a similar crash on a standalone VPS running Ubuntu 18.04 LTS. The only thing running on the VPS is a single Dynode setup with my script. This issue is rare, and it defiantly happens more often on a server running multiple Dynodes, but it just happened on a standalone dedicated VPS.

Is there anything I can do to trap information that will aid in debugging this issue?