Feature/net validator

[jhughesoti]

• Fixes validator issue
• Change default behavior to validate network by default
• Remove validate arg and replace with --no-validate to disable network validation
• Adds new status to session when validating
  • New status needs to be added to UI. UI currently shows no status during validation but continues to work as expected if page is refreshed after validation ends.

[jboddey]

I'm not sure I believe there is a real need for network validation involving a faux device anymore. It is a very rare case that the network services do not function correctly and are now quite stable. For an additional 1:30 on each Testrun, I do not think it adds that much value.

Also, when I have tested this PR, the faux dev reported a failure in ping to google.com and ntp sync failed for some reason. Despite there being network failures reported (though not in practice), this did not have an effect on Testrun. Surely we would want to prevent the user going any further if it was not successful?

Anyway, I think the best approach would be to have network validation not run by default but for us to setup an automated GitHub action that verifies all network services work as expected.

[jhughesoti]

We have had two instances of updates/upgrades to underlying packages that have broken internet access and/or installations that are incomplete that this check could give the warning to a user about something being wrong. I don't think it should be abandoned, but maybe switched back to an optional --validate option so we can instruct testers to run it in validation mode to try to pull out additional issues about what services might be misbehaving.

Not sure what failures you are experiencing with it but since this one has been sitting a while, it might need some updates to correct depending on what they are.

The reason we can't do the automated github actions alone is the same reason we can't do internet based testing in that environment, until that is resolved, it can't be relied on as a comprehensive solution.

[jboddey]

We have had two instances of updates/upgrades to underlying packages that have broken internet access and/or installations that are incomplete that this check could give the warning to a user about something being wrong. I don't think it should be abandoned, but maybe switched back to an optional --validate option so we can instruct testers to run it in validation mode to try to pull out additional issues about what services might be misbehaving.

Not sure what failures you are experiencing with it but since this one has been sitting a while, it might need some updates to correct depending on what they are.

The reason we can't do the automated github actions alone is the same reason we can't do internet based testing in that environment, until that is resolved, it can't be relied on as a comprehensive solution.

I agree with going with the optional --validate so that we can ask manufacturers to do this on demand for troubleshooting. The issue I was having (maybe just not implemented) is that even if some of the services don't work, the output does not seem to be any different (except for the JSON). We should have a clear command line output that states which services are not working and any troubleshooting detail.

[jhughesoti]

Default behavior is now switched to validator disabled and only runs when requested. No system is in place to check these results as I believe we have now decided this will be more of a diagnostic tool when issues are reported.

Since this also adds a new status, the UI still needs to be updated to handle the new status because it will appear broken during the validator state and requires a browser refresh once complete.

[jhughesoti]

@jboddey UI updated to resolve the new validator status correctly.

[MariusBaldovin]

I tested this feature

The test duration was 14m 59s

The backend logs: