Revertive and non revertive behavior

[henderiw]

deviation revertive behaviour should become configurable

• off by default
• on by default
• based on target label

maybe there is a controller who writes the status and another controller that uses this info and updates the spec of the intent

[henderiw]

right now socio implements revertive behavior. When a deviation occurs the original config in the spec overrides the deviation to adhere to the original intent.

now there are scenario's where one wants to adhere to the deviation an not automatically revert. We call this non revertive behavior. We want to discuss how we want to operate on this?

the scenario's we can think off.

• recovery of an intermittent failure in the datastore: in non revertive mode -> we don't want to touch the config and revert
• when the config in the config snippet does not change -> we dont go to the device
• when a config snippet changed (any value) -> we apply the full config snippet again and override everything (this means deviations can be overwritten)
• when a config snippet changed -> we adhere to the deviations (so deviations become automatically part of the config)
  -> the deviation leaf did not change, but another leaf changes -> we would like the config to revert the (another) leaf, but keep the deviation leaf intact.
  -> the deviation leaf changed (to the old or new value) -> the deviation gets removed and we want to adhere to the new value -> could be the value of the deviation or another one.
• delete: do we want to keep deviations and delete the rest of the config ?

[alexandernorth]

Another use case could be with secrets/passwords.
We might want to have k8s secrets which contain these values unhashed (from vault etc) but the device returns the password hashed on next sync, causing SDC to report a deviation/push to the device when this is expected/desired behaviour.

[henderiw]

Is it not safer to keep the hashed value since you dont want to decode it in between?
if we put it in the clear we expose some data not?

[alexandernorth]

In general I would agree, but given the following flow I think we remain secure:

1. Config parameter loaded from k8s Secret (Perhaps based on add the ability to specify sensitive data in the schema #192)
2. The secret is pulled from vault (e.g. External Secrets Operator) - where it is not hashed as vault is assumed to be secure storage
3. SDC applies the config with the plain secret (the transport is secure for NETCONF at least)
4. SDC should not re-apply the same secret to the device, even though the device may send the hashed value back
5. SDC should re-apply the secret if it changes (git source of truth)

This does leave the question of what happens if a user manually changes the secret's value from the CLI of the device as SDC would ignore this/not be able to detect it, but this behaviour breaks the GitOps model and also doesn't fit with the idea of non-revertive behaviour anyway.

[henderiw]

Where I have a question is in step 2/3 why would the vault use unhashed. My understanding/assumption is that even though the data is in the clear it is still hashed per device and hence there should not be a conflict, unless you change it.

I would prefer that SDC does not have to deal with hashes, etc etc. It gets data and potentially merges it as per the secret PR #192 but nothing else. if the data does not match based on the merge, we would see this as a deviation.

I do agree that the secret merge need to be taken into account.

so the deviation diff is handled on the merge of the secret and the user input.

So the steps should be:

• user config + merge with secret references: this is the data we use to compare against the deviations.

[alexandernorth]

It's definitely possible to have multiple vault entries for (different) hashes as well as plaintext, I was wondering if it is possible to take advantage of this 'no revert' behaviour to avoid having the hash stored alongside the plaintext, and to ensure consistency.

While I cannot give a concrete process where this is currently used, it could be that the secret is a shared password/key and while the NF requires this to be hashed, the other consumer needs the password to be in plaintext, or hashed differently. This would result in vault containing duplicated values, just represented differently.

I don't expect that SDC would take care of performing the hashing, but that it could 'ignore' these mismatched responses.

I think having the secret merge would be valuable, so I suggest this being a strong starting point.

The above idea was to show where we might have a use case for the proposed non-revertive behaviour for specific config paths, and is definitely not blocked by the current behaviour. How would you plan to proceed with this functionality?

[henderiw]

ok here is the proposal for this work

1. we will put the deviations in a new CRD:

• unmanaged config will become a deviation with the name of the node
• deviations will be named as per the intent name -> we might have to prefix them with the target name -> TBD

2. in revertive mode the config controller will watch the deviations and if change apply the config. the goal is that a change on the device will be overruled immediately
3. in non revertive mode -> deviations of an intent are put in a dedicated Deviation CR (one per intent). The difference with revertive is by default we accept the deviation (and apply it to the system with a higher priority). meaning we treat the deviation as a config snippet and apply it to the system.
   if a user want to change a deviation -> lets say reject he/she can modify the deviation CR or even delete it in which case you get the revertive behavior.

We will add the secret handling override once we have the base deviation implemented.

Enabling/Disabling revertive/non revertive operation -> can be global or per CR if needed

[henderiw]

In non revertive mode:

• Accept deviation: default
• Deny the full deviation: -> delete deviation-CR. (we reapply the config-CR w/o the related deviation CR
• Deny a piece of the deviation -> delete that part of the deviation-CR (we reapply the config CR w the updated deviation CR)

in the data server we need have a special treatment wrt to deviation config intent:

• configserver will indicate in the deviation field this config snippet is originated from a deviation CR
• this is treated by the data server in such a way: we don't store this as an intent in the dataserver, but use it to calculate the diff + do validation.

add a flag in the config CR to indicate revertive or non-revertive
add a global flag: revertive/non-revertive
-> the config sever will use the global behaviour unless overwritten by the config CR snippet