Consent hash must be stable

[phavekes]

This issue is imported from pivotal - Originaly created at Jan 15, 2021 by Thijs Kinkhorst

We store a hash of the user's released set of attributes and their values in the database. So we know whether the user has given consent for this SP for these specific attributes and values before. If so, skip consent question.

The hash is unfortunately not fully stable. Specifically, it seems that it changes when dealing with attributes that have multiple values. E.g. if the ordering at the idp changes. Or when some values are filtered out and an array is renumbered(?). Also it now changes when the casing of an attribute (name,value) changes. In any case, it should be stable. For the following set (for an SP, for each released attribute, for each of those attributes the released values) the hash is stable.

So at least the following must happen for it to be stable (in the resulting array after ARP which is the input for the hash):

• Case normalize all attribute names.
• Sort all attribute names.
• Case normalize all attribute values.
• Sort all attribute values.
• Renumber any indices or other leftovers of the input array.

Note that the normalization is only done in the hash calculation. Not in the actually released attributes.

A challenge is that we do not want to invalidate all given consents with the current algorithm. So we implement it as follows:

• We do not touch the existing consent hashing method at all
• We create a new hashing method that is more stable.
• We cover this new method with an abundance of unit tests to verify the stability given all sorts of inputs.
• We change the consent query from (pseudocode):
  SELECT * FROM consent WHERE user = me AND consenthash = hashfromoldmethod OR consenthash = hashfromnewmethod
• Newly given consent will be stored with the new hash.
• When old consent matched, still generate new consent hash (without showing consent screen).

The select query ensures that all given consents remain valid. New installs after this version will only have the new hashes so no need for the old code. But given the (in my opinion) low cost of keeping both I'm not currently inclined to add feature toggles or other things around the old hash, seems to cost more than is worth it.

Make it priority to cover all edge cases in test. The reviewer should put extra effort in verifying this!

Ballpark estimation to finish this story: 8 .. 10h

[phavekes]

@thijskh I have created the new hashing algorithm incorporating the 5 items above.  I\'ve currently 4 tests for it, but wanted to check with you which edge cases are possible?

Current tests are for sequential arrays & associative arrays (should always be an associative array is my understanding, but given your 5th point i added the possibility of a sequential array):

• switching order
• switching casing
• switching both
• using an item with multiple values vs using the same item with only a single value

Any other edge cases i'm missing? (Koen Cornelis - May 11, 2021)

[phavekes]

Order and casing both for attribute names, and for attribute values?

Some quick ideas:

What about:
[ "AttributeA" = [ 0 => "aap", 1 => "noot"] ]
versus:
[ "AttributeA" = [ 0 => "aap", 2 => "noot"] ]

What about:
[ "AttributeA" = [ 0 => "aap", 1 => "noot"], "AttributeB" => [] ]
versus:
[ "AttributeA" = [ 0 => "aap", 2 => "noot"] ]
?

(Thijs Kinkhorst - May 11, 2021)

[phavekes]

First case works.  Second case did not, but does now :-)

I've also added a test to see whether the hashes differ if the arrays are different. This raised the following question: two sequential attribute-arrays with the same number of attributes will yield the same hash if we do not want to store the values. Is that a possibility? Something like:

[ 0 => \'aap\', 1 => \'noot\']
versus:
[ 0 => \'olifant\', 2 => \'pepermolen\']

This works if we want to store the values, as that'll be two different arrays. But if we don't want to store the values, we use the keys, meaning both will have the key 0 and 1 after we've finished normalizing the arrays.

In all we now have the following tests:

• switching order
• switching casing
• switching both
• sparse array
• empty attributes
• different hash for two different sequential arrays (with the caveat as above)
• different hash for two different associative arrays
• using an item with multiple values vs using the same item with only a single value (Koen Cornelis - May 11, 2021)