Add custom validation

[em-baggie]

I tried a few different ways of doing this and thought this was the best option, but please let me know if there's a better way!

Changes:

• Added custom regex to CodelistOptions as an optional String
• Added checking if string can be parsed to regex when new codelist is made with the new() method
• Added a new CustomCodeValidator which has method called by the Validate trait method when the custom regex has been set - this means if the custom regex is set, the validate_codes() method uses custom validation but if it is set to None, it does validation depending on codelist type. I assumed there wouldn't be a case where you needed to do both on one codelist.

I initially wanted to include the Regex in the Codelist struct (so only compiled once when codelist is made) but this made implementing the required traits with the attributes difficult (e.g. serialisation, PartialEq). Using String avoids this and also allows checking of valid regex when codelist is made, and give custom error to the user. The regex is currently compiled once for each time validate_codes() is called.

I noticed the python bindings currently don't allow for custom CodelistOptions to be implemented when creating a codelist or codelist factory as the default options are always used, and there is no functionality in the rust for updating the options as well - should we include this in another ticket?

Closes #93

[CarolineMorton]

Hi @em-baggie

Let me know what you think of this appraoch.

[CarolineMorton]

Thanks for adding this in.

[CarolineMorton]

I'm not sure about this. Be interested to get @oylenshpeegul opinion on this as well.

If we go back to the user experience, i think that wil be guide us. I think in python we want to be able to do the following:

# Create a new codelist object
c = CodeList("Pneumonia", "ICD10", "Manually created")

# Add a code to the list
c.add_entry("A119", "pneumonia")

# validate the codelist
c.validate_codes() # validates ICD

# validate custom
c.validate_codes(regex_str="^[A-Z][0-9]{2}\.[0-9]$")

Users would need to use regex in python so we could set if it accepts a string or a regex expression. Let's maybe start with a regex expression and see what that might mean for typing.

I think we want something like a match statement for using the regex or not.

pub trait Validator {
    fn validate_codes(&self) -> Result<(), CodeListValidatorError>;
}

impl Validator for CodeList {
    fn validate_codes(&self, custom_regex: Option<&Regex>) -> Result<(), CodeListValidatorError> {
        match custom_regex {
            Some(regex) => {...actual implementation},
            None => {
                match self.codelist_type {
                    CodeListType::ICD10 => IcdValidator(self).validate_all_code(),
                    CodeListType::SNOMED => SnomedValidator(self).validate_all_code(),
                    CodeListType::OPCS => OpcsValidator(self).validate_all_code(),
                    CodeListType::CTV3 => Ctv3Validator(self).validate_all_code(),
                }
            }
        }
    }
}

With this in mind, putting the regex into CodelistOptions might not be the best place for this.

What do you think?

[oylenshpeegul]

That's a good point! If we want to serialize CodelistOptions, we can't have an Option<Regex> as @em-baggie said. But if we yank it out of there and make it a parameter instead, then why not?

I don't think I understand the Python part of your comment. Does PyO3 even translate regexes between Python and Rust? Python regexes are the fancy backtracking kind, whereas in Rust they are the more restricted finite automata kind.

[em-baggie]

My thinking was the regex could be included in the codelistoptions which can be customisable, and would link the regex to the particular codelist. But realise the way I've implemented it means if the regex is set, you can only validate with the regex and not normal validation which depends on the codelist type. Using it as a parameter is a lot more flexible, but the regex but will not be 'saved' in the codelist struct. But I guess validation notes can be added to the metadata to document if different validation methods have been used.

I think with the pyO3 the python can just pass a string and under the hood the rust can convert to regex within the method. Or maybe can just pass a regex in the python - not exactly sure how that would work. But to be honest think Caroline's method is a lot simpler and I probably overcomplicated this!

[em-baggie]

Is codelistoptions something we want to expose to python api so it can be altered by user? Maybe I'm misunderstanding the purpose of these options 🤔

[em-baggie]

I'll have a go at implementing the other approach in a separate PR then we can see them both

[CarolineMorton]

We will want to expose CodeslistOptions at some point in the Python API i think.

[CarolineMorton]

Good catch