Valid samples very less but no invalids detected

[Akamthan]

Hi @tsegall re-posting here. Attached the profile json for the column along with sample data. Thanks

profile.json

2024-02-21 2_52pm.csv

[tsegall]

You already asked this question on #124 and I answered it there. It is because you are merging results. There are no invalid detected, because there were no invalids.

[Akamthan]

@tsegall thanks for your reply! Yes I see we asked question related to totalCount but looking at your documentation I see the following available -

totalCount - The total number of elements in the entire data stream (-1 unless set explicitly).
totalNullCount - The number of null elements in the entire data stream (-1 unless set explicitly).
totalBlankCount - The number of blank elements in the entire data stream (-1 unless set explicitly).
totalMean - The mean for Numeric types (Long, Double) across the entire data stream (null unless set explicitly).
totalStandardDeviation - The standard deviation for Numeric types (Long, Double) across the entire data stream (null unless set explicitly).
totalMinValue - The minimum value across the entire data stream (null unless set explicitly).
totalMaxValue - The maximum value across the entire data stream (null unless set explicitly).
totalMinLength - The minimum length for Numeric, Boolean, and String types across the entire data stream (-1 unless set explicitly).
totalMaxLength - The maximum length for Numeric, Boolean, and String types across the entire data stream (-1 unless set explicitly).

There is no totalInvalidCount or totalValidCount so we had assumed it works for whole data. Please help how we can fetch these values

[Akamthan]

@tsegall I closed by browser mistake. Please help above query

[Akamthan]

hi @tsegall kindly let us know any updates if this is possible or any workaround.

[tsegall]

No workaround I can see, more generally I am not sure what you are asking for is possible.

If I have a data stream consisting of the number 1 to 13,000 followed by alternating male, female for the next million entries. Then you would expect this to return the semantic type Gender with an extremely high confidence.

OTOH since the maximum cardinality is 12,000 we will lose entries and be unable to tell you how many were invalid since they were not declared invalid until after the maximum number of entries had been passed and some had been lost.

Perhaps you should give me a sample data stream with the results you observed and the results you expected.

[Akamthan]

@tsegall I am talking about matchCount field here. We use that to populate valid count. Now you can see it comes equal to sample count which is wrong because we are merging results so sample count maybe less. You can look at the attached profile and data for same.

[tsegall]

I need to review in more depth. As per my example above this is in general not possible. The question is whether I can provide what you are looking for in most cases.