(question) paired test, data frame sorting, t-test_paired, Wilcoxon

[pas-calc]

When applying a paired test, the sequence of the data is crucial.

Example

>>> df = pd.DataFrame({
...     "person": ["A","B","C","C","A","B"],
...     "day": [1,2,2,1,2,1],
...     "value": [1,2,3,4,5,6],
... })
>>> df
  person  day  value
0      A    1      1
1      B    2      2
2      C    2      3
3      C    1      4
4      A    2      5
5      B    1      6

Applying scipy.stats.ttest_rel on the paired data:

>>> # sort value by person A,B,C
>>> values_day_1 = df[df["day"]==1].sort_values(by="person")["value"].values
>>> values_day_2 = df[df["day"]==2].sort_values(by="person")["value"].values
>>> values_day_1, values_day_2
(array([1, 6, 4]), array([5, 2, 3]))
>>> from scipy.stats import ttest_rel
>>> ttest_rel(values_day_1, values_day_2) # call the function on the paired data
TtestResult(statistic=0.14285714285714288, pvalue=0.8994962184740788, df=2)

But

x = "day"
y = "value"
pairs = [(1,2)] # day 1 to day 2

ax = sns.boxplot(x=x, y=y, data=df)
ax = sns.stripplot(x=x, y=y, data=df)

annotator = Annotator(ax, pairs, data=df, x=x, y=y)
annotator.configure(test='t-test_paired', text_format='star', loc='inside', verbose=True)
annotator.apply_and_annotate()
plt.show()

returns wrongly
1 vs. 2: t-test paired samples, P_val:6.667e-01 t=5.000e-01

Of course it cannot know that it should compare the one person with the same person for the relative test.

Fixing it:

>>> df.sort_values(by="person", inplace=True)
>>> df
  person  day  value
0      A    1      1
4      A    2      5
1      B    2      2
5      B    1      6
2      C    2      3
3      C    1      4

returns correctly 1 vs. 2: t-test paired samples, P_val:8.995e-01 t=1.429e-01 (same as from scipy.stats TtestResult)

The same would go for 't-test_paired', 'Wilcoxon' (scipy.stats ttest_rel, wilcoxon) or any paired test.

Question
Is there a function to tell our annotator which is the identifier label for objects to pair (in this case it is by "person") ?
It could also then find out if pairs exists or some person did not participate on the second day for example.

Note
If we transform from wide (with identifier as index) to long format, the data is already sorted correctly in each group.
Or we change it to wide, so that identifier is grouped correctly:

>>> df_wide = df.pivot(index='person', columns='day', values='value') # long to wide
>>> df_wide
day     1  2
person      
A       1  5
B       6  2
C       4  3

[jhaberbe]

I sent a fix to this. There is a question I would ask, maybe whoever is reviewing this, to ask what behavior they'd prefer.

I made the function use pd.concat and joins on index, because it preserves more data in more cases. Suppose we had comparisons between groups (1, 2, 3), where the specimens in each look like:

1: {A, B, C, D}
2: {A, B, C}
3: {A, B, D}

Then doing pivot, depending on if you decide to drop comparisons where there were NAs, would lead to a bunch of data being lost for no reason, which we can keep if we set a data index (with multiple values) in the case of paired data, and no paired data in other cases.

Just blindly doing it has some problems:
1: If you potentially have a single draw of a sample in one group, with multiple draws in another group, then this will duplicate it once. Just in this case, I could see someone seeing it as a bug or a feature.
2: If you have multiple draws of a sample in both groups, then it would produce n * m comparisons, which is just awful.

Right now, I've put up a drop_duplicates, which just blows past the problem. It might be better to yell at the user that this could be a problem, in which case we can just make a function that checks for duplicates at the beginning that looks like:

if data.value_counts([paired, hue]).max() > 1:
     print("Duplicate samples within groups, consider update the names so they're unique.")

Here is the main code that solves the problem, I ran the unit-tests and it came with a single error, but looking at it I don't think its connected? I'll follow up on this, and I've posted the message below.

    def _get_results(self, num_comparisons, pvalues=None,
                     **stats_params) -> List[Annotation]:

        test_result_list = []

        if num_comparisons == "auto":
            num_comparisons = len(self._struct_pairs)

        for group_struct1, group_struct2 in self._struct_pairs:
            group1 = group_struct1['group']
            group2 = group_struct2['group']

           # Changes here
            if self.paired: 
                group_data1 = group_struct1["group_data"]
                group_data2 = group_struct2["group_data"]
                table = pd.merge(
                    group_data1.reset_index(),
                    group_data2.reset_index(),
                    on=group_data1.index.name
                ).drop_duplicates(g1.index.name)\
                .set_index(g1.index.name)

                group_struct1["group_data"] = table.iloc[:, 0]
                group_struct2["group_data"] = table.iloc[:, 1]
               # End of changes

            if self.perform_stat_test:
                result = self._get_stat_result_from_test(
                    group_struct1, group_struct2, num_comparisons,
                    **stats_params)
            else:
                result = self._get_custom_results(group_struct1, pvalues)

            result.group1 = group1
            result.group2 = group2

            test_result_list.append(Annotation((group_struct1, group_struct2),
                                               result,
                                               formatter=self.pvalue_format))

        # Perform other types of correction methods for multiple testing
        self._apply_comparisons_correction(test_result_list)

        return test_result_list

.........................................
======================================================================
FAIL: test_seaborn_violinplot (test_positions.TestPositionPlotter.test_seaborn_violinplot)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspaces/statannotations/tests/test_positions.py", line 44, in test_seaborn_violinplot
    assert value_maxes[("Sun", "Male")] > 0.80
AssertionError

----------------------------------------------------------------------
Ran 178 tests in 31.370s

FAILED (failures=1)

Sorry if this is a bit of a long response.