Question about NormDistBase

[niklasnolte]

Hey,
this is a question more than an issue.

In your paper the sortnetlayer looks something like this: x_k = w.T @ sort(act(x + b_k)).
This should be implemented in NormDist(Base), right? I assume a simple MLP, no dropout or anything, so the first significant line in NormDistBase.forward is this:
https://github.com/zbh2047/SortNet/blob/main/core/models/norm_dist.py#L94
x = norm_dist(x, self.weight, self.p, self.groups, tag=self.tag)
and then the next thing (no mean shifting) is
x, lower, upper = apply_if_not_none((x, lower, upper), lambda z: z + bias.unsqueeze(-1))

In norm_dist (not using the custom cuda fn, because im on cpu):
y = input.view(input.size(0), groups, 1, -1, input.size(2)) - weight.view(groups, -1, weight.size(-1), 1)

thats the only time the weight seems to be used, and its not being used for multiplication.

also i don't find the sorting or biases that are of the right shape (from your equation in the paper it seems like the biases should have something like [fan_out, fan_in] as shape or so, but in NormDistBase i only find the neural network default [fan_out] bias.

I am clearly missing something, can you enlighten me how to write a full connected SortNet MLP?

My attempt was at a single layer was this:

class SortNetLayer(torch.nn.Module):
    def __init__(self, fan_in, fan_out, lip=1, kind="inf"):
        super().__init__()
        self.weight = torch.nn.Linear(fan_in, fan_out, bias=False).weight
        self.bias = torch.nn.Parameter(torch.randn(fan_out, fan_in))
        self.norm_fn = lambda x: "<a longer function that norms in the right way>"
        self.act_fn = lambda x : x
        self.sort_fn = lambda x: torch.sort(x)[0]

    def forward(self, x):
        x = x.unsqueeze(1) + self.bias
        x = self.act_fn(x)
        x = self.sort_fn(x)
        W = self.norm_fn(self.weight)
        return torch.einsum("oi,boi->bo", W, x)

[zbh2047]

Hey,
Thank you for raising this question! Actually, since the code is adapted from L-infinity distance net, there is some mismatch between the name in the code and that in the paper. Here, the name weight in the code is the bias in the paper, which is of shape [fan_out, fan_in]. The code does not consider sorting or multiplying weights in training, because we can simply use dropout to accelerate training.
For testing, sorting and multiplying weights are used. See the code
https://github.com/zbh2047/SortNet/blob/main/core/models/norm_dist.py#L88
Note that the weight is fixed by geometric series determined by parameter dropout.

Hope this can address your questions, and feel free to ask if you have any other questions.

[niklasnolte]

is there anything i should look out for when performing a regression task with this architecture?
without having read very carefully, dropout and fixing weights sounds like something that might be a little problematic for regression

[zbh2047]

Yeah, I think it should work for regression tasks if the target function is 1-Lipschitz continuous with respect to L-infinity norm. One difficulty I can figure out is that SortNet (or L-infinity distance net) is not smooth. If the target function is smooth, there could be a mismatch (though this may not be a severe problem I think).