Very small gradients causing no weight update from your model

[oneTaken]

Thanks for your code.
It helps me understand the BiDAF in details.

However, I found the model had no performance increasing.
Every epoch, the metric is always the same.
And then, I found it's the optimized gradients too small.
it's the order of 10^-3~10^-8.

I can't find what's wrong.
And I think your code is good to understand.
So, what may be the problem?

[jojonki]

@oneTaken Hi. Thank you for your comments. That will be nice to debug. I am also still facing the low performance problem. I am exactly checking now.
If you find anything wrong points, please let me know.

[jojonki]

I am checking the code by removing p2 layer (only predicts the answer of beginning)

[jojonki]

Last layer should predict the index of word instead of char level index.
91aa342#diff-8f329902bf6ef6e1af03f01d0b9633e2

[oneTaken]

I should comment earlier. I thought your code runs well, and I have it's due to my incorrect coding.
There are some points during my debug phase:

1. your p1 = F.log_softmax(self.p1_layer(G_M).squeeze()) # (N, T) seems have something unresaonable. G_M is size(N, T, 10d), Linear layer seems must be ndim 2. I code as the new p1 layer as. The ans_size is 1, which is said in the paper, and then:

G_M = G_M.view(N*T, 10d)
p1 = self.p1_layer(G_M).squeeze()
p1 = p1.view(N, T)

so, the answer begin must be in the index [0, T).

2. I am also removed the p2 layer , and to test the begin index performance. And I found that the p2 may be not the point. Just use begin (or end) and CrossEntropyLoss, the performance is still very low. The loss value is high, ~5 for one, ~10 for two both. And the gradients is very small, 10^-3~10^-8, maybe this is the point.

I'm trying to use some visualizing analysis to help debug. Do you analysis the gradients?

3. Besides, I also have a little different coding in the character embedding layer. But for now, it
   seems doesn't matter.

[jojonki]

About 1. I have to confirm this. The code only affect last dimension in master, it does support it. I am not sure in 0.2, actually it looks working.

fc = nn.Linear(10, 20)
a = autograd.Variable(torch.randn(32, 5, 10))
b = fc(a)
print('b', b.size()) # 32, 5, 20

• 0.2
  http://pytorch.org/docs/0.2.0/nn.html?highlight=nn%20linear#torch.nn.Linear
• master
  http://pytorch.org/docs/master/nn.html?highlight=nn%20linear#torch.nn.Linear

About 2. I also confirmed that. I think it is because learning rate is too big. For test, I changed the optimizer to Adam with default values. Now the parameters data looks good.

About 3. There are ablation tests in the paper. Yes, currently we can ignore char embedding layer which does not big impact to the performance.

[oneTaken]

Thanks for the question1 answer, i got it . I should follow the upddates.
As for 2, have you got a not bad performance?
I have some question about the character embedding, after solveing the performance problem. We can talk about it.

[jojonki]

loss is decreasing and weights of parameters looks natural. But the performance was not still good (13.9% on test data) after 10 epochs..
It will take a more time to debug this model. If you're familiar with TF, you should choose official bidaf model, https://github.com/allenai/bi-att-flow.

Anyway, I am debugging this model.

[oneTaken]

Actually, I trained a keras model, which reproduced the paper result totally. But it's hard to understand the paper deeply, even though I got the result. So, I try to code with pytorch. I have to say, it's a hard way. And I am debugging the model too.

[jojonki]

@oneTaken That's nice. Do you have the code of keras? If it's possible, I want to see it. Before I start to use pytorch, I was a keras user. :)

And current master version (t 36b9e4c1)'s performance is bad. (around 9% for answer start/end after 4 epochs)

[oneTaken]

Emmm, I know you are familar to keras. I watched your respositories. Good job.
But sadly, the keras code is not mine. So, I have to train with code myself.
I am debugging, my model performance is even lower.

[jojonki]

I see. BTW, training accuracy was about 21% and 25% for answer start/end after 10 epochs on current master.. Today I'm gonna debug the model

[jojonki]

So far, there are not improvements even I modified following items...

• RNN -> LSTM
• original loss function
• use BiDAF's script to build dataset

[oneTaken]

Emm, I am trying to wrap the code into tensorboard. So I can compare with the keras training log, to have a more clear knowledge. By the way, I have a deadline recently. So I can't spend all my time to solve this. But If I have some improvement, I will tell you.

…

On Thu, Dec 7, 2017 at 3:02 AM, Junki Ohmura ***@***.***> wrote: So far, there are not improvements even I modified following items... - RNN -> LSTM - original loss function - use BiDAF's script to build dataset — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARNbsCNdzmMJQPpB2oxXYegMrYmozT0fks5s9uTPgaJpZM4Q0IWy> .

[jojonki]

I may notice most weights are not updated and quite huge values. Now i am using tensorboard through https://github.com/lanpa/tensorboard-pytorch.

[oneTaken]

Yeah, I am also use this repository. The weights says they were not updated. And there are gradients, so the graph can flow through the layers. So, there may be something unexpected about the *optim*.