Skip to content

Comments

added learning rate bias correction and corrected m_t#15

Open
dchatterjee172 wants to merge 1 commit intoopenai:masterfrom
dchatterjee172:master
Open

added learning rate bias correction and corrected m_t#15
dchatterjee172 wants to merge 1 commit intoopenai:masterfrom
dchatterjee172:master

Conversation

@dchatterjee172
Copy link

Added learning rate bias correction and,
m_t = m.assign(tf.maximum(beta2_t * m + eps, tf.abs(grad)))
this line is wrong because if (beta2_t * m+eps)<0 and grad==0 then g_t = v_t / m_t will be undefined. correct line is,
m_t = m.assign(tf.maximum(beta2_t * m,tf.abs(grad)+eps))

@dchatterjee172
Copy link
Author

looks like correction for m_t was not needed. as (beta2_t * m+eps) is always >0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant