Target and Local Brains Simultaneous Update

[davera-017]

In the class Agent the local and target brains are getting updated with the same frequency. In the case of DQN, for example, this is an undesired behavior since the local brain should be trained more frequently than the target brain gets updated.

class Agent(BaseAgent):
  ...
  def update(self, experience):
          self.memory_buffer.add(experience)
          if self.update_step % self.update_every == 0:
              try:
                  batch = self.memory_buffer.sample()
              except RuntimeError:
                  # If the batch can't be obtained, skip the update proc
                  return
              pred, target = self.estimator(batch)
              self.local_brain.update(batch, pred, target)
              self.target_brain.update_from(self.local_brain)
          self.update_step += 1

Also, in a similar direction, the Train class forcibly performs an update of the brains via agent.update() at every step. I think that it is a good idea to add a train_freq parameter such that one gains control over the experiencing/training ratio.

[aristizabal95]

There's a small detail that is kind of hidden here, which is that currently brains have a soft update implementation, which accomplishes the same thing as the hard, infrequent update of the target brain.

https://ai.stackexchange.com/questions/21485/how-and-when-should-we-update-the-q-target-in-deep-q-learning

What could be improved is providing a way of choosing between soft and hard updates. If we can come up with a modular and simple to grasp approach that would be awesome!

We could maybe have two implementations of the QNetwork brain which only differ by the update_from implementation? Another option would be coming up with a good design approach that decouples the brain implementation from the soft/hard implementation. Let me know if you have any thoughts on this!

As for the train_freq, here it is represented by the update_every, but this is unrelated to the hard-update approach.