Reward structure for custom MARL environments

[nathanDuncan]

Hello, I'm new to MARL and am trying to create more environments for multi-agent competitive games (pursue/evade, reach/avoid, etc.), like the dogfight example except each agent may not have the same objectives. I've found structuring the rewards to be very difficult and my models just don't seem to converge to the behaviours I am expecting. Even when trying to copy the structure of the provided multi-agent environments (which do work very well) as closely as I can, the agents just don't seem to learn. I was wondering if you had any advice for how the rewards of these environments were designed e.g. how the shaping rewards were made and how you balanced them relative to the sparse rewards.
Also I am currently training the agents using a variant of Self Play with stablebaselines3 (only one agent is learning at a time against the other agent's fixed policy), but if you have any recommendations for changing my strategy I would be open to changing it too.

Thanks!

[jjshoots]

Hi, apologies for the late response, it's been awhile since I've done reward shaping and RL training. But from what I can recall, what I do is:

1. Define a set of sparse rewards, which define the absolute best behaviour that the agent can have. For example, in dogfighting, agents get points for each hit, and also when the enemy is downed.
2. Define smoothing rewards for the agent to achieve those sparse reward goals. But these have to be well bounded values, and the long tail sum (reward / (1 - gamma)) should be less than the sparse reward value. This is so the agent doesn't exploit this reward and ignores the sparse reward one.
3. Define progress rewards, which help the agent make progress toward the sparse reward goals. These are usually defined as $\Delta x$, where $x$ is some state. For example, in the dogfighting case, $x$ can be the nose angle relative to the opponent -- the agent is rewarded for making progress on this angle, and penalized for regressing on this state. The scale of this can be fairly high, but should be ~10x less than the absolute value of the sparse reward. I also found it useful for this reward to be assymetric (more positive for making progress than negative for losing progress).

That said, reward design is still challenging, and I took a couple months to nail down the appropriate reward landscape for the environment.

Hope this helps!