Clicking on bounds handle that isn't over the selection breaks selection

I assume because IsMouseOver isn't returning the right value and so then the brush is deselected. But I am not sure why it's then not possible to select it again...