Twice I have recovered my stumble only to have it fall into the hands of the enemy (hee hee!)! Until they fix this one beware of recovering stumbles as they will turn against you! (this has only happened when playing on my own - I don't know if it will happen with other players)
I think I know what is going on! When I open the recover box my opponent's stumble is (sometimes - if used) in there and if I choose it it goes into their set of dice. So you just have to keep close track of the order of dice used and choose your own. When I choose my stumble it works just fine. Hope this isn't too complicated!