I switch to the other team when it has like a ZERO percent chance of winning. Or if they were doing good before but lost some team members and are being overwhelmed.
But most of the time I think it's fair-game when teams are unbalanced. Like in life, the odds aren't always in your favor and a lot of the time both team-work and your own strategy move up to a higher level when you try to survive in such unforgiving environments.