It switches? If that is the case, then I'm going to bet that you have a gyro going out and the only way to replace that is to replace the whole board. I'm no expert on these, but that would be my guess.
You are letting it sit on the ground very still while you initialize it after plugging in the battery, right? You can't hold these in yoru hand when you plug them in. It will cause the gyros to calibrate wrong.
This could also be dry bushings in the motors. Sometimes they grab, sometimes they don't. I have seen that, but not on a quad.