Yea, the issue is resonant vibration. Every helicopter has a specific resonance that it vibrates at, even full scale. If you have ever been in one when it was firing up, you felt it shake quite hard. Even though you do speed the head to, through then past this point, there is still vibration that, in one way or another, could make it to the sticks with heavy balls on them. They may be hollow, but they are weight. The vibration may be 1/4 frequency, 1/8 frequency, 1/64 frequency, but they will vibrate and that vibration is transferred into the gyro. So now you have the gyro not only fighting the vibration of the helicopter, but a much more coarse vibration as well from the training gear wobbling around.
Now, with all of that said, if you feel in your gut that you need it because you are trying some new part, setup or whatever, then definitely use it. Just know the vibration could get out of hand and you will have to land quickly. Have your finger ready on that Hold switch.