The rudder setup on 3gx took me 2 days and probably 6 tries before I figured mine out.
Not sure of what radio your using, but I have a DX7s and could not find much info on the web for using this radio and the 3gx, I seen plenty on DX8 and they are mostly the same but some features and wording in the setups are a little different.
I have only messed with fly-bared and gyro was always (gear channel 5).
1st. AUX2 Chanel 7 for the tail gyro on the 3gx, I also didn't know what switch to assign this too, seen some say the F-mode switch, and others the Rudder D/R switch, the DX7s has a gyro switch with 3 positions I assigned mine to that switch.
The whole rudder setup has to be done before any gain is set in your radio, it will not work or find center if you have gain already set in the TX,
( ask me how I know this....

6 times...)
I left position 0 on my gyro switch to 0% for this, so I guess this is RM, on fly-bared I never set two different modes I only fly in HH, so the RM in the manual was throwing me for a loop..
After you go through all the rudder setups,
Band
DS/AS
Gyro Nor/Rev
Limit
Size and Delay, Red 450, Green 500+ (never could get my delay to 0 can go to computer and get 10 but when I press set it goes back to 30)
Anti torque face up green, face down red.
power down, set your gain in your radio then if you select a switch with 2 positions leave one at 0% and the 2nd to your gain. when the switch is on the 0% both lights on the 3gx will be red for NR mode, when switched to the one with gain status light will change green for HH mode.
I also noticed that my radio % to the actual gain on the gyro is 6 points off, I had to put 41% to have 35% in the gyro.
Hope this helps.
I'm a newbie, and this drove me nuts. lol