If you got it through being a (half) kui-tan, TFing into a different race will remove it. If you got it through 'Nuki Nutbutter, no, removing your balls is the onlny way to lose it - though if you regain the balls, you won't regain the perk. Note that it'll also take Cum Cascade with it, if you have that.