I didn't think too much of it at first, but I don't think I'm supposed to be able to keep the perk below 4 score? Think something messed up when the racial score system got reworked...
This is how certain perks worked in older versions of the game, going to keep this report up in case you are actually supposed to lose this perk when below 4 point of racial score.
I've been keeping it without the requisite racial score needed to gain it for a long time. Even back in the Flash version, this has been my experience. So, either there was no racial score requirement to keep it or it's a bug that nobody noticed or bothered to care.