Seems like the androgynous perk cannot be lost by losing the muzzled face flag like explained in the wiki, currently have a human face with no muzzled flag and still have androgynous perk
Post save pls. As far as I can tell the code is working as expected, so I'm wondering if there's something odd about your character specifically causing it to fail.
The perk gets removed as expected; you've just got to toggle some form of time passage. It's tied to an update routine for the PC that is only triggered when the gametime updates, so take a single step and the perk will clean itself up.