Indeed, not everyone will share the same interest for the flavor of ice cream you like, so tastes will vary from person to person (no matter what sexual preference you subscribe to or sex you are born in). Some flavors will invariably be more commonly liked than others thanks to the freedom of choice, but whatever turns you off or on is your own personal preference. As for others, people will like what they like, so as long as you aren't telling them they are wrong for liking something, then you'll be okay. Once you start accusing someone's opinion of being wrong (especially on a moral level) and that they should change it, then that's when you'll get a community backlash. Simply stating an opinion shouldn't get the pitchfork mob on your case, at least not normally anyway.
I am certain you are not alone in your preferences for the average. When you get to know a lot of people, you realize that tastes will vary from person to person, both in and out of smut. There are too many variables to what makes up a personal preference so I cannot say with certainty why exactly someone likes what they like, but I do know that, generally, part of it is social and the other part mental/biological, and like the actual taste buds of the tongue, a person's tastes may change over time with some preferences becoming stronger, weaker, broader, narrower, more attractive, more repulsive, etc. over time based on the individual's age and the amount/level of content they consume.
As far as fantasy smut is concerned, I don't think I have a strong dislike in anything--unless defecation and gory violence is involved, then it gets a bit irksome for me. I do favor the exaggeratedly endowed because that tickles my taste buds more than just average on average. The tickling is even more pronounced to me if the character starts off as an average or below average figure and transforms into the overly endowed sex diety. The size of the figure being a visual/conceptual representation of how lewd/horny/lusty the character may be. To me, that's pretty hawt, but I know not everyone will share that interest. Some people are more attracted to the acts of doing something or even mental states of certain characters and these can be the things that gets the boners--it could even be a combination of any of these--again, this stuff will vary for different people on the spectrum of interests.
In addition, I also make sure that my personal preferences in fantasy stay in the realm of fantasy, so my real-life preferences will differ dramatically because I know I'm dealing with a different set of variables. From my understanding, some people like to have their reality and fantasy bridged and have them cross over--while you can only do so much to alter your reality, your fantasy is what you make it; so if you want your alternate reality to be a bit more average in terms of a character's proportions, by all means have at it. I do agree that content for such an interest is limited, but it's out there--you'll either have to dig more to find it, or start creating your own if you want that audience to be relevant to you (if you build it, they will come!).