Height and the associated extra muscle mass/reach is a big advantage for the kind of lifestyle barbarians live. There's a reason sports like Boxing, Wrestling and Weightlifting have weight categories. You might have a shorter-than-average barbarian, but they're a Guile Hero-type, comparatively.
Of course, that's if you're being realistic. If you're talking fiction, actual muscle mass doesn't mean much. Fiction is filled with smaller people defeating bigger people. We like to root for the 'underdog'.
The other thing is the architype is scaled to modern human height. People were shorter in a lot of places, historically, because of poor food. I've seen speculation that nobility were actually physically better than peasants on average. Some cultures attributed this to divine parentage (which also helped solidify the noble's rule), but the actual reason was access to more and better food, particularly meat.
