The initial starting stats don't represent your character's background, no. That's just how game design generally is - if each new character was maxed from the get go (or at least have one high core stat) it would take the progression out of the game.Obviously.
Let me say it like this, it's disconcerting that all three of these together put such great emphasis on martial tradition yet the result is someone who isn't really any stronger than your typical elf or catgirl (at least at first). I'm sure you can attest to someone like Brint fulfilling the very classical "Barbarian Warrior" archetype and he has an abundance of Strength and Toughness. There's a certain juxtaposition here between what the game's descriptions tells you and the actual mechanical outcome, no? Yes there's the affinity system I'm well aware of, but fresh out the gate how does 2 strength represent my character's background?
The game kind of alludes in your PC's backstory that their affinities are the boon they have gained from their prior life - not raw stats. So, your PC would be able to progress much faster in those specific stats than other baseline characters with no affinities for those stats.
It's not always a perfect 1:1, as with the Brint/Brienne Wayfarer's scenario - they have much higher Strength than the PC can achieve - and Atugia has a much higher Toughness than the PC can achieve - but ya know, Companions need something to differentiate themselves.