The Wayfort as it stands isn't a very impressive or defensible home being essentially just a Motte and Bailey (Keep and Wall with a courtyard) with one Watchtower. The Keep seems to be square and doesn't have any bastions or surrounding towers for to split advancing troops and subject them to enfillading fire. The wall doesn't mention any towers either although they are necessary for visibility, improving range for archers and artillery, and providing defense for the mages / archers within. I feel like Matiha's office has more description than a lot of the areas in the Wayfort even though its the seat of your power.
I am certainly not the be-all lorekeeper or arbiter of Savarran history or what makes sense to us in a different world. I'm just sharing thoughts on the subject and geeking out with possibly wrong conclusions.
It's a fine observation. I don't think the wayfort was ever intended to really be a fortress or stronghold. I think it was basically a border crossing. It said 'You're entering Njorholt here at the River Riddell (or the Belharan Empire if going south)'. And it's a toll station for collecting taxes and tariffs and inspection of goods coming through at a natural crossing point of the river. I don't think it every was meant to withstand sieges. Smugglers and shady merchants aren't going to be blasting down your walls.
I am not sure if you're just making observations in general, or you're commenting on the wraith attack in the Gods War that Champ and Antonina experience in visions at some point. I think fighting the wraiths, those were mostly shadow-touched beasts and such, and any actual wraiths likely come from a place without walls and such in the way we have them, and likely were having to deal with such novel obstacles in different ways, but also I think they were more trying to drive the defenders mad and make them submit as servants, rather than just smash through.
As for defensiveness, I can't recall all the lore and detail of the Wraith siege with Thalia from the vision quest, but it's possible the fort was much more sturdy, built up, and supplied before it was cut off... (Belhar liked to do things grandly). After that huge meteor smashed it... it may have been rebuilt a little more modestly, in the spartan fashion that it has today, before the frost elf schism got more out of hand. Or just... what you have no is all that's functionally left from a slightly larger footprint with a few more outbuildings or lesser bastions.
There are flying enemies, but you counter that with your archers, your own flying troops, or magic and blast them from the skies
(which we don't do in our world because the government hordes all the magic for themselves and their Illuminati masters). In those days, you may not have had teams of valkyrie coming to your aid, since the Seven weren't really active, but in those days the Rift was more cohesive, the elves didn't have a falling out with the jotun or avanai and their fliers until recently. Most likely you were dealing with harpy bandits... but they'd just fly past the place anyway. Or a random flying beast coming out of the mountains, but you'd deal with it the same way.
For reinforcements and resupply: when it was built, it had a Waystone, which was for more used in the good old days, and could easily allow from troops from leagues and leagues away as well as supplies to come in and out in record time for emergencies unless the enemy was equally versed and able to travel the Ways Between as the mages of Estelore (which most weren't, to my knowledge; Kitsune having their own Torii gates being a noted exception as currently written). They could try and block access, but then that's just another front on a war trying to stop troops.
As for now, the place may be Champ's seat of power, but they're ultimately the baron of a kingdom that's shrunk to almost forgotten status to the rest of the world and is basically one city with scattered outposts and enclaves in its immediate area (until trying to take back Dracia). And that's in a forgotten, frozen little frontier area ruled by a Baroness that has little interaction with anyone else. It's not going to ever be a palace, and if it came under siege, it would rely on the heroes and troops there to break it immediately, most likely having its gate, walls, etc, smashed in or flown over and the battle being sword, arrow, and magic, not catapults or ballista and waiting it out.
But, if you're just trying to think of things to spend in-game money on or something, certainly there can be a rune-engraved ballista, blessed by Sorra, that Champ acquires and installs that gets used in an end game assault scene to fend off some big flying beastie, or otherwise you have to fight it.
