Then what would be the best perks for a spellcaster?
Perks like Mage, Archmage, Grey Mage, those types are good for basic increases. I'd also suggest traditional spellcasting, as it boosts your spell damage when using a staff and tome, never mind the boosts the staves and tome already give you to spellcasting.
My build tends to start out going full dragon, including the 3 draconic lung perks to get as many points out of that, and then trying to balance out kitsune transformations to eke out the most int/wis boosts, while still retaining some str/tou to keep my HP high. Since Kitsunes and Dragons get boosts to speed, you can also use perks like Cheetah, which gives you a bonus to your HP based on how many points of speed you have.
Fortress of Intellect is also a must get, as it converts your str bonus from using Might to an int bonus, boosting spellcasting and mana pool for the duration of the spell. Set Might on autocast, get some enchanting perks, and sweep for days. Also get the conjuror perks that boost your int/wis at a cost to your str/tou. By finding the 3 colored, 5 colored, and 7 colored pearls and using them, you can gain the int/wis bonuses without losing out on str/tou