This pretty much goes without saying, but in the lategame or even in the midgame, you never really build more than 1 copy of a recruitment building type. Because it takes too long to build them, you can just send a single lord back to recruit at the home base and money is king. Not to mention money is always better than more recruitment options. It's just kind of the meta. Larger armies win over better armies.

So what I'm suggesting is in the future to remove garrison buildings as a concept, and instead put the garrison units on the recruitment buildings tab. That way there is one more reason to build recruitment buildings.

I would even go as far as significantly increase the supply lines penatly, just because having the ability to field an extra army over upgrading your units gives such a significant advantage.

If I have 2 armies of peasants, and you have 1 army of men at arms, I can block your army advance by placing one peasant army in the city closest to your men at arms army, and with my remaining peasant army I can obliterate all your minor settlements. More armies is better than elite armies. And i haven't even touched on that I can let them reinforce one another, and having a ton of peasants will overwhelm the men at arms army, causing me to win anyway when the fight is fair.

So in short:

-Remove garrison buildings

-Add garrison units to recruitment buildings

-Increase supply lines penalty