A lot of people I have chatted to over the lifetime of TW:W have all brought up the same minor inconvenience when it comes to managing your empire, and that is how ports make some minor settlements extremely cramped, basically leaving you with only 2 slots, or 1 if you tend to build walls everywhere and aren't playing Bretonnia.
My simple suggestion is to integrate it into the main settlement building. Similar to how when playing as the High Elves, you can have an elven colony to settle, which gives different bonuses and units defending than if you settled a non elven colony.
These could be classed as port-towns or whatever classification you see fit. By doing so it would allow for more flexibility to province management.
This could be balanced by making the settlement building more expensive to upgrade and much more of an investment. Perhaps this would only apply to minor settlements where real estate is harder to come by and you can still upgrade ports separately in Province capitals. Or You can make them all port settlement buildings but give some special port cities like Marienburg, a special Landmark port building that will replace the current special ports these regions have.