There is currently a loyalty penalty for removing a governor from office (-2 loyalty for office lost) while removing a general does not confer the same penalty. I think that this penalty should be removed entirely.
There are times when you want to replace a governor with one that has better or more appropriate traits. You could have hired a governor who has good command starts, buffs unit stats and has a primary trait geared towards fighting because there wasn't one available that had governing traits ( bonus to farm, church or overall income) but down the line one does become available and you want to swap him out. Perhaps you want to build some traits for a character before they become a general by governing certain areas that offer specific traits or you want to build some general only traits for a future governor. I like to build up my heir's traits, stats and influence safely in a governor position before I assign them to an army so that once they do become King they will be on a similar level to the previous one. I also like to have generals who have some starts or traits built up before they lead an army so that they perform better in battles.
The current penalty of -2 loyalty needs to be reduced or removed as it does not makes sense that you can swap out generals with no penalty but doing the same for governors does incur the penalty.
Furthermore, I think that we should have more than just 3 candidates to choose from for governors or generals, excluding family members or any statesmen that may be around. This really limits your choice in who to appoint and usually you have three candidates who are all better potential generals than they are governors. It is also immersion breaking as I find it hard to believe that there are only 3 people in a kingdom to choose from for a position.