There are 2 reasons why I think that this effect should be removed:-
1. The effect is opaque; it's virtually impossible to determine who is a subordinate of whom and why.
2. There's not enough offices to go around; with only 3 offices available and a penalty associated with anyone being removed from one, it often turns into a lose-lose situation where you can't figure out who should get the office and why(due to point 1) and any removal also nets you a loss of loyalty. Perhaps the number of offices could grow as the kingdom grows.
The whole trait is opaque and adds more frustration than any sort of enjoyment or immersion.