I personally think this system is annoying and forces the player to play a certain way. For example if you are Tyrion or Arial once you conquer ulthuan going to dark elf lands is always the next place to go because other directions penalize you with orange or red climates. Another example of this is teclis currently. I think his campaign is challenging and fun in terms of start location but all he has is the small southern section of the continent then after that he is surrounded by red climates except for the sparse mountain range where the dwarfs are. The next green climates for him are lustria cathay empire and ulthuan all of those are incredibly far. (not including the islands around the continent) This makes expanding either a huge pain because of the red and yellow debuffs or it makes your empire fragmented as you sail to the next habitable locations. High elves are my favorite so that's why I have a bunch of highelf examples but another example would be karl franz and the empire. Once the empire is consolidated your next choices for land to conquer is all mountains and not optimal but skipping that land makes your empire fragmented. This in my opinion is made even more silly by the fact that if you play gelt also a human empire faction now mountain ranges are great for you.
My solution is to consolidate all legendary lord faction specific climates under the race or just rework the system. If gelt can settle mountains idk why franz cant. Another annoyance this will solve is certain legendary lords having to work way harder for legendary buildings than others. For example by the time tyrion captures the settlement Tor Anloc there's a special building there that gives a cool defensive advantage from dark elf invasions. The only problem is it takes so long to get the settlement to tier 5 and build the building special building its not even useful against the dark elfs they are long dead before you finish building it. But eltharion and imrik can settle this climate just fine along with the rest of ultuan so only Tyrion and Arial are losers.
I don't have any real solutions other than the one I mentioned so if you have thought lets hear it cause I am really tired of the current system. Whatever solution works as long as it solves for the problems I mentioned above