Nerf patrols
Patrols are incredibly annoying right now. They can be anywhere from a pushover to several times stronger than anything else at your current level, and they tend to cluster and block off areas like sandworm making you run back and forth until they move because the risk isn't worth it.
Experienced players solve this by just pre-casting Movement Infusion before they enter the world map. Everyone else just suffers.
Still, there is value in the mechanic, the challenge of patrols is interesting and they're a unique loot source for builds that want to manage the risk. The only real problem is how tedious avoiding them is. This cuts the Orc and Adventurer patrol movements to 50% so you can outmaneuver them without silly move speed gimmicks.
If this is for some reason unacceptable, another method would be to open up parts of the world map a bit so there aren't so many chokes where avoiding a patrol at 100% move speed is not guaranteed to be possible. Honestly, maybe we should just do both.