The only time I think it really becomes a problem is if you level up too quickly and don’t get the full stat bonuses and end up with a busted build that can’t damage the now godlike bandits

There’s a mod that addresses this! I would go for the “lite” version if you just want the armor and level scaling tweaks. That mod as a whole changes a bunch of other stuff I didn’t care to change, but the lite version is nice.

But it’s also goofy in Skyrim when they keep much better gear in chests while attacking you with fur armor on