I want to hide nameplates that are further away than my current character's range. So I don't want nameplates to show after 40 yards. Since blizzard introduced the 60 yards min nameplate distance, that seems impossible. So is there a way to reduce the alpha of nameplates after 40 yards from my character? I tried messing around in settings but I could never make this work.

The best you can do is probably to mess around with Min/Max Alpha & Min/Max Alpha Distance but it's hard to get right and it won't be perfect.

Yep I tried with those and as you say it's hard and not exactly the results I want.