Minecraft Ver. 1.12.2
Mouse Tweaks Ver. 2.8

When trying to use the scroll when transferring items from one inventory to the other, it just picks up the whole stack. I've tried messing around with the settings and nothing works.

This usually means some other wheel tweak mod is interfering (like item scroller, simple invertory sorter or NEI). Try disabling Mouse Tweaks' wheel tweak.