If winch head starts oscillating it will never stop. Standard spring damper setting doesn't help since it only affects the spring.

Using of ConfigurableJoint may be the solution. It allows settings damper forces on every axis.

Another option could be shifting cable attach anchor relative to the head's CoM. In this case gravity force will eventually calm down the head.