The golem cannot track chest until collides with him for the first time via pushing or maybe detecting it. He cannot put items inside chest if the chest is 1 block below (haven't tested above) and is blocked to that chest entity not pathfinding a better option. Im using performance mods such as:

Performant
RandomPacthes
AIimprovements

Maybe one of the is making the error