First off, let me give my highest compliments, to the fine job, you are doing, with this mod.
Down to business, I can set up a classic J-K flip flop using an R/S Latch and AND gates.
This performs quite well in an UP/down counter circuit. However any attempt to reduce the circuitry in the IC workBench produces the overload error.
Again, thanks ever so much for a great mod.

Direwolf20 modpack for MC 1.12.2 with Project Red added (all downloads including compat)
ProjectRed-1.12.2-4.9.1.92
RS latch can be constructed from Nor and Nand gates in real life,
The Nor gate RS latch does not respond at all on the ground or in the IC workbench, so I could not repeat the error.
The Nand gate RS latch does work quite well on the ground, but not in the IC workbench.
https://www.dropbox.com/s/vndujh6bpjybxf4/2018-10-27_14.49.10.png?dl=0
https://www.dropbox.com/s/lb80yhei3zvuy2v/2018-10-27_15.11.54.png?dl=0
A standard logic chip RS latch from Project red is doing well in both the ground layout and IC workbench.
The initial report was a stand alone of Project red in MC 1.12.2 using Creative mode.
Why there is a difference now, I have no idea. I hope this helps.

Can you provide a screenshot of your exact setup?

This overload error is because the gates inside of an IC don't produce any delays. Therefore, you can't design it the same way as you would in world. You are basically creating an infinite loop between two gates. You need to introduce delays with a repeater.