Issue type:

• 🐌 Performance issue

Short description:

As advised in #156 , I'm creating a new issue for this problem and providing further details so the team can determine whether there's a problem able to be fixed, or whether the logic at work is simply too complex to be handled elegantly at this scale.

While waiting for a fix to #112 to be implemented, I've devised a logic chain to try and prevent these race conditions before they happen. Specifically, my experience has been that if a recipe "X" is outputting materials at the same time that those materials are being requested elsewhere, whether it's simply being requested out of the network or whether for recipe "Y", recipe "X" will deadlock.

In the writers, I've used a Choice card in the Craft Recipe slot, running every 100 ticks, to craft the recipe I want if true, and to craft an empty recipe if false.

I'll spare listing the recipes for now for the sake of brevity, but I can provide them if relevant.

My problem is twofold - first, while the variable store with the logic is connected to my main network, the tickrate of my server drops to below optimal levels; I observe anywhere from 12 to 18 TPS, caused primarily, in my observation, from particularly long ticks, ranging from 400 to 600ms to resolve.

Secondly, the logic does not always seem to resolve properly. I've not been able to determine what the trigger is, but I presume it is related to the longer ticks. For this reason I don't know how to reproduce specifically this problem. But I will observe, sometimes, when I have the logic showing in a display panel, that it will update on the same tick as I see a new crafting job pop up in another panel. Other times, it will not update at all; I'll take the logic out of the panel, and put it back in, and it will update to the value it should have been; which is a value that is no longer accurate. During this time, I will sometimes observe the crafting jobs that were previously operating smoothly beginning to deadlock, as if the logic was no longer in place.

My storage solution is 7 Sophisticated Storage multiblocks; 4 10x11x1 blocks of Limited Barrels I through IV at varying tiers and with varying stack upgrades, 2 7x11x1 blocks of Limited Barrels with the same idea, and then 1 2x2x1 multiblock of (non-limited) Netherite Barrels each with a Tier 5 stack upgrade. I have an item interface on the Storage Input and Storage Output for each multiblock, operating every tick. Currently my storage has ~4.15 trillion item capacity, but there are only about 6 million items in the network.

My main network is connected to this storage solution and to the crafters. It is also connected to all of my energy providers, and also runs through a number of Compact Machines to provide power to the machines inside. A separate issue I've been having is that my Omni-Directional Connectors on Network 1 are beginning to take a long time to register being placed; if that's an unreported issue, I'll report it where it's relevant.

Steps to reproduce the problem:

. Step 1: Go get a coffee, or some other drink of your choice. The reproduction here is very involved, and might take a while.
2. Grab the following cards from the Network Reader: Aspect: Active Crafting Jobs named currentJobs and Aspect: Network Count of Item named getCountInNetwork
3. Task 1: We need to get a List of the currently being crafted items from currentJobs. Create operator cards for Operator Map map, Recipe Recipe Output Ingredients getOutputIngr, and Ingredients Ingredient Items getItemListFromIngr
4. Go to apply, and put Map on the left, getOutputIngr on the right, creating card Recipes -> Output Ingredients
5. Go to apply, and put Map on the left, and getItemListFromIngr on the right, creating card Ingredients -> List of Items
6. Pipe Recipes -> Output Ingredients to Ingredients -> List of Items, creating a new card called Recipes -> List of List of Output Items
7. Get the following operator cards: Operator Reduce reduce, List Concat listConcat.
8. Apply with reduce on the left and listConcat on the right, creating new card collapseList unfinished.
9. Create an empty list card, named EmptyList.
10. Flip collapseList unfinished, creating collapseList FLIP unfinished.
11. Apply with collapseList FLIP unfinished on the left, and EmptyList on the right, creating collapseList
12. Pipe Recipes -> List of List of Output Items to collapseList, creating Recipes -> List of Output Items
13. Create the following operator card: Item Item with Stacksize (setStackSize), and then flip it. (FLIP - setStackSize)
14. Create an integer card, with a value of 1, creating stackSize: 1
15. Apply with FLIP - setStackSize on the left, and stackSize: 1 on the right to create List of Item Stacks -> List of Single Items
16. Pipe Recipes -> List of Output Items to List of Item Stacks -> List of Single Items, creating List of Recipes -> List of Single Output Items.
17. Apply with List of Recipes -> List of Single Output Items on the left, and currentJobs on the right. This gives us our list of currently being made items, named List of Current Output Items.
18. Task 2: We need to get a card that we can apply a recipe that tells us whether its inputs are currently in List of Current Output Items. Create the following operator card: Recipe Recipe Input Ingredients (getInputIngr).
19. Pipe getInputIngr to getItemListFromIngr (from step 3) to create Recipe -> List of Inputs
20. Create the following Operator cards: List Contains Predicate (listContains_p), Item Raw Item Equals (rawItemEquals), Logical Not (!Not)
21. Apply listContains_p on the left, and List of Current Output Items (from step 17) on the right, creating currentOrdersContainsP
22. Pipe rawItemEquals to currentOrdersContainsP, creating listCompare_step2
23. Pipe Recipe -> List of Inputs to listContains_p to get Recipe -> Apply to ListP
24. Flip Recipe -> Apply to ListP to get listCompare_step1
25. Apply listCompare_step1 on the left, and listCompare_step2 on the right, to get Recipe -> Are inputs being crafted?
26. Pipe Recipe -> Are inputs being crafted? to !Not (from step 20) to create Recipe -> Am I allowed to craft this?
27. Task 3: Now we have a card that takes a recipe, and tells us whether it's a valid craft. That's the bulk of the questionable logic. But it's being performed alongside another check, which I will include here for propriety. We need to check how much of the item exists in the network, and only craft it if it's below a certain threshold. Every item being continually crafted has this logic; assume ItemX in these names is the name of the item being crafted.
28. Create Recipe and Item cards for ItemX, Recipe-ItemX and Item-ItemX
29. Apply with getCountInNetwork (from Step 2) on the left, and Item-ItemX on the right to get countOfItemXInNetwork
30. Create an integer card with the desired threshold; in my case, I use a number of proxies to set different thresholds. My highest threshold is 8192 currently. Call this minItemX
31. Use the > operator in the logic programmer with the setup minItemX > countOfItemXInNetwork, to get needItemX
32. Apply with Recipe -> Am I allowed to craft this? on the left, and Recipe-ItemX on the right to create canCraftItemX
33. Use the && operator in the logic programmer with needItemX and canCraftItemX to create the card shouldCraftItemX.
34. Create a blank recipe card; this card is shared between all recipes, called _BlankRecipe
35. Use the choice operator in the logic programmer with the setup: shouldCraftItemX -> Recipe-ItemX, _BlankRecipe. Name it craftItemXToThreshold
36. Repeat steps 28 to 35 in order to create craftItemXToThreshold for every itemX you want to craft. I currently have 21 of these cards.
37. Put each craftItemXToThreshold card in a crafting writer in the Craft Recipe slot, and set the crafting writers to operate every 100 ticks.
38. Ensure these crafting writers are connected to the Sophisticated Storage solution mentioned in the problem description, and the Crafting Interfaces on the same network. I have about 30 or 40 Crafting Interfaces; some of them are scattered so I don't know the exact count.
39. Task 4: Another card I'm using to try and circumvent #112 is much less involved, and there are only 4 of them active, but I thought I'd include them here. This one is probably optional.
40. Pipe rawItemEquals to currentOrdersContainsP to get Item -> Is this being made?
41. Pipe Item -> Is this being made? to !Not to get Item -> Can I transport this?
42. Apply ItemX to Item -> Can I transport this? to get canTransportItemX
43. Make a choice card: canTransportItemX -> Item-ItemX, _BlankItem called transportItemX
44. Put these on importers and exporters if you're moving around an item that might be continually crafted. I currently have 4 of these on exporters on Network 1.

I've tried my best to reduce human/input error in this report, and to keep it as brief as possible. But I'm happy to clarify anything that might be unclear or erroneously reported.

Versions:

• This mod: 1.3.2
• CyclopsCore: 1.27.1-835 beta
• Common Capabilities: 2.10.1
• IntegratedDynamics: 1.28.1
• IntegratedTerminals: 1.6.17-617 beta
• IntegratedTunnels: 1.9.0
• Minecraft: 1.21.1
• NeoForge: 21.1.211
• Sophisticated Storage: 1.5.13 (for posterity)
• Modpack: All the Mods 10-4.14

Profiler output:

Here are server and client profiles of my server; the logic in these profiles is different than when I reported it in #156. This involves me simply running my server while I document this issue. At some point, I think roughly 10 minutes into the profile, I disconnect my variable stores and crafting writers containing the mentioned logic from my network. Any further performance issues are said to result from the fact that I am using Sophisticated Storage, mentioned here.

Thanks for reporting!

Note to self: have a look at this aspect's implementation:

In the meantime, I'm going to start switching my storage to a Colossal Chests solution to isolate the issue here.
Do you know of any other solutions that implement your slotless item API or in general any other mods that are known to function well as a storage solution for Integrated Dynamics? If I just put an interface directly on each Sophisticated Storage barrel would that work better?

I'm not keeping track of other mods implementing this API, so I don't really know I'm afraid.

(FYI, NeoForge for 1.21.10 contains a similar API, so from that MC version forward all mods should become more efficient in that regard)