EDIT: (for Topic Summary)
Requirements:

• Having many items stored with many(/large) NBT-Tags.
• A ME Storage Bus connected to ANY Compatible Item Container (Vanilla Chest/DSU/Barrel,...)

What to do to get the issue:

1. Open Terminal
2. Have something change the content of the attached Item Container rapidly
3. Notice that the complete content of the ME Network is resent to clients

ORIGINAL POST:
Hi,

I just experienced an issue with my ME System, that caused Applied Energistics eating up the complete network bandwidth.

Steps to reproduce:

1. Have many unique items in your ME system (In my case about 2520 types most of them Enchanted Books with different Enchantment combinations like they are dropped when fishing, and enchanted fishing rods and enchanted bows ...)
2. Then build an enderquarry with efficiency III upgrade (from ExtraUtilities) and make it input its items directly into the ME System (ME Interface) (alternativly an other block produces masses of items per tick)
   -- Note this alone doesn't create lag --
3. Now open Any ME Terminal (/Pattern Terminal /Crafting Terminal /Interface Terminal)
4. If your server has a too low upload or the client has to low download speed you will now notice that your ping will build up very fast! (In my case my download is too low with 2 Mbit/s resulting in a ping up to 250000ms meaning i was more than 4 minutes behind!)

Attached you can find 2 Ping Graphs one when the ME is closed, one when I open a Terminal
Also you can see the used network Bandwidth by mod in 2 additional Pictures (again one closed, one opened)

I'd like to see an option to limit the inventory updates of the me system to 1 update every 2 seconds when the terminal is opened

I'm using the FTB Infinity Evoled ModPack in Version 2.6.0 (MC 1.7.10)
containing AppliedEnergistcs 2 rv3-beta-6
The Forge Version irrelevant for this issue, but to be sure: 10.13.4 Build 1614
I tested this also in rv2-stable10 without any other mods by just building 64 hoppers to 64 interfaces ...
The result was equally devastating
My Performance date was captured using Opis

Greetings
Pingger

@yueh a view inside @Pingger ME Storage could probably help .. well at least i would be interested in viewing it with an NBT viewer!? can you lead @Pingger to the save file and let him upload it?

i have no idea where you store the hard drive data

also @Pingger do you have any other (more complicated) storage components besides drives with storage cells and probably barrels in your test setup?

You can find a very recent Version here: http://www.iskariot.info/2016-08-30-15-10-58/

ME System here: x:224 y:4 z:-115
BufferChest here: x:214, y:4 z:136
EnderQuarry in RFToolsDimension! use Left Teleporter at x:214 y:9 z:-113
The Quarry is disabled after a server restart, also the Resource seeds of AgriCraft need to be enabled.
Take a Dialing Device with you!!!!!

And yes it's most probably the NBT data that causes the huge overhead of data ...
Remember: FTB Infinity Evolved 2.6.0 was used in this world

I found some more Information:
It has to do with the Storage Bus! I have in both world used sotrage busses connected to Deep Storage Units. It seems that the full network inventory is resend when the content of the item container connected to the storage bus changes.
In my test world normal chests where used to simplyfy things...
in my server world I used 2 DSU to store Cobble stone and normal Stone (ID 1)

Maybe this helps somehow

OK
I just started voiding additional cobblestone and clean stone so I just get items that are stored into ME Storage Cells now the Network bandwidth is at about 120 Byte/s while the Quarry is active ... this is about a 2000th of what it was before ...

This is nothing we can actually solve or really improve as it is actually an issue with other mods being pretty stupid about what they save to NBT and how.

A simple AE itemstack is about 7 to 21 byte as packet (7 for a stacksize < 256, 9 < 65535, 13 < 4 billion)
So even 2500 items would just be around 17 to 52 kB.

It is synchronized once when a terminal is open, because otherwise we had to constantly synchronize every single terminal. So pretty much your problem would apply to everyone by default.
Afterwards it will only synchronize the changed stacks, which by default is really small. Synchonizing 100 changed stacks should still be less than 1 kB.

The problem now are items with NBT data, as this has to be also synchronized for every itemstack and some mods do stupid things like attaching a few kB or even MB to it. Which has to be send to the client. You can also pretty much expect, that these items will certainly not stack and can easily be obtained in a few dozen variants, each adding more data.

It is really something the actual mod has to solve or forge has to provide a way to just synchronize the essential part of a NBT tag (like just synchronize that the bag should glow, but not the 1MB of data attached to it). But with forge not caring about performance, I doubt we will ever see something here.

I sadly cannot really determine which items are the offending one. Mostly because the way forge handles packets makes it impossible to use the normal netty pipeline (server client communication) and for example log very large packets including information about what makes them so large.

Limiting the packets would not really improve it. It has to be synchronized at some point, so all it does is bundled everything and send it at once. So it is not 2 MB in smaller packets of about 50kB every tick but 2MB at once every 2 seconds. Maybe it saves a few bytes to accumulate the changes for cobblestone over 2 seconds, so it just sends 13byte every 2 seconds instead of multiple times over 2 seconds. But if that is already an issue...

The actual items creating such large packets, will still have to be send. So you will most likely increase the update times constantly. But where is the point to synchronize it ever 5 minutes?
Also the next mod saving 4 MB instead of 2 MB as NBT is just around the corner or some player will just drop 20 instead of 10 into the network.

Note I am not sure how often AE synchronizes the itemlist. It might be every tick or maybe less. But only when Container.detectAndSendChanges() is called.

good ol storage monitor :) combined with unconventional containers always a source for debug fun :)

keep in mind to prevent an known bug to NEVER lock DSUs as this severly f***** up ME Storage in the past :)

thx. But there were no storage monitors placed and the DSU's weren't locked

Storage Bus i meant, my fault