Introduction - Why?

This seems like a weird title but I can not describe it better.
In our current situation we use a colony.dat and colonyX.dat file to save our colony data to disk. This is by far from optimal. As can be seen in Issue #2297.

On 2/9/2018 I discussed this with @Raycoms, before the issue, was created, and we came to the conclusion that the current system is just not good enough for several reasons into which I go a little bit later. Since my hiatus this last weeks is coming to an end, I have been looking into the bigger issues at hand here. One of them was, and is, our storage mechanism. So I came up with a better mechanic based on Minecraft and Forge core mechanics that allows us to make the current system more stable and less dependent on our own files.

Current System

Features:

Our current system has several features that we are proud of and that are very usefull:

• Independent of any other System, easy to port and maintain.
• Colony and global data are stored seperately.
• Backups are created by us automatically.

Bugs:

Even though our system works very well there are some issues with it (resetting and losing colonies, for example):

• When the world is stopped without saving (Closing the game process, game crash etc) we still save, causing a desync between the data in the world and the data in our .dat files.
• We update them every tick, causing a lot of performance issues since IO takes a lot of time.
• It is not easy to load a back up of the data into the world. We do not compare the world state with the stored data.

In short these bugs would seem to be not all that hard to fix, or not of a significant value. But they are. It is very inconveniant for a Player to have to copy around files to load backups when something goes wrong. And it is easy to make something go wrong.

New System.

Core Design:

The new system builds on the already designed and working square colony chunk claiming mechanics we have. They are stored in something called an ICapability of a Chunk, a mechanic provided by forge that allows us to attach data to several things, ranging from worlds and chunks to entities and ItemStacks. The current problem with that is that we can not claim loaded chunks when the townhall is placed (we could but the IO of unloaded chunks and the generation of new chunks would cause a major lagspike) so we write these to disk with the other colony data, slowing the ticks down even further due to IO.
To prevent this we should switch to ICapabilities, in combination with TileEntities, entirely:

The new Capabilities - where and how:

• DimensionColoniesCapability: Functions as the ColonyManager now. Behaves as the central point for ticking and handling of the colonies in the world, read dimension, that it is attached to.
• DimensionClaimingCapability: Stores the chunks that still need to be claimed when they are generated or loaded from disk. Events allow us to track which chunks gets loaded or generated and we can hook our data into that then.
• ChunkClaimingCapability: Exists already currently. Can be reused.

TileEntities - Who now?

As of today we are already using TileEntities for buildings etc. We should expand on this: use them to store building data etc.

Why these changes:

Well the most obvious thing: It is a lot less error prone. These mechanics are used by hundreds of mods and minecraft itself to store data. Saving and loading from disk is then done automatically. Synchronisation is a lot simpler then to. We do not have a shared colonymanager which causes issues on a singleplayer world. We do not need to care about when we save the data, when to world is saved our data is saved. Which prevents desyncs from happening.

The issue of Backups:

Well you might be thinking now: Nice but what about the backups, using backups of small files is much simpler then backups of the world. How would we handle them?
Well the idea is to combine several elements together. First and foremost: We can still write all the data ourselfs to disk, The readFromNBT(NBTTagCompound) and NBTTagCompound writeToNBT() method both exist on the ICapability and the TileEntity. Using some simple threading this would not have to cause a lot of lag if implemented properly. Second we can then use a command to simply call a read of the stored NBT data on all elements and try to load their backups. The last problem is that a world breaking crash still needs to be repairable. We would use a flag in either the config file or the startup parameters to load a backup directly in the first onPreWorldTick event. Causing it to load all data from the backup. This might cause a major amount of lag, since a lot of chunks need to be loaded, but it would work.

Issue creation.

This issue was created to gain the opinion of the team and the community on these changes and to track its implementation progress. Feel free to ask questions in the comments.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

I only have a couple of minutes before work, but here's some quick thoughts.

Back when I was writing my own take on the MineColonies idea, I started with by using a tileentity instead of a ColonyManager singleton. You will lose control over WHEN things are processed, such as when data is loaded. I remember having problems with entities not being available when my tileentity loaded. Locating the tileentity by x,y,z coordinates is a lot harder than locating a colony by calling a static method on ColonyManager, and there's no guarantee your colony tileentity is loaded. You might need to have a hybrid approach in which the data is stored, saved, and loaded in the tileentity, but the the singleton still provides central access to the data. Even if the hybrid approach isn't kept over the long run, it would provide a migration path to using a tileentity approach.

I was in the process of researching how to use Capabilities in addition to my tileentity when I discovered MineColonies, but I hadn't actually added them to my project.

@mkienenb exactly, that's why we thought about a hybrid approach.
We store the colony somewhere where it's always loaded when the world is loaded (In the dimension itself).

We will only write the buildings in the tileentities which don't need to be doing anything anway if the building is not loaded.

Moved to capabilities already