Per player quest files?

Question

Per player quest files?

Sir-Will opened this issue 7 years ago · 5 comments

Sir-Will commented 7 years ago

hey,

is there any reason why everything is stored in one file instead of having one file per player?
If the file gets large enough it always causes lag spikes when it gets saved or sent to players.

[05:40:02] [Thread Locksmith Watchdog/INFO]: [DedicatedServer]: -#-#-#- START THREAD LOCKSMITH -#-#-#-
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]: --- Thread 20 ---
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]: Name: Server thread
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]: Status: RUNNABLE
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]: Stacktrace:
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - java.lang.String.intern(Native Method)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - java.lang.Class.searchFields(Class.java:2956)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - java.lang.Class.getDeclaredField(Class.java:2068)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraftforge.fml.relauncher.ReflectionHelper.findField(ReflectionHelper.java:108)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraftforge.fml.relauncher.ReflectionHelper.getPrivateValue(ReflectionHelper.java:140)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraftforge.fml.common.ObfuscationReflectionHelper.getPrivateValue(ObfuscationReflectionHelper.java:67)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.NBTConverter.getTagList(NBTConverter.java:273)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.NBTConverter.NBTtoJSON_Base(NBTConverter.java:52)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.NBTConverter.NBTtoJSON_Compound(NBTConverter.java:118)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.NBTConverter.NBTtoJSON_Base(NBTConverter.java:45)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.NBTConverter.NBTtoJSON_Base(NBTConverter.java:56)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.NBTConverter.NBTtoJSON_Compound(NBTConverter.java:118)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.NBTConverter.NBTtoJSON_Base(NBTConverter.java:45)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.NBTConverter.NBTtoJSON_Base(NBTConverter.java:56)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.NBTConverter.NBTtoJSON_Compound(NBTConverter.java:118)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.NBTConverter.NBTtoJSON_Base(NBTConverter.java:45)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.NBTConverter.NBTtoJSON_Base(NBTConverter.java:56)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.NBTConverter.NBTtoJSON_Compound(NBTConverter.java:118)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.handlers.EventHandler.onWorldSave(EventHandler.java:198)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraftforge.fml.common.eventhandler.ASMEventHandler_452_EventHandler_onWorldSave_Save.invoke(.dynamic)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraftforge.fml.common.eventhandler.ASMEventHandler.invoke(ASMEventHandler.java:90)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraftforge.fml.common.eventhandler.EventBus.post(EventBus.java:652)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.world.WorldServer.handler$onSaveAllChunks$zlm000(WorldServer.java:2377)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.world.WorldServer.func_73044_a(WorldServer.java:1010)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.command.server.CommandSaveAll.func_184881_a(SourceFile:41)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.command.CommandHandler.func_175786_a(CommandHandler.java:119)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.command.ServerCommandManager.func_71556_a(SourceFile:1083)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.server.dedicated.DedicatedServer.func_71333_ah(DedicatedServer.java:432)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.server.dedicated.DedicatedServer.func_71190_q(DedicatedServer.java:397)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:666)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.server.MinecraftServer.run(MinecraftServer.java:524)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - java.lang.Thread.run(Thread.java:748)
[05:40:02] [Thread Locksmith Watchdog/INFO]: [DedicatedServer]: -#-#-#- END THREAD LOCKSMITH -#-#-#-

Funwayguy · Answer 1 · 2018-05-18T04:38:05.000Z

Having one per-player would be an absolute mess and still wouldn't fix anything. It would actually make things much worse. Your report here is showing the database of all quests saving to file when the world saves all its chunks (converting NBT to JSON). Assuming you even could split it into pieces (you can't because it isn't just player progression being saved), you'd end up having to search through the database, split the data per player (including ones that have since logged off) and then start/stop new file output streams just to accomplish this. Stitching it all back together when the server loads up would be even worse.

Similar kind of problem occurs with syncing the database with players. Client side can't just work with fragmented pieces of a database. Players and quests also need to be aware of shared progression in parties on the client side too for display purposes. BQ only performs this mass sync either upon login (and only to that particular player) or if a major database edit has been made and from then on its cached locally for future use. On all other occasions it can sync single quests just fine without hammering the network or database.

I have plans on how to reduce the amount of data being saved or loaded off disk plus some other optimisations with indexed searching and cached sync data but that does NOT include fragmenting the database for no reason nor does it change the previously mentioned sync requirements. These changes are still a while off and require huge database/networking rewrites so there is no ETA for it.

TLDR: Unless you want to cripple your server trying to split up data into numerous unnecessary pieces that will then have to be stitched back together anyway, it's not happening.

NOTE: If this is such a huge problem that'd you sacrifice backup safety on your server to fix it, you can disable the server's automatic saving of the world to disk using /save-off and this will also stop BQ from saving its data at the same time. Still won't stop BQ syncing joining players though.

Sir-Will · Answer 2 · 2018-05-18T07:28:28.000Z

Why not save it as NBT instead of converting it to json?

Can this below not be done async?

[09:00:03] [Thread Locksmith Watchdog/INFO]: [DedicatedServer]: -#-#-#- START THREAD LOCKSMITH -#-#-#-
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]: --- Thread 20 ---
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]: Name: Server thread
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]: Status: RUNNABLE
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]: Stacktrace:
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:691)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:579)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:271)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - java.io.Writer.write(Writer.java:157)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.stream.JsonWriter.newline(JsonWriter.java:603)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.stream.JsonWriter.beforeName(JsonWriter.java:618)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.stream.JsonWriter.writeDeferredName(JsonWriter.java:401)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.stream.JsonWriter.value(JsonWriter.java:417)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.internal.bind.TypeAdapters$29.write(TypeAdapters.java:762)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.internal.bind.TypeAdapters$29.write(TypeAdapters.java:776)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.internal.bind.TypeAdapters$29.write(TypeAdapters.java:776)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.internal.bind.TypeAdapters$29.write(TypeAdapters.java:776)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.internal.bind.TypeAdapters$29.write(TypeAdapters.java:776)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.internal.bind.TypeAdapters$29.write(TypeAdapters.java:776)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.internal.bind.TypeAdapters$29.write(TypeAdapters.java:776)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.internal.bind.TypeAdapters$29.write(TypeAdapters.java:776)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.internal.bind.TypeAdapters$29.write(TypeAdapters.java:714)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.internal.Streams.write(Streams.java:72)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.Gson.toJson(Gson.java:745)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - com.google.gson.Gson.toJson(Gson.java:703)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.api.utils.JsonHelper.WriteToFile(JsonHelper.java:211)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - betterquesting.handlers.EventHandler.onWorldSave(EventHandler.java:198)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraftforge.fml.common.eventhandler.ASMEventHandler_452_EventHandler_onWorldSave_Save.invoke(.dynamic)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraftforge.fml.common.eventhandler.ASMEventHandler.invoke(ASMEventHandler.java:90)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraftforge.fml.common.eventhandler.EventBus.post(EventBus.java:652)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.world.WorldServer.handler$onSaveAllChunks$zlm000(WorldServer.java:2377)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.world.WorldServer.func_73044_a(WorldServer.java:1010)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.command.server.CommandSaveAll.func_184881_a(SourceFile:41)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.command.CommandHandler.func_175786_a(CommandHandler.java:119)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.command.ServerCommandManager.func_71556_a(SourceFile:1083)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.server.dedicated.DedicatedServer.func_71333_ah(DedicatedServer.java:432)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.server.dedicated.DedicatedServer.func_71190_q(DedicatedServer.java:397)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:666)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - net.minecraft.server.MinecraftServer.run(MinecraftServer.java:524)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [threadlock]:  - java.lang.Thread.run(Thread.java:748)
[09:00:03] [Thread Locksmith Watchdog/INFO]: [DedicatedServer]: -#-#-#- END THREAD LOCKSMITH -#-#-#-

Funwayguy · Answer 3 · 2018-05-18T13:35:10.000Z

It's in JSON so admins and pack devs can make edits using external text editing programs if necessary (many do). Even if someone was familiar with how to manually edit a compressed NBT file, it still wouldn't be suitable for bulk edits. This was also a requested feature in opposition to HQM's locked down files.

The conversion and file output itself, after it has been collated from the database, could be done asynchronously but it would still need the aforementioned changes to be completed first.

Sir-Will · Answer 4 · 2018-06-26T22:13:16.000Z

What about a option where admins can configure if they want to use json or nbt for saving and or a command which allows to export/import the nbt to and from json?

Funwayguy · Answer 5 · 2018-07-01T07:35:00.000Z

That would still do absolutely nothing for player database syncing. You're missing the bulk of the issue here and that is any considerably large database will take time to move around, conversion or not. The syncing doesn't even do any JSON conversion at all. The only time JSON is ever actually used in the mod (since BQ3) is when saving to disk. All other times it is already handled in native NBT.

Look, I know you want this solved but there is really no easy fix to this. The issue involves managing huge amounts of data across multi-threaded processes, caching changes and somehow networking this across potentially hundreds of players all without crippling servers, backwards compatibility or editor flexibility.

As I stated before, I have solutions in mind for this but there are a lot of things that still need to be upgraded to deal with it. The first of which I've already completed and that was making the databases much faster and thread safe (build 257).

Share to