Replay Mod (Fabric & Forge)

Replay Mod (Fabric & Forge)

Mods
1M Downloads

Heap overflow when splitting large files

ByThePowerOfScience opened this issue ยท 0 comments

commented

Context:

I have a replay that is eight hours long. I tried to fast forward halfway, and my client ran out of the 8GB of heap memory I have allocated. I'll be analyzing the heap dump and making another issue once I identify the problem with that.

Anyway, I relaunched Minecraft and tried splitting the replay into sections. Soon after, it ran out of heap memory again.

Problem:

// Don't be sad, have a hug! <3

Time: 2024-05-11 00:09:15
Description: Running marker processor

java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3481) ~[?:?] {re:mixin}
	at java.util.ArrayList.grow(ArrayList.java:237) ~[?:?] {re:mixin}
	at java.util.ArrayList.grow(ArrayList.java:244) ~[?:?] {re:mixin}
	at java.util.ArrayList.add(ArrayList.java:486) ~[?:?] {re:mixin}
	at java.util.ArrayList$ListItr.add(ArrayList.java:1068) ~[?:?] {}
	at com.replaymod.replaystudio.stream.IteratorStream.insert(IteratorStream.java:54) ~[replaymod-1.20.1-2.6.15_mapped_srg_1.20.1.jar%231153!/:?] {re:classloading}
	at com.replaymod.replaystudio.filter.SquashFilter.add(SquashFilter.java:819) ~[replaymod-1.20.1-2.6.15_mapped_srg_1.20.1.jar%231153!/:?] {re:classloading}
	at com.replaymod.replaystudio.filter.SquashFilter.onEnd(SquashFilter.java:773) ~[replaymod-1.20.1-2.6.15_mapped_srg_1.20.1.jar%231153!/:?] {re:classloading}
	at com.replaymod.editor.gui.MarkerProcessor.apply(MarkerProcessor.java:168) ~[replaymod-1.20.1-2.6.15_mapped_srg_1.20.1.jar%231153!/:?] {re:classloading}
	at com.replaymod.editor.gui.GuiEditReplay.lambda$apply$12(GuiEditReplay.java:164) ~[replaymod-1.20.1-2.6.15_mapped_srg_1.20.1.jar%231153!/:?] {re:classloading}
	at com.replaymod.editor.gui.GuiEditReplay$$Lambda$52301/0x00000008063c9278.run(Unknown Source) ~[?:?] {}
	at java.lang.Thread.run(Thread.java:833) ~[?:?] {re:mixin}

ReplayMod/src/main/java/com/replaymod/editor/gui/MarkerProcessor.java

Lines 162 to 175 in 706d46f

try (ReplayOutputStream replayOutputStream = outputReplayFile.writePacketData()) {
if (cutFilter != null) {
cutFilter.release();
cutFilter = squashFilter.copy();
} else if (splitCounter > 0) {
List<PacketData> packets = new ArrayList<>();
squashFilter.copy().onEnd(
new IteratorStream(packets.listIterator(), (StreamFilter) null),
timeOffset
);
for (PacketData packet : packets) {
replayOutputStream.write(0, packet.getPacket());
}
}

Essentially, it seems that the entire replay gets buffered into a list before being committed to the file. This is fine for smaller replays, but for larger ones it ends up overflowing the allocated heap memory.

Solution:

Directly append each packet onto the file without buffering them first.

I can make the PR for this, but could you please help me understand what this section of code is doing so I don't break anything when fixing it?