I have automatic recording ON and joining to the server. Few seconds passes, I press Stop Recording button in the pause menu, so it behaves like it stopped (button becomes Start Recording, no logo about recording), as should be. Then I continue to play, for around few hours (travelling, using nether portals etc).

After that I quit the server and it starts to save recording file, but it takes a lot of time (> 1 minute compared to ~15 seconds as usual). Also with time it fills all heap space and game starts lagging on GC. I looked for files and found this:
https://i.imgur.com/mv1yvZs.png

Seems it just didn't stop recording and recorded everything for hours?

Info:

• Windows 10 x64
• Java 14 (OpenJDK)
• Replay Mod 2.3.0 b1 (1.15.2 / Fabric)

P.S. It does not just continue to record, but probably save some data into file after stop. If I record from joining the server, then stop recording and to some actions, then go to replay, the end will be in position I stopped recording.

Seems it just didn't stop recording and recorded everything for hours?

This is expected behaviour.
Due to how replays work, there's no actual way to to start/stop the recording. All the mod can do is condense everything happening prior to the "start" of the recording and throw out everything after its end.
So, if one wants the option to start recording at any point, then one needs to have already recorded everything prior to that point.

Also with time it fills all heap space and game starts lagging on GC.

Hm, this may either be because the condensing is leaking memory or just because it legitimately needs it. Did you frequently switch worlds during the recording or were extended sections of the replay in the same world? (former would be a strong indication of it being leakage, as it should be able to drop most stuff on world change)
If you haven't yet deleted the raw folder, you should be able to find the raw (uncut) replay in there. If you could send it to me, I'd be able to try myself and see what actually takes up all the space.

After that I quit the server and it starts to save recording file, but it takes a lot of time (> 1 minute compared to ~15 seconds as usual).

This could be improved. Atm it unconditionally condenses the whole replay and slices in a copy of the current condensed state whenever you press start. Given condensing happens after the recording is over, it can know when you never pressed start again after a stop and can just end it early right there.

"So, if one wants the option to start recording at any point, then one needs to have already recorded everything prior to that point."

But it records data NOT prior, but after. I join the server, recording starts automatically. I stop it. And just play for a while, without recording. But it still records something into file.

But it records data NOT prior, but after.

It records the data prior to you potentially pressing Start again in the future.

Hm, this may either be because the condensing is leaking memory or just because it legitimately needs it. Did you frequently switch worlds during the recording or were extended sections of the replay in the same world? (former would be a strong indication of it being leakage, as it should be able to drop most stuff on world change)
If you haven't yet deleted the raw folder, you should be able to find the raw (uncut) replay in there. If you could send it to me, I'd be able to try myself and see what actually takes up all the space.

I didn't switch between worlds during recording, the recording itself from the time I join the server was about few seconds (up to 10). I switched between worlds a lot after recording Stopped (by button). Usually, even 5 minutes of recording with world switch takes less memory usage, and saves much faster.

I removed a raw file with cache already, but can record everything again if needed.

But can't we just make something like a snapshot of everything surrounding us before starting?

No, there is no generic way to make a snapshot on the client. That is, it's entirely infeasible with third-party mods and would require an unreasonable amount of code (and maintenance) for vanilla, basically handling every thing manually, every entity type, every tileentity type, lots of miscellaneous things (weather, time, scoreboards, teams, etc.).

The same way we do it when joining the server?

We don't. The server does and sends it to us. And it has a generic way to do so (cause that's like half its job), which is why the whole thing is feasible in the first place.

I didn't switch between worlds during recording, the recording itself from the time I join the server was about few seconds (up to 10).

As explained above, the recording is actually the whole X hours. Only while it says "Saving replay" are the Start/Stop/Pause/Resume actions applied.

I switched between worlds a lot after recording Stopped (by button).

Then there may be indeed some kind of leak.

I removed a raw file with cache already, but can record everything again if needed.

No need to hunt for it. If there's indeed some kind of leak, there's a good chance you'll eventually run into another recording which has the same issue (doesn't matter whether you used Start/Stop/Pause/Resume), just send me that one then.

Here is the folder with temp files as you asked.
Here I joined the server, did actions, stopped recording (button), then did more actions including chunk loads, switching between worlds via nether portals. The ".tmcpr" file was about 13 MB when I pressed stop button.

P.S. No other folders (like raw) were in recordings.

2020_04_24_13_54_14.mcpr.tmp.zip