Render audio separately
TheChocolateOre opened this issue · 4 comments
I know, I know the thousand-times-asked feature. But, hear me out.
Let’s start from the best case scenario, what we all want.
Best Case:
Render video with the audio embedded in. The one-button-press solution.
Unfortunately, as we all know this is not possible or extremely hard, not worth pursuing, and this is where all of those “Add audio”, “Include sound”, “My video doesn’t have audio”, “plz audio”, … requests end.
But, why? Yes, we can’t have that, but why end there? What’s the next best thing we can have? Well, let’s first overview what we manually did and are doing all those years to actually get sound to our replays. Recording the loopback, by playing each replay one by one manually. So, we get the next better case:
Render audio separately by recording the loopback:
Now, there are a million ways and tweaks to do that, but just like we press a button and we render video, we could have a button that renders audio by loading each replay, playing it in real time, and while doing so recording the loopback on its way. Finally, exporting the sound file. Then, it’s the responsibility of the user to stitch it in the video.
Of course, naive me, there could be many many technical obstacles that make this difficult or even impossible, e.g. what if the user plays other audio in the background, etc…
It’s a call for discussion, on what we can do, what we can’t, what a few solutions might look like, or even maybe what's the next good case we could tackle if this one is not feasible.
I see two ways to implement audio recording:
Option A: the dirty/janky/easy Way
Just let the replay play and use ffmpeg to not only stitch the frames, but also to capture the audio.
I belive that is fairly simple as ffmpeg has audio capture capabilities on (most/all?) OS.
https://trac.ffmpeg.org/wiki/Capture/Desktop
Cons: It could happen that ffmpeg captures audio data from other process too or the computer volume is on 0% or muted and no audio is captured, also the audio quality might suffer. Also some OS dependent code is needed to generate the right ffmpeg command depending on the OS.
Option B: the clean/complicated Way
In normal Gameplay minecraft generates an audio stream and send it to the OS. If we can "just" hook into that and send it for example to a tcp socket or a tmp-file ffmpeg can use the raw audio stream to mux it into the file. I think the biggset problem would be that minecraft uses a library for audio mixing. At least the last time i investigated (some years ago) it was that way. This would be an OS independet way that should retain 100% audio quality.
I think this is the way everybody would love to see it.
I've created an addon for ReplayMod that renders audio using OpenAL loopback device: https://github.com/Apehum/replaymod-audio-render.
This addon is more like a proof of concept, so it currently doesn't have any configuration nor support versions below 1.21. It simply renders stereo audio in AAC format alongside the video render.
If anyone finds this addon useful, I'd appreciate feedback, and I'll consider improving it further.