RM has two mixins which are active during rendering which effectively force all chunks to be fully updated before every frame (rendering isn't real time, so frame time spikes are not an issue):
a) force all chunks to load by repeatedly updating the visibility graph and uploading all chunks until there are no more changes (Mixin_ForceChunkLoading)
b) block the main thread until background chunk updates are done and uploaded, used in aforementioned loop (Mixin_BlockOnChunkRebuilds)

From a quick glance over Canvas' renderer it appears that it completely replaces setupTerrain, making those mixins not apply.
Would be great if Canvas could provide functionality equivalent to these mixins, either by checking if WorldRenderer.replayModRender_hook is non-null and then force-loading all chunks or by providing an API with which RM can en-/disable that functionality.

This applies to all MC versions (dunno if you are still updating Canvas for 1.16) and those two mixins are unchanged for 1.17. I can make available the beta version of RM for 1.17 to whoever wishes to implement this if needed for testing.

Have you had time to take a stab at this yet?
It's not particularly urgent in general, just nailing down the exact API would be nice, so I can implement it in RM and other mods. Kinda blocked on this (though not exclusively) for the RM 1.17 update cause we're dropping Optifine support, so at least Iris should work with it but currently that's a bit tricky to install because it bundles Sodium which has a very similar compatibility issue as this one (just worse cause it's actually unusable) which I'd like to resolve by implementing this API in Sodium, hence why I'd need the API nailed down.

I’ll try to have a look this weekend (today or tomorrow). Thank you for the reminder.

The new "flawless frames" feature is now implemented in Canvas, starting with build 1987.

de01258

Thank you for the thoughtful post. I'll be happy to make the necessary changes in the 1.17 version. (1.16 was a work-in-progress version and will not be maintained.)

Seems like three changes are needed for Canvas:

1. Force terrain iteration to run on the main thread. (Already a configuration option - easy to do.)

2. During iteration, build regions (chunk sections) that need it as they are encountered instead of adding them to the build queue, so that iteration can progress to completion every frame. This is safe because we'll be running on the main thread and don't have to worry about frame rate. The tricky part here may be forcing/waiting for chunks to be loaded.

3. After iteration, disable throttling of region uploads, ensuring that all uploads complete before terrain is rendered.

ReplayMod will need some way to control when Canvas operates in this mode. There are at least three ways to do this:

1. Canvas exposes some kind of toggle mechanism. I would define in the FREX API so that any other FREX-compliant renderer can implement the same hook. FREX can be used as a soft dependency and this would also be defined as an optional feature so that you can test if the feature is available.

2. Replay mod exposes a way to accept a boolean consumer, which Replay would use to enable or disable this mode. Almost the same as first option, but defined by the Replay mod. Canvas would have to use a loader mechanism or reflection hacks to avoid a hard dependency.

3. Replay mod exposes a boolean supplier, which Canvas polls before each frame to know if the mode is active. Otherwise the same as the second option.

Do you have a preference or a different way in mind?

During iteration, build regions (chunk sections) that need it as they are encountered instead of adding them to the build queue

Ideally it would still add them to the queue, such that it can take advantage of threading, and then wait for the queue to be fully built (the Vanilla implementation and the provisional Sodium implementation work this way). Whether this is feasible or too complex does ofc depend on the queuing mechanism, which I've got no clue about, just mentioning it in case it is (even busy waiting may be faster in the end than doing everything on the main thread as long as there's more than two threads). Speed isn't at all critical, but it would of course be nice to have.

Do you have a preference or a different way in mind?

I actually strongly tend towards 2 with a loader mechanism (I assume you mean entrypoints?) because
a) It does not have to be specific to ReplayMod: I could conceive of other mods potentially wanting to use such a feature as well.
b) It does not have to be specific to Canvas: As alluded above, a similar compatibility issue exists with Sodium as well (though only with the second Mixin), and that could then use the exact same API (the current proposal adds an API to Sodium, which I really don't like very much) while FREX would be too heavy for it.

I'm thinking something along the lines of RM registering an entrypoint of type Consumer<Consumer<Boolean>> and Canvas/Sodium simply invoking that entrypoint once at boot to pass its setter:

Consumer<Boolean> setter = enabled -> { this.forceChunkLoading = enabled; };
FabricLoader.getInstance()
    .getEntrypoints("frex:force_chunk_loading", Consumer.class)
    .forEach(it -> it.accept(setter));

(better names are welcome. dunno if there's a convention for naming entrypoints, I just add the frex: prefix to ensure that it's unique, could also do replaymod: or canvas:, doesn't really matter)

Ideally it would still add them to the queue, such that it can take advantage of threading

Probably worthwhile. Would have to loop and run iteration/region builds multiple times until it is fully progressed. I happily forgot how vanilla iteration works long ago, but Canvas iteration is what triggers regions to be read from chunk sections and analyzed/built for rendering and iteration can't progress past unbuilt regions. It's generally faster than vanilla and iteration normally runs off thread so this scheme helps maintain frame rate while the world is loading.

I also like the idea of an entry point in the FREX namespace. FREX is meant to be a community standard but doing it this way means no actual dependency on a FREX library is needed.

I'll probably name it something like frex:force_frame_quality to convey that the intention is for a complete and pristine frame. If a renderer is playing loose with some visual elements to achieve interactive frame rates it should try to disable those if it will give a noticeable boost to quality. I can't think of any off hand, but Canvas and some shader pipelines probably do or will have those situations.

I can make a little test mod for now to get this done. I'll respond here again when I have something you can test with. Will probably take a week or so.

I forgot to ask earlier: how are you currently handling chunks that aren't yet loaded from disk or available from the server? I could be wrong, but if I remember correctly that isn't fully handled within WorldRenderer.

Are you somehow force loading all the chunks within render distance elsewhere before world rendering happens, or is there something I need to do for it?

Sounds great.

how are you currently handling chunks that aren't yet loaded from disk or available from the server?

When the ReplayMod renders a camera path, it does so within a replay (basically just a packet dump created during recording which it can then re-play by pretending to be a more or less normal server). The mod has full control over how the packets are read from the replay file and injected into Vanilla's netty stack, so it just switches that to the main thread during rendering, meaning that packets can always be fully read and handled before it starts rendering the next frame. The server is generally pretty quick with sending you the chunks during recording, so that's generally not an issue.

Are you somehow force loading all the chunks within render distance elsewhere before world rendering happens, or is there something I need to do for it?

Nope, it only concerns itself with chunks which have already been received from the server (at the specific time in the replay), anything more is out of scope. Nothing you need to worry about.

(slight tangent)
I did actually write a separate, independent mod named Bobby (horrible name but too late now) to be able to load more than the server view distance by caching chunks to disk on the client and then loading from the cache on demand around the camera.
Bobby was originally written specifically for the ReplayMod and as such did all its work on the main thread. After some time however I began using it just by itself in regular multiplayer and started optimizing it to for that use case: do things in the background, limit chunk loading such that there are no spikes in frame time, etc.
That did unfortunately make it less ideal for the ReplayMod because chunks were now only being loaded from the cache very slowly during rendering and I've had to tell people to just use the old versions. So I guess I've got a proper solution for that as well now, it just needs to provide this API.