People keep confusing them for the actual replay file and having them in a separate folder also makes manually cleaning up all the caches (e.g. to recover disk space) easier.
Maybe even put them in a system-dependent folder (e.g. $HOME/.cache/ on linux).
Name of the cache folder should then depend on the content of the replay, not its name cause there's no guarantee for that to be unique across folders (e.g. instances).