With the block entity changes in 1.17, it seems that the complex tracking that is done at the moment might not be necessary anymore, as BlockEntity#markRemoved seems to be called consistently for chunk unloads now. To be confirmed though.