I like having the left sidebar as favorites, but 90% of time I don't have any favorites.

A way to remove everything but the page switch button when the panel is empty would be very much appreciated.

I didn't initially comment on this when I tagged it. I believe it'd definitely be a good idea to include an option like this, but I've been considering the most intuitive functionality for a long time.

• The simplest functionality is just the name on the tin, if your currently active sidebar type has no content, just do not render anything, no background, no buttons, no page number. This is fine in a lot of cases but would be kinda tricky in cases where the other sidebar types have information and you'd be unable to switch to them. However, preventing this might seem an anti-feature, as anyone configuring their sidebar would be conscious of their settings.
• The more complex functionality is to remove everything other than the sidebar switch button. This kinda also presents an anti-feature where users are unable to actually hide the thing like they want. Providing this as 2 different options would lead to a lot of confusion for users, so I'm not sure that is a good call either.
• A hybrid solution? Slightly more logic involved, but hide everything, unless there are other sidebars to cycle to that have stacks, then show them. This would present interesting issues, especially with the searched sidebar, as the "searched stacks" of non visible types are not calculated unless visible for performance reasons.

Also worth consideration is subpanels. If the primary content of a sidebar is empty, but the subpanels are populated, should it still hide?

Here are my current conclusions, I feel like in a majority of situations, if a user is using subpanels, they will always be populated, so in combination with hiding settings, subpanels should just be ignored, if a user doesn't want hiding they'd just not enable it. I think the hybrid solution might be the most intuitive, presenting the button only if alternative populated sidebar types exist to display.

As an unrelated comment, there currently exists the option to simply remove the header for a sidebar, and I personally use this for the sidebar where I keep my favorites, as I don't value the page information. I don't see this as a proper solution to sidebar hiding, but I thought I'd mention it.

there currently exists the option to simply remove the header for a sidebar

Oh I totally must have missed this! Where's the option for it? Is it per sidebar?

Yeah it's a setting you can set per each of the 4 sidebars, along with stuff like size, background, subpanels, etc, I believe it's called "Header"

What about hidding arrows and the number of pages when there's only one? And maybe center the favorites icon? Reduced visual noise when there's no use to displaying buttons that can't do much unless you move to the next page.

Maybe something similar could be expanded to other sidebars?