Both wooden and metal scaffolding are affected, also in their slab form, but noticably the scaffold covered ladder DOES retain the speed boost.
So far the speed boost seems to only be retained on the metal ladder (stand alone [huehue] and covered), conveyors of any kind and connectors/relays/etc. of any kind.

So the following are also NOT included in the mining speed boost list (and tbh, I don't remember which if any of these ever were included):
Pipes, Fluid Routers, Item Routers, Fluid Outlets, Fluid Pumps, Metal Barrels, Radiator blocks, Engineering blocks (Light, Heavy & Redstone) and Sheetmetal Chutes, Sheetmetal blocks, IE Fences

also not included: Hoppers, Cauldrons & Pistons (the latter already mines pretty fast w/o tool, so eh)

not saying any of them should be included, I just thought I' test through all the blocks used in multiblock structures (that aren't straight up bricks) and then you got the overview ^^