To get music discs you need to open chests that can be obtained by defeating monsters.
The chance that there is a music disc in the chest is 25%.
To open the chest you need appropriate melodyte stone.
Melodyte Ore generates in the overworld below level -16. You have to craft melodyte with a specific mineral to obtain needed melodyte (e.g. Melodyte + Iron Ingot = Iron Melodyte).
To increase your chances of dropping a music disc from a chest, you'll need a Symphonyte stone. This stone can be found in any regular chest. When Symphonyte is crafted with any chest, it transforms into an enchanted chest that has a 100% chance of dropping a music disc.
With the release of update 1.0.7 / 2.0.0, it is now possible to obtain Music Discs Cases. These cases are used to store music discs. The cases drop from Diamond and Netherite chests, with a 5% chance of dropping.