Music, even when instrumental, can distract me if I hear a song I know or a song I just really don't like. If I know the song, I'll get zoned out and follow the song. Alternately, if I dislike the song, I'll get zoned out because I'll feel the need to skip or change it.
Silence can be distracting for me as well, because we don't actually live in true silence. There's constant ambient noise: the kids across the street playing basketball, the car driving by, the dog barking next door, etc… (all happening as I type!) I'm not someone who can ignore every little thing – even if it's not necessarily a bother. I don't mind the aforementioned sounds, but I'm compelled to investigate or see what's going on instead of staying focused. And even just 30 seconds can sometimes be enough for a task to end up going uncompleted.
If I need to focus less deeply – gardening or exercising, for instance – I listen to my favorite podcasts or music that keeps me moving. These things help the time pass and keep me occupied without fully distracting me from the task at hand.
I hope this helps!