In general, drums are much harder to play than many people think they are. And, I’m not sure that there is just one drum that is the hardest to play. To be able to play the drums well, as with any instrument, it requires practice, and more practice. It requires a lot of coordination to play the drums.
Each hand and foot independently plays different rhythms while your eyes read the music. There are 40 basic patterns or rudiments that can be played on drums. It’s true you can play many songs without learning all the rudiments but a good drummer should know them all.
