I rode there on my motorbike in 2015. Riding the washboard was like one long cattle gate. Until I got to about 15 MPH, then it was smooth sailing, the bumps soaked up by the suspension. But when I slowed down to stop, I was afraid I was going to be thrown off by the bumpiness! I made a video, so if you haven't seen the road, here it is
https://www.venturacountytrails.org/...heRockPop.html It's very pretty countryside when you don't focus on the road. The second half of the video is riding up the Left Hand Collet Canyon Rd to Smokey Mtn Road. That's more rugged.
Regarding driving down Hole in the Rock to the bottom, I don't think that would be possible. It would be a pretty difficult hike. More likely a climb. The video shows what the Hole looks like.