I'd certainly enjoy a break from the incessant yammering of humanity.
It's not just a break from humanity, it's a break from everything (including nature, which typically fills the void when humanity is absent).
If you haven't been in a very quiet space, it becomes disorienting very quickly. The quietest space I've been in has been <12 dBA (my measurement equipment only went that low so I couldn't get the true Lmin). At that level, I could hear the blood rushing through my ear. I have a minor case of tinnitus which is typically inaudible, but in that space it was like an alarm clock screaming into one ear. The other ear seemed to desperately find something to "lock" onto, so it was amplifying every little thing it could.
After about 15 minutes I wound up opening a window to let a little wind noise into the room. That raised noise levels up to about 19 dBA and was much more comfortable.
I've been in an anechoic chamber and it's not so much the lack of sound but the intense visual shock. You're usually in a room with incredible depth levels, sharp angles and so on. It can induce a feeling of sea-sickness.
The absolute silence exacerbates the issue but it's not the only factor at play.