Master's degree in US

I keep hearing from everyone how it is extremely difficult to do grad school in the US. They say that studying at school is very easy, but university/college and, especially graduate school knocks you down, that is how hard it is... Now, I am not a genius, but also not stupid, that makes me nervous.  
Can students share their experience with me, please? Those who have been there and done that. I have not been in class for years and going back to school scares me as much as it excites me. Is it doable?