Hmm... It sounds like your son is the same age as mine, and a somewhat similar personality. Our little boy happily plays with adults wonderfully, but even the friendliest kid gets the cold shoulder. The best way I can describe it is that kids his age are treated you would treat a strange dog wandering around the neighborhood unleashed.
He also doesn't talk much - we had him evaluated back in February and he was a year behind in his talking. He has since started speech therapy through the Charles County Infants and Toddlers program, which includes a once a week small group session with two or three other kids, and this summer he will start second weekly group 'school' session there.
Knowing absolutely nothing about your son other than what you posted, my only thoughts are this:
If he's uncomfortable around kids, a smaller group might be preferred to a larger group, since there is so much less to process. If he's getting overwhelmed and flipping out, putting him a room with a dozen three year olds and just expecting him to 'deal with it' may not get the best results.
My mom (who worked with child development from 0-3 for over 20 years), always recommends preschool. Kids learn a lot from each other, and while you always have the option of un-enrolling him if it doesn't seem to work out, if you skip it entirely you can't go back in time and try again.
as others have said, your first priority is to do what you think is best for your son.
finally, at the risk of derailing this thread entirely, if you're really worried about his social interaction or level of verbal communication, you can talk it over with your doctor or call the county Infants and Toddlers program directly and ask for an evaluation. If he's 25% behind for his age in any category (gross motor, fine motor, cognitive, social/emotional, language, self help) he'll qualify for services and potentially some form of preschool through the school system.