We're both on the same college and the same course. We've met on the first lab class and became friendly over time. We often walk 'home' together because we both need to catch a train to get home. Currently I'd say we're in a playful friendship.
He's funny, a bit full-of-himself but kind, and I think he likes hanging out with me. We used to study together a lot too, and he'd often ask if we're going home together after class. Sometimes he'd even give me a fist bump or a slight nudge (missing a word here, not a native English speaker), or he'd send me a picture of something he'd made or seen (which I find cute). Yet we don't really talk as much over text as we do in person.

First problem - he's a flirt and he's very direct. I actually like he's direct, but he's... not beating about the bush at all. If something is on his mind and he wants to say it, he'll say it straight to the point. He's friendly with everyone and talks like that with everyone, so I'm not sure I'm so special to him.
Second problem - I have huge trust issues. Having been in a very bad community (society? surrounded by bad peers? missing a word again) where I have been more or less ostracized, manipulated and used for a long time, I've become an insecure and self-conscious mess. I've improved over years and have progressed so much over last year, as I've finally got my fresh start (getting into uni where I literary knew no one previously).

I've become much less shy and am much happier. My self-esteem is much higher now and I've finally reached a point where I've decided to overcome my fears and build up my social skills. I want to go out with people and have fun, now that I'm not bound to the previous community and am in a new one where people actually want to know me as a person.
But after deciding to overcome my fear of rejection - I realized I have a hard time trusting other people, even those that are closer to me than he is.
What I fear most is that if I do get rejected or it doesn't end well, he'll spread the word around and I'll be back at the beginning. I really can't see why he'd do that, yet the fear remains.

I don't know what details to write further, so if you're missing a part of the picture in order to help please tell me.
I really want to start something with him, and I've found the courage to ask him out, but am confused with my own state of mind. What to do?