I'm thinking of playing it but what game should I start off with?

Hmmm depends! Persona 2 Innocent Sin/Eternal Punishment (there are two games) are great but it's hard to get one of them in English and its outdated, gameplay and graphics wise, but if you can handle how old it is then go for it. Persona 3 is great but the combat is kind of outdated as you really only control the MC and the other players will do their own thing unless if you tell them to attack a certain enemy or heal/support and even then they will still do their own thing. P3P fixes it with having control of all members and adds the option of playing as a Female MC but it loses some cutscenes and some places are changed. Persona 3 FES has more content added to the base game and has a good sized epilogue. Persona 4 is great even though it is a bit easier than other Persona games and is more light hearted than other entries into the series. That being said it's a fun game all around and has a nice story to go along side it. Persona 4 golden has a lot of stuff added to the base game but that's only on the Vita and Playstation TV. I'd have to say play them in order from P2 to P3 FES, P4/ Golden since for some people it's hard to go back and play the earlier titles since P4 had a better combat system.

Are they all on the ps vita though? Someone told me I should start with 3 or 4 but 3 isn't on the vita or so I've heard.

Persona 3 is, It's called P3P which is Persona 3 Portable. I have that version on my Vita. P4 is also on the Vita but it's called Persona 4 Golden.

Ok so I'll either start with 3 or 4 and I'm gonna get persona 5 when it comes out.

Nice! Can't wait for 5