Depending on how crowded it is you should generally be fine.
So, how PvP zones work: If you are not flagged up, you cannot attack others who aren't flagged up. If you are flagged up, you can attack anyone, but you can also be attacked by anyone, whether or not they are flagged.
I think you have to mark yourself for PVP to actually be able to attack or be attacked. At least that is what I understood when the tutorial messages popped up while uncovering the map there. Never got attacked while riding through there.
That’s what I thought too, except I definitely didn’t mark myself and a lvl 60 attacked and obliterated me the second I stepped out of the town next to the PVP area