You build your track dead straight - like, not conforming to the surface direct through the crust straight. Now the train accelerates downhill for the first half of the journey, and decelerates uphill for the second, neatly coming to a stop at the destination. Oddly enough, in the spherical cow universe where you build this, all the maths cancels such that you get a constant travel time regardless of the start and end locations. On earth it's about 40 minutes
I was assuming the rails are strong enough to keep the train on the Earth, but I guess infinite friction from the movement and rotation of the Earth probably isn't survivable by any railway material. Hypothetically, if you had a material unaffected by gravity (train), and a material that is absolutely invincible (the rails, and they are anchored to the center of the Earth), now does it work?
It's impossible to even reason about what something entirely unaffected by gravity might or might not do because everything, including gravity, is relative. Something not affected by Earth's gravity would probably get carried away by the wind, but something not affected by the Sun's gravity would float away from the sun (relative to the Earth) and appear to have some sort of mysterious acceleration from our perspective. If the object isn't affected by the Milky Way's gravity, even more shenanigans.
No, the problem is not gravity, is that the train attached to earth has velocity dictated by the Earth movements, and keeps it because of inertia. In your theoretical experiment, the train would be launched on space at constant velocity.