By "pulling them out" I hope you mean till them under. Legumes are net consumers of N from the soil unless the plants are tilled under and allowed to decompose.
If you remove the plants you remove all the N. Also if you did not add inoculant to seed, you won't get much N.
I'm in a community garden where I've watched other people grow favas - for nitrogen and also for beans. But this is my first year doing it for myself. I'll add compost to the soil in the spring too. It's easy but also exciting. One of the gardeners in the community had snails eat all their sprouts but I got lucky with a different variety.