Access to military installations is considered a privilege, not a right. The possibility of entry varies by location. The base's CO and existing agreements with local governments play a role in determining who has access.
Having a VA ID card doesn't entitle you to entry. Even with my retired military ID, I've faced denial at certain installations.