When the big bang happened and all that exist today came into existence from singularity, how do you determine which part of the universe is the oldest? Its natural to think that looking back at the center of the universe, we would find the oldest galaxies(not sure if this is true); but isn't there a certain logic that the farthest from the center should have the oldest galaxies? Is there a proven theory behind where we would find the oldest galaxies? or is this one of those space-time enigmas?
So, I agree with the other answers, that as we look further away from our Galaxy, we see galaxies as they were in the past. Therefore, by and large, these galaxies are younger than ours. That is, we see them as they were when they had an age that could be at most 13.7 billion (the age of the universe) minus their distance in light years.
Therefore to find the oldest objects in the universe, we must look locally. Probably the most ancient galaxies that we can study, are those that are thought to be the building blocks of larger galaxies. These are the "dwarf spheroidal galaxies", several of which have been identified as companions to the Milky Way and M31, and are probably remnants of a larger population of such objects. Their stellar populations appear to consist of mostly intermediate age stars through to extremely ancient stars.
Other ancients objects include the extremely metal-poor stars of the halo population of our Galaxy and a number of very metal poor globular clusters.
Estimates for the ages of such objects are uncertain, but are likely in the range 12-13 billion years old and so only a short time after the big bang.
The key point is that the Big Bang happened everywhere at the same time, hence there is no centre (or alternatively, the centre is any point). This was not an explosion into space, it created space. (Think of a balloon, when you blow it up points on the surface all move further apart without there being a centre of the expansion on the surface itself.)
A related idea is the "cosmological principle" (CP) which, simply put, states that, at a large scale, the universe looks the same everywhere. That means the "oldest" parts (as measured by how long it look radiation to get here) of the universe can be seen in any direction - and the CMB is just that and it does come from all directions. You can think of it as the burst of light that was emitted when the universe, through expansion, cooled just enough to let atomic nuclei recapture electrons. The electrons emitted photons and those have now been redshifted by the universe's expansion to match the spectrum of a body 'glowing' at about 3 degrees Kelvin.
(In fact there is some evidence to suggest that the CP is not valid or at least that there is a question about what scale constitutes "large scale": there is more here https://en.wikipedia.org/wiki/Huge-LQG#Cosmological_principle)
If all that exists originated in the big bang, as per the begining of your question, why then would you be searching for the oldest part of something that was globally formed at the same instant? I wouldn't ;-) Thankfully, the finite light speed allows us to view objects as they were during younger times at increasing distances.