After all, the key moment of meditation is when you realize you were wandering. Buddhism sees that tiny moment as enlightenment, because you somehow wake-up.
So, getting back into your question: as soon as you realize you've been wandering, come back to focus. Don't judge yourself, at the contrary, be happy for noticing it. And simply return. Where to? It can be anything, really, as long as you keep track of it. For me it is breathing, for others it can be mantras, visualization or even more.
Glad to help, have a nice meditation.