1. Make sure your trainer walks the walk -- but DON'T judge by looks alone.
Choose a trainer with a genuine passion for the fitness lifestyle, but remember that looks can be misleading. Often people choose the trainer with the body that most resembles the one they want to achieve, but just as wearing glasses and a tweed jacket doesn't make you a college professor, having big muscles or a tight stomach doesn't make you a good trainer.
I hate to say it, but sometimes the trainers with the best bodies put the least effort into their continuing education and effective coaching methods---because they assume that they're a walking business card, and their physiques are all the credibility they need. Wrong.
2. What are the trainer's certifications? What does the trainer do to keep getting better?
I'll be frank: becoming a "certified" personal trainer is not that difficult. You study, you pass an exam, presto-chango, you're a trainer. While it's important that your trainer carry a reputable certification (NASM, NSCA, ISSA, etc.), in my opinion, the best trainers will go above and beyond to seek out education and experience that separates them from others.
To use myself as an example -- shameless plug -- my certification is with the National Academy of Sports Medicine (NASM), but I'm also certified in the Functional Movement Screen (or FMS, an invaluable tool for assessing a client's motor patterns) and will, in four weeks, take a rigorous course to become a certified Russian kettlebell instructor (RKC Level 1). This summer, I attended two amazing seminars on program design and periodization: one at Peak Performance here in NYC, another with Coach Robert Dos Remedios at my home gym Equinox. In 2012 I have my sights set on becoming a CSCS (Certified Strength & Conditioning Specialist), but that's a longer-term goal.
Ask your potential trainer about his or her continuing education efforts. Gauge his or her passion for learning and staying on the cutting edge. This field changes rapidly, and a trainer who doesn't try to keep up isn't worth your time and money.
3. Observe him or her with other clients.
I think this one is the most important. What is the trainer's coaching style? Drill sergeant? Cheerleader? Silent observer? Make sure it fits your personality. Maybe you want someone to scream in your face -- or maybe that would give you an aneurysm.
Is the trainer engaged with the client---giving cues on form, adequately explaining the exercise selection -- or is the trainer just counting reps, a glorified metronome?
And, perhaps most importantly: is the client sweating? Sounds silly, but a lot of people just want to go through the motions, and their trainers are content to let them do it.
Bonus: is the client having fun? One of my favorite memories from my gym is watching my colleague put his client through a brutal metabolic circuit. The client was drenched and grimacing, and then, without warning, laughed. He was working his ass off, and having a blast doing it.