Try enabling the overhead health bars for mobs that are close. Default key is V i believe. Also there is an addon called "Aloft" that can change those bars more to your liking. I know that doesn't help you with tab targeting, but it does make it much easier to click and target.
Also tab targeting is fairly dependent on the way you are facing, I believe.