The general rule of thumb:
Being over the Defense minimum (540) is never a bad thing as long as you don't actively try to acquire it, meaning one shouldn't ever gem or enchant for Def if you are over 540. Stamina, Armor, and Avoidance will all be better for you. Now if you end up with a bunch of defense from gear that were all clearly upgrades then power to ya. You'll just have more avoidance which is rarely bad.
"A man can learn twice as much from milking a cat as two men can from a very tall fourth man.
- Mark "Alec Baldwin" Twain