It makes sense if you can reach full CTC (combat table coverage) meaning that you reach
5% base miss chance (+ 2% NE racial) + % chance to block + % chance to parry + % chance to dodge >= 102,4%
At that point ,you won´t get hit by unblocked hits anymore, very hit you take is either blocked (40% less damage) or critically blocked (60% less damage), which smoothens incoming damage extremely well. Also, mastery is overall the best stat in terms of taking the least damage.
So, iy you are near CTC, it makes senso to leave out non-mastery bonusses and gem straight for mastery.