I suppose the better question I should of asked is at which point does, if ever that slam, overpower, and mortal strike beat execute dmg
Execute dmg formula is 1456+AP*0.2+1520 with max 30 rage and glyph of execute
Slam, Overpower, and Mortal strike are based on weapon damage, (Weapon DPS + AP/14) * Weapon Speed, slam adds 250, mortal strike 380 and talented for 10% more damage.
Using Betrayer of humanity, 222.6 dps 3.40 speed, the above formula stated for weapon damage and simplifying the equations...
2976+AP*0.2 execute
1006.84 + .243AP slam
1107.52 + .267AP mortal strike
756.84 + .243AP overpower
Now if I did my simple alegbra right, Overpower dps will overcome execute dps around 51.6k ap, slam around 45.8k ap, and mortal strike around 25.8k ap. This may change with higher content and superior weapon dps(though doubtful), but currently execute dps will always do more damage than any other ability so it would be logical to assume stacking as much crit as possible to make it proc more is the best way to improve dps.
The next argument would be at what point is ap more beneficial to execute dps rather than more crit. Which i might do at a later point but its 4am