You are right. The difference shouldn't be that big between you two. Just, try to keep shield block up, remember to thunderclap and Demo shout to lessen the incoming damage.
As I can see, he has a little more avoidance than you, a little more health and a little more armor. So shouldn't be a big issue.