They don't give a feel of being too difficult to conquer. It seems if our Gang comes together they should easily difeat the lords. Why didn't they think of it yet?
It strikes me as the same situation as The Black Thorn: 'Circle of the Black Thorn is the most powerful group in this plane of existence. Together, they'd vaporize us, but separated, they're just demons.'

All the Lords rule over thousands of humans & (one presumes) have plenty of soldiers to fight for them too - remember Sebassis, who apparently had 30000 troops? Doing what Angel is doing and issuing a direct challenge, where the winner gets the other's territory, might be the only way of doing it without being ripped to shreds by bloodthirsty hordes.