Have you tried moving the small Actuator arm on the Brake hub a couple of teeth?
Part number 9 on here,
http://www.cmsnl.com/honda-cx500a-1980england_model16465/partslist/F12.html
Part number 9 on here,
http://www.cmsnl.com/honda-cx500a-1980england_model16465/partslist/F12.html