Even if drivers were assessed as individuals and a tiered licensing system introduced, you are still at the mercy of the driver who is perfectly safe at a fraction of your licensed speed but not able to assess your velocity.
Isn't this why police and other emergency responders take on a greater responsibility than other road users when they have to use their exemptions ?
The training is to watch others , make eye contact , always be looking for a positive response which indicates that they have seen you and know what you are going to do , before doing it .
Should the lesser able be banned from the road even if they are perfectly safe poodling around?
Of course not : the roads are there for everyone , and other than motorways , which are only for licences users , everyone has the inalienable right to use them in one way or another : pedestrian , cyclist , rider ......