Indeed, a driver travelling at 50 mph in the outside lane is perfectly justified to be there if overtaking traffic moving more slowly in the other lane(s). If someone behind wishes to go faster, he (because it invariably is a "he") should pull back to a safe distance and wait until the lane is clear again.
It depends why he or she is travelling at 50 MPH. If it's because he or she is being held back by a slower moving vehicle then ok. If it's because he or she is too timid to travel at a higher speed, or is just being provocative, then the circumstances may support the argument that he or she is causing an unnecessary obstruction, and may be it's him or her who should "pull back and wait until the lane (behind) is clear".
"Invariably"? I don't think so.