I'm not saying they aren't suitable, but like the other posters you've highlighted extremely high spec engines in £40,000 + vehicles where, if we're honest, economy isn't the number one concern.
Eventually we'll see diesels in sports cars, after all Audi did very well at Le Mans in them.
The other thing is the weight of the engine, a diesel engine of similar power and performance to a petrol engine is generally substantially heavier.
Now in an ML, GL or S class, even an E class this isn't particularly noticeable, but in smaller vehicles it is hugely noticeable.
Anyhow we've kind of gone off at a tangent.
The Audi A8 4.0 TDi is an absolute gem of a car, but if I'm in the market for a vehicle of that value or a 420CDi for that matter, I really don't give a monkeys how much it costs to run. If the running costs are too much, you're probably looking at the wrong vehicle to start with.
Whilst Diesel engines have come on immeasurably in the last 5 years, the ones that really matter, that sell in the tens of thousands - those under 2 litre, aren't there yet....
Look back up through this thread, and every argument has been for a 320CDi or larger capacity, all of which are luxury vehicles and cost more than strictly necessary.
Maybe this means the average buyer can afford to get into a car that they wouldn't normally be able to afford to run, in which case that's starting to make more sense.
Personally, If I'm buying a £60,000 or £70,000 S class Mercedes Fuel consumption is pretty low down the list, (if it's on it at all).