I'm not sure I agree with your 'weekend-car' analogy for an E39. Ok I dont drive the M5 and i have a measley 540i but its just too big to have any real fun in. Mine has the sport pack with the M suspension etc, so its like an M5, just without the running gear and the engine.
It handles really well for its size, but it does give you the feeling you are in a smaller car when you are sitting in it, even though its a bit of a porker really. Dont know if you'll like this type of thing, but I know I dont. I like to have crystal clear feeling to what the car can and cannot do, the e39 is very predictable and easy to have a bit of fun in, but it crosses the margin of too big for a sports car in my eyes, mainly because of the 'feel' that it gives you, misplaced confidence almost that it is a smallish car that handles really well, but when you give it an obstacle which only a small car can do, you will quickly see that it isn't! Hard to explain but at least I tried!
On the other hand its an amazing motorway cruiser, I recently went to europe with it to 5 different countries and it was absolutely beautiful. Returning an average of 33.6mpg aswell, couldn't ask for much more really. Its very fast yes, but still doesn't handle as well as you'd might think, its on a steering box, not a rack & pinion like the 6 cyl 5er's or 3er's.
All that said though, i'd still say have a drive and see what you think. But thats my 2p.