I suppose that's possible, but I've had small cars and never noticed a difference. I just had a quick Google and only pages that say 10 or 20% never have a research source, it just seems to be like a urban myth. I'm not saying it doesn't increase fuel consumption, it must do, but I reckon it will be almost undetectable unless one is a hypemiler.
If your car has six horsepower, and you are using a air conditioning compressor that draws one horsepower, leaving you with five horsepower, then switching it off will increase your horsepower from five to six, which is a 20% increase in power, and hence, in my mind at least, efficiency...