Another IT Meltdown..

[st13phil]

Every update to core software involves a risk. In the past we've had fights with IT departments about dogmatically mandated updates that have knocked out internal networks. They just won't listen. Automated updates are a boon from the point of view of keeping software up to date against threats but they are also a point of failure that nobody wants to worry about.

What most people outside the world of IT (and a disturbing number within it) don't appreciate is that just as in the physical world there are unmitigated single points of failure in lots of IT infrastructure just as there are in the physical world.

Also, just as in the physical world - the Covid pandemic shone a light on this regarding complex supply chains - these single points of failure are unmitigated because they are often hidden deep in the depths so people don't realise that they they exist, or because people are totally trusting of their supplier.

In the instant case of the Crowdstrike issue today, despite it having been caused by an automated update, the big issue is that it is impossible to remedy it by automated means as it causes the affected machines to fault with a BSoD at boot time. This means that each and every machine has to be manually restarted in Safe Mode so the fix can be deployed, and then re-started again. Lots of work for those who now have no choice but to do it.

 

[Aristo]

I suspect we may not ever really know, but I'd be interested to understand how this can happen - with something as critical as that, you'd think there'd be a multitude of test scenarios undergone to mitigate against such scenarios.

 

[MikeInWimbledon]

Our government spent millions of our money some years ago in order to get England's NHS Trust systems to talk to each other. They miserably failed.

No.

Tony spent £12 billion, not million, on getting NHS Trust systems to share patient records.

To be fair, they started off with a budget of just over £2 billion, but they just added a few bits. £10 billion worth of extra bits.

The Trusts and medics couldn't agree how to do it.

So they gave up, and threw the entire £12 billion away.

A lot of consultancies and IT staff were extremely grateful to the Labour party for their work. Many retired early on the back of it. "Even easier money for old rope than Y2K," they said.

Could be worse. At least they didn't rush to spend £1 billion over 2 years on a tent in Docklands to host a Millennium Party.

NHS Cancels £12 Billion IT Program

The National Health Service's National Programme for IT was created in 2002 to ensure that every NHS patient had an individual electronic health record that could be easily shared across the NHS.

www.commonwealthfund.org

 

[SpikyMikey]

Have Mercedes had to stop everything during practice at Hungary to update all their software? At least they probably have first call on the Cloudstrike guy who knows how to do it.

Microsoft IT outage live updates: Worldwide travel and banking hit after cybersecurity update causes IT chaos

 

[Pontoneer]

I never knew anything about it until I read this thread!!....no problems at work yesterday....and we use Swift a lot.

Same here , but I have only Apple computers , and no Microsoft malware whatsoever on any of them .

 

[Dryce]

Same here , but I have only Apple computers , and no Microsoft malware whatsoever on any of them .

Most of my experience of large clients enterprise systems is that they are Windows. Only one of the large organisations I've encountered used Apple.

BUT

The one that used Apple had added in additional software to deal with locking the system and network down with the same sort of slow down / hassles / strange and useless messages and glitches that the corporate Windows stuff experiences in this situation.

So I suspect that what we're seeing isn't down to Apple's success - but Apple's failure to penetrate these markets in any significant way. How many Apple error messages do you ever see on public information displays or kiosk systems ? Well you won't see Apple error messages if Apple don't exist in that space. So if Apple dominated the corporate sector we might well be seeing posts today on how some enterprise defence and lock down software had taken down Apple systems after a problematic update.

.

 

[Meldrew2]

I had a rather quiet day at work today - I think that some customers avoided coming in thinking that we could be affected, but with Apple computers and Android stand-alone credit card terminals everything seemed to work fine - and at least I managed to catch up on some admin...

 

[Rayny]

Haha ... I remember being paid a small fortune to be sitting there ready on New Year's Eve "just in case".

<Jealous mode>
I remember spending two years looking into the 'problem'.
I went in to the office for 10 minutes on new years eve to ensure that all systems had been shut down.
I went in to the the office for a couple of hours on new years day to ensure that all systems worked OK.
I did not get paid overtime (At that time I was paid less than the current national minumum wage).
</Jealous mode.

Many people earned mega-bucks because of Y2K ( I still have the Midland Bank video showing how the bug might affect computers), in the same way that many people earned mega-bucks during the Covid era..

 

[190]

A lot of misinformation in the general media or perhaps just ignorance in that they are calling it a Windows outage which would have you believe that all computers running Windows are potentially affected. As ever it's a good story to make it sound worse than it is, though in this case it's bad enough without any need to exaggerate as one consequence has been that 4.6% of flights worldwide have been cancelled today.

It's hard to pin down numbers but only something in the region of 1 in 10 computers or systems could be affected. If you didn't buy Crowdstrike Falcon then there isn't a problem and no need to switch to Apple or Linux. The only reason those users are feeling smug is that Crowdstrike Falcon wasn't available for their operating system.

It seems before this Crowdstrike Falcon was considered a very good product so a shame they stuffed up.

 

[markjay]

and the result is different how?

It's really a question of whose insurer should pay-up.

 

[m80]

It's really a question of whose insurer should pay-up.

Do we know the insurer is capable?
Summat tells me thar MS are as tad more affluent, and the one's the litigators might be best targeting.

Maybe we discuss this during a court case, in 5 years or so.

 

[fabes]

Whilst Crowdstrike will have Cyber insurance, they won't have enough. Nobody would (if indeed it paid out).

And they are too important to take to court and send the bailiffs into as yesterday evidenced.

Most (my lot included) will value this as a valuable BCP test scenario.
IT managers will do well in terms of getting beefed up budgets etc.

If nobody died, 'we' did well, although delayed surgeries etc will be awful if that was a consequence

 

[Petrol Pete]

But if you try spending it everyone asks where you got it from.
Keep a fair amount and it can be confiscated, until you prove it was lawfully gained.

I only keep a bit of it for car parks nowadays. But that's becoming outdated too.

Where I live the council (just last week) installed new pay meters in all of the car parks, Better tech than the ones they replaced .

They still have coin slots in them , I used one yesterday . Nice to se someone is paying attention .

 

[ALFAitalia]

I can't remember the last time I put coins in a car park meter.....its not 1990! And with the cost of them if you plan to be there more than a couple of hours....well who ever has that much money in change. A note slot would be more useful!

 

[Petrol Pete]

No.

Tony spent £12 billion, not million, on getting NHS Trust systems to share patient records.

To be fair, they started off with a budget of just over £2 billion, but they just added a few bits. £10 billion worth of extra bits.

The Trusts and medics couldn't agree how to do it.

So they gave up, and threw the entire £12 billion away.

A lot of consultancies and IT staff were extremely grateful to the Labour party for their work. Many retired early on the back of it. "Even easier money for old rope than Y2K," they said.

Could be worse. At least they didn't rush to spend £1 billion over 2 years on a tent in Docklands to host a Millennium Party.

NHS Cancels £12 Billion IT Program

The National Health Service's National Programme for IT was created in 2002 to ensure that every NHS patient had an individual electronic health record that could be easily shared across the NHS.

www.commonwealthfund.org

I also hear there are something like 500 fax machines (remember them kids ?!) still in use in the NHS. Might be an urban myth , buy it could make some sense in a total meltdown of IT.

 

[Petrol Pete]

One side of this whole hoo hah that has kind on not been highlighted by main stream media is how during the event people were being asked to pay cash on the London underground (and various other transport locations) , this of course proved problematic for some people .

Retailers, theatres, visitor attractions worldwide ....

Bit of a wake up call for some of those being led by the nose into the cashless society that those in power so strongly desire.

Needless to say there is always a 'chunk of change' kept somewhere on my property (and bit on my person) all of the time. I'm just old fashioned like that.....

 

[Darrell]

One side of this whole hoo hah that has kind on not been highlighted by main stream media is how during the event people were being asked to pay cash on the London underground (and various other transport locations) , this of course proved problematic for some people .

Retailers, theatres, visitor attractions worldwide ....

Bit of a wake up call for some of those being led by the nose into the cashless society that those in power so strongly desire.

Needless to say there is always a 'chunk of change' kept somewhere on my property (and bit on my person) all of the time. I'm just old fashioned like that.....

I’m big on cashless tbh and don’t keep money at home. Can’t remember the last time I had any cash.

 

[MikeInWimbledon]

I also hear there are something like 500 fax machines (remember them kids ?!) still in use in the NHS. Might be an urban myth , buy it could make some sense in a total meltdown of IT.

The NHS certainly had them just before Covid.

And I believe it was also the world's largest user of .... pagers....at the same time.

 

[m80]

Some pubs were asking for cash,
some even closing their doors.

A no Guinness society becomes a serious matter.

 

[markjay]

Why are we relying on IT, in spite of the obvious risks? Essentially, because the benefits outweigh the harm this is occasionally causing us.

This trend has been going on for hundreds if not thousands of years. It started with early human societies where people no longer hunted or grew their own food, but relied instead on others to do that while they themselves embarked on other tasks that benefited the community.

Mishaps do happen. The Irish wouldnt have been affected as badly as they did by the Great Famine, had each family been growing its own food in their back garden. The US economy wouldn't have grinded to a halt in the seventies when gasoline was in short supply, had they not built an economy that largely relies on people's ability to travel by automobile. Etc etc.

The point is, that IT - like cars' seatbelts - on occasion can be harmful, but on balance it does far more good than bad.

Have we gone too far with our reliance on IT systems, on complex supply chains, on globalisation, and now also on AI? Well, I don't know, but given that there's nothing particularly unusual about the current decade, I would argue that from thevstatistical perspective it is an arbitrary claim that 2024 is the year that the reduction in our self-sufficiency has gone too far - simply because this argument would have been equally valid in any given year over the past ten millenia.