Ping!....................
Practicing for wealth confiscation….
everything is 1’s and 0’s now.
FWIW, I find AWS more reliable, mainly because Amazon actually runs their business on it.
Here is the explanation of what happened
https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident
Google Cloud Customer Support
A Google Cloud incident earlier this month impacted our customer, UniSuper, in Australia. While our first priority was to work with our customer to get them fully operational, soon after the incident started, we publicly acknowledged the incident in a joint statement with the customer.
With our customer’s systems fully up and running, we have completed our internal review. We are sharing information publicly to clarify the nature of the incident and ensure there is an accurate account in the interest of transparency. Google Cloud has taken steps to ensure this particular and isolated incident cannot happen again. The impact was very disappointing and we deeply regret the inconvenience caused to our customer.
Scope of the impact
The below listed impacted technologies and services is a description of only Google managed services.
This incident impacted:
One customer in one cloud region.
That customer’s use of one Google Cloud service - Google Cloud VMware Engine (GCVE).
One of the customer’s multiple GCVE Private Clouds (across two zones).
This incident did not impact:
Any other Google Cloud service.
Any other customer using GCVE or any other Google Cloud service.
The customer’s other GCVE Private Clouds, Google Account, Orgs, Folders, or Projects.
The customer’s data backups stored in Google Cloud Storage (GCS) in the same region.
What happened?
TL;DR
During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank. This had the unintended and then unknown consequence of defaulting the customer’s GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period. The incident trigger and the downstream system behavior have both been corrected to ensure that this cannot happen again.
This incident did not impact any Google Cloud service other than this customer’s one GCVE Private Cloud. Other customers were not impacted by this incident.
Diving Deeper:
Deployment using an exception process
In early 2023, Google operators used an internal tool to deploy one of the customer’s GCVE Private Clouds to meet specific capacity placement needs. This internal tool for capacity management was deprecated and fully automated in Q4 2023 and is therefore no longer required (i.e. no need for human intervention).
Blank input parameter led to unintended behavior
Google operators followed internal control protocols. However, one input parameter was left blank when using an internal tool to provision the customer’s Private Cloud. As a result of the blank parameter, the system assigned a then unknown default fixed 1 year term value for this parameter.
After the end of the system-assigned 1 year period, the customer’s GCVE Private Cloud was deleted. No customer notification was sent because the deletion was triggered as a result of a parameter being left blank by Google operators using the internal tool, and not due a customer deletion request. Any customer-initiated deletion would have been preceded by a notification to the customer.
Recovery
The customer and Google teams worked 24x7 over several days to recover the customer’s GCVE Private Cloud, restore the network and security configurations, restore its applications, and recover data to restore full operations.
This was assisted by the customer’s robust and resilient architectural approach to managing risk of outage or failure.
Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and, along with third party backup software, were instrumental in aiding the rapid restoration.
Remediation
Google Cloud has since taken several actions to ensure that this incident does not and can not occur again, including:
We deprecated the internal tool that triggered this sequence of events. This aspect is now fully automated and controlled by customers via the user interface, even when specific capacity management is required.
We scrubbed the system database and manually reviewed all GCVE Private Clouds to ensure that no other GCVE deployments are at risk.
We corrected the system behavior that sets GCVE Private Clouds for deletion for such deployment workflows.
Conclusions
There has not been an incident of this nature within Google Cloud prior to this instance. It is not a systemic issue.
Google Cloud services have strong safeguards in place with a combination of soft delete, advance notification, and human-in-the-loop, as appropriate.
We have confirmed these safeguards continue to be in place.
Closely partnering with customers is essential to rapid recovery. The customer’s CIO and technical teams deserve praise for the speed and precision with which they executed the 24x7 recovery, working closely with Google Cloud teams.
Resilient and robust risk management with fail safes is essential to rapid recovery in case of unexpected incidents.
Google Cloud continues to have the most resilient and stable cloud infrastructure in the world. Despite this one-time incident, our uptime and resiliency is independently validated to be the best among leading clouds.
Your 1's became 0's.
Coincidentally, the Australian government reported an unexpected $135 billion surplus. “Crikey,” a spokesman stated, “We don’t know where it came from but we’ll take it.”
Funny, that's the same excuse the Clinton White House used when all of Al Gore's subpoenaed emails suddenly vanished back in early 2000, along with all the backups.
The linked article has links to all the back stories by Paul Sperry.
-PJ
Man, people complain about the littlest things. It was only one blank field...
Yeah, The Cloud is safe!..............
xxxxxxxxxxxxxx
good thing they didn’t have it in cryto
I surmise that Google was funded, organized and operated as just another CIA entity.
This is why you should never opt out of getting physical statements mailed to you for financial accounts.
Anyone who would design a system that was this vulnerable should not be in IT!
You have to assume that disasters will happen, and develop systems accordingly.
i dropped a box of 80 column fortran iv program cards once.
fortunately, i had marked the cards with a diagonal stripe, so it was not difficult to put the box of cards back together correctly again.
but it was a close one.
Like Google Maps telling you to take a left, then take a left, then take a left, then take a left, ad infinitum.
Had that happen to me more than once.
If you don’t have the data on your own computers, you don’t own it.
I can restore any data that existed on my computer at at 3AM on any of the following dates. You’d think that a trillion dollar company could do better...
20161231
20171231
20181231
20191231
20201227
20211231
20220630
20220731
20220831
20220930
20221031
20221130
20221231
20230131
20230228
20230331
20230430
20230531
20230630
20230731
20230831
20230930
20231031
20231130
20231231
20240131
20240229
20240331
20240430
20240512
20240519
20240524
20240525
20240526
20240527
20240528
20240529
20240530
I work in cybersecurity. One of the domains is disaster recovery which is the ability to recover compute and recover data. There has been a disturbing trend in the last 5 to 10 years to “trust the cloud”. Folks, the need for backups does not go away just because your systems are “in the cloud”. “cloud” is not some magic wand that makes all of your problems go away.
If the account had been completely erased including all copies would that mean that Google has expropriated the fund. It is all funnymunny I suppose but could google have transferred the funds to itself as the “erasure” was done? Then again if all the accounts google holds were to be erased it would go a piece toward lowering inflation?