What really happened to Facebook

[Bilgerat]

[NightTrain]

Whatever happened to old Tom?


Conspiracy theories being bandied about the Facebook whistle-blower about to testify tomorrow morning.

That was a worldwide outage. I've never done in-depth IT on servers, but maybe @jimnyc and @Russ can give us some insight on if taking everything offline for 7 hours is necessary to thoroughly nuke embarrassing data.

[Kathianne]

Whatever happened to old Tom?


Conspiracy theories being bandied about the Facebook whistle-blower about to testify tomorrow morning.

That was a worldwide outage. I've never done in-depth IT on servers, but maybe @jimnyc and @Russ can give us some insight on if taking everything offline for 7 hours is necessary to thoroughly nuke embarrassing data.

Speaking of conspiracy theories, it seems Lin Wood was at a paid event and wanted to let those attending know that one of the ugly truths/facts of America is that we were all played, 9/11 was a computer generated image. One of the 'facts' is that there were no aircraft debris at any sites and WTC 7 collapse was to cover the financial misappropriation of military funds.

Hopefully he'll join Powell on personal non gratis list for Trump, whether he runs or not.

[NightTrain]

Speaking of conspiracy theories, it seems Lin Wood was at a paid event and wanted to let those attending know that one of the ugly truths/facts of America is that we were all played, 9/11 was a computer generated image. One of the 'facts' is that there were no aircraft debris at any sites and WTC 7 collapse was to cover the financial misappropriation of military funds.

Hopefully he'll join Powell on personal non gratis list for Trump, whether he runs or not.

I just watched that cringe video. What an absolute nutjob.

The audience was stunned by the lunacy, too.

[JakeStarkey]

oh my

[Russ]

Whatever happened to old Tom?


Conspiracy theories being bandied about the Facebook whistle-blower about to testify tomorrow morning.

That was a worldwide outage. I've never done in-depth IT on servers, but maybe @jimnyc and @Russ can give us some insight on if taking everything offline for 7 hours is necessary to thoroughly nuke embarrassing data.

I was just suggesting to Abbey yesterday that the timing was so coincidental that it might be Mark Zuckerberg intentionally deleting all their data to keep it out of the Congressional investigation. But that doesn't make any sense because FB needs all its data, plus you can never really delete data once it is out on the Internet.

What I'm hearing is that FB's network team made some change that accidentally deleted all of its externally-facing DNS records that tell the world where the name 'www.facebook.com' is located. And along with that they deleted all the BGP routes that directed network traffic in and out of FB data centers. That doesn't really pass the sniff test with me, though. FB has too many data centers. Too many records would have to be deleted, and these are critical records that have to be approved by 3 million review teams before anyone can touch them.

What I think is more likely is that they accidentally created network routes that pointed at each other, creating a 'routing loop' which is the most dreaded problem in the network world. It's beyond just DNS - FB people couldn't even use their access badges to get through doors in the FB buildings.

It will be interesting to see if FB news articles start using the word "loop" in the next few days.

[JakeStarkey]

Friend said to me that Zuck sent a msg to the government as well as you and me: "Don't fuck with me."

[Bilgerat]

[NightTrain]

I was just suggesting to Abbey yesterday that the timing was so coincidental that it might be Mark Zuckerberg intentionally deleting all their data to keep it out of the Congressional investigation. But that doesn't make any sense because FB needs all its data, plus you can never really delete data once it is out on the Internet.

Right, a worldwide outage seems really weird and I don't think the procedure is to do a global networking change. I would think you'd do one at a time, and certainly not in the middle of the day. Telecom always does network-impacting work at night, from midnight to 6AM to minimize customer outages and it's ONLY one at a time in case there's a problem.

The kind of data I immediately thought of were the censorship, tracker and suppression algorithms. I don't know why you'd need to go dark for nuking that stuff, but again, I'm not an expert on the big boy servers by any stretch.

What I think is more likely is that they accidentally created network routes that pointed at each other, creating a 'routing loop' which is the most dreaded problem in the network world. It's beyond just DNS - FB people couldn't even use their access badges to get through doors in the FB buildings.

I've done that. It was embarassing.

I'd just built a brand new LTE microwave network from Valdez to Cordova, I think there were 12 hops.. the furthest was 22 miles between sites. Everything was humming along, but as I was looking at the nifty GUI, it seemed to me that I could make her run faster by creating a loop from my end using the A-side radios to Cordova, and route the return traffic on the B-side radios. Seemed like it was worth a try, and why not? No one was on it yet, so I could experiment.

It locked up every single site within 30 seconds with the biggest Ethernet Storm you could imagine. I had to manually pull the power off the equipment in front of me to kill it. I had to reboot the system and keep the transmitters offline, change it back to where I had things, then turn it back up. The problem was that all the other sites weren't listening to anything except the massive ethernet storm I'd unleashed.

That little experiment cost a helicopter ride for me and my laptop to every site so I could manually kill the power and change it back before turning the transmitters back up... probably a $10k joyride at least.

[Russ]

I'd just built a brand new LTE microwave network from Valdez to Cordova, I think there were 12 hops.. the furthest was 22 miles between sites. Everything was humming along, but as I was looking at the nifty GUI, it seemed to me that I could make her run faster by creating a loop from my end using the A-side radios to Cordova, and route the return traffic on the B-side radios. Seemed like it was worth a try, and why not? No one was on it yet, so I could experiment.

It locked up every single site within 30 seconds with the biggest Ethernet Storm you could imagine. I had to manually pull the power off the equipment in front of me to kill it. I had to reboot the system and keep the transmitters offline, change it back to where I had things, then turn it back up. The problem was that all the other sites weren't listening to anything except the massive ethernet storm I'd unleashed.

That little experiment cost a helicopter ride for me and my laptop to every site so I could manually kill the power and change it back before turning the transmitters back up... probably a $10k joyride at least.

Wow, that is a great example! I'm sure it feels better to refer to it in the past tense. At least you got a bunch of helicopter rides out of it.

[NightTrain]

Wow, that is a great example! I'm sure it feels better to refer to it in the past tense. At least you got a bunch of helicopter rides out of it.

I love those things. Such wonderful machines!

The great part about that was I was the only one fine tuning it, and I hadn't handed it over to the customer yet. It was very educational without the pain usually accompanying such unforgettable lessons.

I'm very cautious about Ethernet Storms since then, especially when I'm modifying remote sites.


Back to the Facebook outage, I wonder if a hacker group gained control and held it for ransom. Zuckerberg has the best IT guys money can buy, but there's real talent lurking in the dark underbelly of the interwebs.... and he's got deep pockets.