What really happened to Facebook

[NightTrain]

I was just suggesting to Abbey yesterday that the timing was so coincidental that it might be Mark Zuckerberg intentionally deleting all their data to keep it out of the Congressional investigation. But that doesn't make any sense because FB needs all its data, plus you can never really delete data once it is out on the Internet.

Right, a worldwide outage seems really weird and I don't think the procedure is to do a global networking change. I would think you'd do one at a time, and certainly not in the middle of the day. Telecom always does network-impacting work at night, from midnight to 6AM to minimize customer outages and it's ONLY one at a time in case there's a problem.

The kind of data I immediately thought of were the censorship, tracker and suppression algorithms. I don't know why you'd need to go dark for nuking that stuff, but again, I'm not an expert on the big boy servers by any stretch.

What I think is more likely is that they accidentally created network routes that pointed at each other, creating a 'routing loop' which is the most dreaded problem in the network world. It's beyond just DNS - FB people couldn't even use their access badges to get through doors in the FB buildings.

I've done that. It was embarassing.

I'd just built a brand new LTE microwave network from Valdez to Cordova, I think there were 12 hops.. the furthest was 22 miles between sites. Everything was humming along, but as I was looking at the nifty GUI, it seemed to me that I could make her run faster by creating a loop from my end using the A-side radios to Cordova, and route the return traffic on the B-side radios. Seemed like it was worth a try, and why not? No one was on it yet, so I could experiment.

It locked up every single site within 30 seconds with the biggest Ethernet Storm you could imagine. I had to manually pull the power off the equipment in front of me to kill it. I had to reboot the system and keep the transmitters offline, change it back to where I had things, then turn it back up. The problem was that all the other sites weren't listening to anything except the massive ethernet storm I'd unleashed.

That little experiment cost a helicopter ride for me and my laptop to every site so I could manually kill the power and change it back before turning the transmitters back up... probably a $10k joyride at least.

[Russ]

I'd just built a brand new LTE microwave network from Valdez to Cordova, I think there were 12 hops.. the furthest was 22 miles between sites. Everything was humming along, but as I was looking at the nifty GUI, it seemed to me that I could make her run faster by creating a loop from my end using the A-side radios to Cordova, and route the return traffic on the B-side radios. Seemed like it was worth a try, and why not? No one was on it yet, so I could experiment.

It locked up every single site within 30 seconds with the biggest Ethernet Storm you could imagine. I had to manually pull the power off the equipment in front of me to kill it. I had to reboot the system and keep the transmitters offline, change it back to where I had things, then turn it back up. The problem was that all the other sites weren't listening to anything except the massive ethernet storm I'd unleashed.

That little experiment cost a helicopter ride for me and my laptop to every site so I could manually kill the power and change it back before turning the transmitters back up... probably a $10k joyride at least.

Wow, that is a great example! I'm sure it feels better to refer to it in the past tense. At least you got a bunch of helicopter rides out of it.

[NightTrain]

Wow, that is a great example! I'm sure it feels better to refer to it in the past tense. At least you got a bunch of helicopter rides out of it.

I love those things. Such wonderful machines!

The great part about that was I was the only one fine tuning it, and I hadn't handed it over to the customer yet. It was very educational without the pain usually accompanying such unforgettable lessons.

I'm very cautious about Ethernet Storms since then, especially when I'm modifying remote sites.


Back to the Facebook outage, I wonder if a hacker group gained control and held it for ransom. Zuckerberg has the best IT guys money can buy, but there's real talent lurking in the dark underbelly of the interwebs.... and he's got deep pockets.