My trip to the NAP Datacenter was an unexpected journey. I wasn’t supposed to go, actually. Trevor had invited me to go months ago and I couldn’t make it that time, but when I heard HiringSolved was going on another field trip, I asked if I could join.
Shon was enthused about the outing for a moment, but then realized I couldn’t go. “You need two forms of ID, so it’ll have to be next time.”
“It just so happens I’m carrying two!” I was back in on the away mission.
Photos of Phoenix NAP facility.
We parked at a building that I had passed many times before. A very nondescript and boring building from the outside. I had driven past the corporate art and boring block architecture many times on my way to Arizona State University back in my college days.
We exited at the boring building and walked up to a tinted glass door–a common accent for the hot Arizona sun. Nothing impressive so far.
“This is by design,” Shon told me. “Notice how there are no logos or labels of any kind on this building”
Now that you mention it, this building is pretty nondescript… I was thinking. Has anyone even moved in? I see the cars around and everything but surely they’re just here for the Waffle House across the street.
Nope. This is NAP. And despite the cozy name, the security here never sleeps. We got up to the nondescript door of the nondescript building and Shon hit the nondescript buzzer on the nondescript pole in the nondescript cement.
A nondescript voice answered: “Bed—fft ba—ft and be—-ndfssf”
I didn’t really hear it.
“Shon Burton, HiringSolved,” Shon said into the speaker.
Click. The doors were unlocked. We walked over and Shon was saying, “See? There’s nothing signifying what this place is. Again, that’s by design. I’m surprised they even have numbers on the door. Usually they try to hide even the address at these things.”
Clearly this wasn’t Shon’s first rodeo.
Entering the Center
I sheepishly walked in with my hands firmly at my sides. Then clasped behind me. Then crossed in front of me. I didn’t know where the hell to put my hands.
The two security guards were wearing kevlar vests and a bulletproof pane of glass separated them from us.
One of them started grilling Shon on what hard drives he was taking in.
“Brand X,” Shon said.
“How many gigs?”
“How many drives?”
“And your guest?”
“This is Christopher Murray. He’s new and we’re giving him a tour.”
They looked at me and said, “Two forms of ID please.”
I fumbled around in my Beatles wallet and pulled out my green card and Arizona driver’s license.
“Wait, you have a green card?” Shon said.
This was the first time I explained my Irish heritage and how even after living in the USA for 26 years (since I was 2 years old), I am still an Irish citizen. I saved some of the hairier details for later.
The mission that day was to infiltrate the data center, remove the failing drives, and replace them with newer, more reliable ones–some real cyber-punk shit!
The guards handed Shon and I a pair of badges to get us into the data center.
Behind them, I noticed a grip-ton of TVs set up to surveil the surrounding campus as well as the hallways of the complex. They probably had them on display like that to remind people they’re being watched.
The place was a fortress.
Data Meditations, Scorpoyotes, and Porn
I started to think about how malleable data is.
The Pirate Bay, for instance, has node accounts at data centers all over the world that activate in the event that one is taken down. It doesn’t matter if their building blows up because there will always be another backup of the data somewhere else.
For Gangplank startups like HiringSolved, however, this building blowing up would be quite a different situation.
If this building exploded, imploded, burned down, fell into the swamp, was raided by scorpions, coyotes, scorpion-covered coyotes, or giant scorpion-coyote hybrids that stabbed all the security with their scorp tails while half of them ran high powered magnets through the complex to wipe everything; all the startups would be doomed.
Or at least you’d think so.
When I brought forth my Scorpoyote concerns Shon showed me how I shouldn’t be so concerned.
“Technically, at least for us, it’s inaccurate to say we would be doomed. We securely replicate our data to [web server]. So in the event Phoenix NAP becomes a smoking crater, we would be back up and running in a short amount of time (minutes not hours).” Shon explained.
15 years ago I would have been right on the money. Multi-datacenter replication was so expensive that only the very top companies could afford it. Today, thanks to OpenStack, Rackspace, and most of all AWS, if your startup is taken out because your DC goes down, you’re not doing it right.
As it turns out, most of the security was there to prevent theft and non-destructive intrusion. Not destruction. Data and infrastructure are fairly easy to replicate today. Theft or worse, subtle infiltration is a much more significant risk for most. Again, replication and virtual hardware took much of the destruction risk away.
CC Bill, the emperor of porn payment processing, dedicates an entire floor of NAP to process transactions but they are no better off than the smaller startups housed there. Despite their massive scale and tenure in the digital space, they use a lot of the same data techniques that younger organizations use to mitigate destruction risk.
The Value of Data
Bits… they’re just a bunch of 0s and 1s. All theoretical, yet valuable. It’s a lot like how we value a person for their personality. It’s not the mass of the person that makes them valuable, it’s the synapses and electronic messages firing through the brain. The memory signature of a human being is quite like the data in this building. Really it’s just an extension of our own brain, isn’t it?
Perhaps that thought progression is what’s driving Trevor the astrophysicist, engineer, and CTO of HiringSolved to consider a possible future in neuroscience one day after he makes his buh-zillions.
Script Kiddie Danger
Not many people get to see this kind of thing. When I was a kid they took us to go see the factory where they made Coca Cola. Sure the assembly-line machinery was pretty cool, but nobody was carrying a taser.
They never take kids to things like data centers. Well, they never did in the 90s. There weren’t as many of them, in their defense. I have no idea where kids are field-tripping to these days, but if we really wanna trip them out we should take them to a data center.
On second thought, I’m not sure a group of rowdy kids could be trusted in a place like that. With things as fragile as they are, there isn’t much a kid couldn’t destroy.
Even with everything is locked away in black lockers like a goth Saved By the Bell.
Servers are just like your tower or laptop at home really, they need a power strip to plug into and an operating system to run their programming.
If a kid got in there and pressed the red button, they’d muck up the ongoing processes that the software company was performing. Right?
It does not work this way. Dennis the Menace could literally go into the datacenter, open the locker, plug in a Makita and start drilling holes in servers while our users happily went about their business.
Also, before Dennis could drill enough servers to make a difference, he would have been tazed in the head by one of the Kevlar-clad heroes.
If he turned everything off, yes, things would go down. HiringSolved would have to fire up the web server nodes until NAP security fixed the issue.
As a side note, HiringSolved’s entire cluster has in fact been completely powered off before. Not by a menacing kid, but by a clumsy ASU Computer Science intern and his stray elbow. Guess what happened? Nothing! HiringSolved powered back up to crawl sites and serve users like nothing happened.
We have a highly available redundant design. A datacenter being taken out is rare but if it happens we’re ready. If we weren’t, we would be morons.
Security Upon Security
After the security check and being processed, we walked through a glass door on the right that clicked to unlock and we went down a narrow hallway.
Halfway down the hallway there was a wall made of glass where we could see a bunch of tech security experts looking over their monitors right into our eyes as we passed.
They were seated on a series of graduated steps much like the bridge on a Starfleet C-class ship or some kind of secret nerd council or tribunal.
Their demeanor was very serious. A curly, long-haired ginger one I looked at gave me a stoic glare that sent chills through my soul.
I shuddered and kept walking, keeping my mouth shut so as not to set off some kind of security lip-reading software that detects stupid people.
When we got to the end of the hall we arrived at the most interesting part of the technology. The airlock room. Or whatever it was. It’s an airlock in the fashion that it was a room you entered and could not progress through until the doors on both sides had been locked.
Trouble in the Airlock
There was a guy in there fumbling with the communication device on the wall. He seemed flustered and I couldn’t hear what he was saying through the door but it didn’t look good.
All of a sudden a green gas poured through hidden vents in the ceiling and in flew two automated drones with mounted machine guns and quadcopter flight.
They pointed their lasers at him and started firing. The portly young man in the death trap started trying to fight them off immediately and started coughing as the green gas started melting his skin. The gas billowed through the room until it was fully opaque. We couldn’t see anything.
Then *boom* we saw his disfigured bullet-ridden face smack against the glass door. He slumped to the ground and a giant spatula lowered from the ceiling to scrape his body from the floor and into a vacuum opening that had appeared on the far wall.
He was swiftly brushed into the vacuum where him and all the green gas flowed into the abyss. A pipe came out of the floor in the middle of the room and sprayed a solution everywhere. Squeegees dropped out of the ceiling to clean up the remaining specs of human. It all ended with a chirpy *ding* sound and the glass door slid open.
“They just murdered somebody,” I said.
“Shhh,” Shon said.
Shon badged first and motioned for me to follow.
I began to walk in and a piercing horn sound filled the air. Red lights flashed at the door’s entrance.
Shon whipped around and grabbed the badge on my neck to put it on the sensor. “You have to badge in too!”
But it was too late. We hit the deck as bullets went flying all around us.
What actually happened:
The befuddled gentleman in the enclosure finished whining into the speaker. We heard another voice come through our speaker and say, “Okay. You two can badge in and get in there with him now.”
“Alright. Back up,” we heard the loudspeaker say. By that time, Ian had caught up with us. I looked at him for some reciprocation and said “Dude, this place is badass.” He nodded. It wasn’t his first time here, but he couldn’t deny the coolness either. (He tried).
“Yeah, your first time visiting one of these things is usually pretty intense. It gets less so when you’re running in and out to change hard drives all the time, but I have to admit it’s still pretty cool,” he said.
What I didn’t realize until after the trip is that the ‘airlock’ room we were in was actually measuring our weight down to a fraction of a gram and our weight while exiting would be compared to the original. Just another security measure to ward off theft.
That must be why the guards needed to know the brand and capacity of the drives we were bringing in as well as taking out. Hmm…
I exited the airlock totally ignorant of my weigh-in and entered the actual farm. The farm is a habitat for many a scurrying nerd. I spotted a silverback neckbeard, the duckbilled hipster, and even a water buffalo.
There are vents on the floors between each stack of servers. If left alone, servers can generate quite a bit of heat, actually. They’re kept to an appropriate temperature by a cycle of cool air that blows up from the floor on one side, then up and over the lockers to the other side where the hot air is vented out. So, if you get too cold, you can go to the hot side or vice versa.
I found this useful as I hadn’t dressed for either cold or hot weather. I was kind of in the middle (t-shirt with slacks). I found myself making the rounds once or twice.
Every part of the process was delicately and carefully handled. Removing a drive and replacing it is a serious undertaking for a business that relies on data. Each drive was handled like a newborn. It was clear that this server meant their livelihood. It was like defusing a bomb.
Ian added some nice-to-know stuff as I checked out the digs. Such as how the yellow housings that tracked along the tops of all the servers in the farm were the hamster highways where hamsters delivered written network codes from one side of the farm to the other.
Trevor explained how segments of each drive’s data was copied throughout the other drives, so if you lost one it was easy to recover from the other. With limited resources as a startup, this method serves very well and has prevented any major catastrophes at HiringSolved.
The guys joked about a marketing piece I wrote about our “thousands of servers” when in fact we have a much fewer number.
When it was time to go, I had soaked in the importance of data centers like this and the reason security is so important for them. When HiringSolved’s data does eventually need the capacity of thousands of servers, we’ll look back on this moment and think of how cute things were back then.
About the Author: Christopher Murray is a growth hacker at HiringSolved. He’s also a Gangplank Chandler community member and volunteer where he supplies a weekly newsletter; edits blog posts; and handles community promotions. He started with HiringSolved in November 2014, and has immersed himself in the knowledge of sourcing, HR, and Recruiting. Christopher has a background in freelance writing, hacking, and marketing. His duties at HiringSolved include site content growth, writing patents & press releases, conducting email blasts, and charting their trajectory in the new media landscape.