My HiringSolved Away Mission to Phoenix NAP

Getting Going

My trip to the NAP Datacenter was an unexpected journey. I wasn’t supposed to go, actually. Trevor had invited me to go months ago and I couldn’t make it that time, but when I heard HiringSolved was going on another field trip, I asked if I could join.

Shon was enthused about the outing for a moment, but then realized I couldn’t go. “You need two forms of ID, so it’ll have to be next time.”

“It just so happens I’m carrying two!” I was back in on the away mission.

Arrival

 

Photos of Phoenix NAP facility.

Photos of Phoenix NAP facility.

We parked at a building that I had passed many times before. A very nondescript and boring building from the outside. I had driven past the corporate art and boring block architecture many times on my way to Arizona State University back in my college days.

We exited at the boring building and walked up to a tinted glass door–a common accent for the hot Arizona sun. Nothing impressive so far.

“This is by design,” Shon told me. “Notice how there are no logos or labels of any kind on this building”

Now that you mention it, this building is pretty nondescript… I was thinking. Has anyone even moved in? I see the cars around and everything but surely they’re just here for the Waffle House across the street.

Nope. This is NAP. And despite the cozy name, the security here never sleeps. We got up to the nondescript door of the nondescript building and Shon hit the nondescript buzzer on the nondescript pole in the nondescript cement.

A nondescript voice answered: “Bed—fft ba—ft and be—-ndfssf”

I didn’t really hear it.

“Shon Burton, HiringSolved,” Shon said into the speaker.

Click. The doors were unlocked. We walked over and Shon was saying, “See? There’s nothing signifying what this place is. Again, that’s by design. I’m surprised they even have numbers on the door. Usually they try to hide even the address at these things.”

Clearly this wasn’t Shon’s first rodeo.

Entering the Center

I sheepishly walked in with my hands firmly at my sides. Then clasped behind me. Then crossed in front of me. I didn’t know where the hell to put my hands.

The two security guards were wearing kevlar vests and a bulletproof pane of glass separated them from us.

One of them started grilling Shon on what hard drives he was taking in.

“Brand X,” Shon said.

“How many gigs?”

“X gigs.”

“How many drives?”

“Just two.”

“And your guest?”

“This is Christopher Murray. He’s new and we’re giving him a tour.”

They looked at me and said, “Two forms of ID please.”

I fumbled around in my Beatles wallet and pulled out my green card and Arizona driver’s license.

“Wait, you have a green card?” Shon said.

This was the first time I explained my Irish heritage and how even after living in the USA for 26 years (since I was 2 years old), I am still an Irish citizen. I saved some of the hairier details for later.

The mission that day was to infiltrate the data center, remove the failing drives, and replace them with newer, more reliable ones–some real cyber-punk shit!

The guards handed Shon and I a pair of badges to get us into the data center.

Behind them, I noticed a grip-ton of TVs set up to surveil the surrounding campus as well as the hallways of the complex. They probably had them on display like that to remind people they’re being watched.

The place was a fortress.

Data Meditations, Scorpoyotes, and Porn

I started to think about how malleable data is.

The Pirate Bay, for instance, has node accounts at data centers all over the world that activate in the event that one is taken down. It doesn’t matter if their building blows up because there will always be another backup of the data somewhere else.

For Gangplank startups like HiringSolved, however, this building blowing up would be quite a different situation.

If this building exploded, imploded, burned down, fell into the swamp, was raided by scorpions, coyotes, scorpion-covered coyotes, or giant scorpion-coyote hybrids that stabbed all the security with their scorp tails while half of them ran high powered magnets through the complex to wipe everything; all the startups would be doomed.

Or at least you’d think so.

When I brought forth my Scorpoyote concerns Shon showed me how I shouldn’t be so concerned.

“Technically, at least for us, it’s inaccurate to say we would be doomed. We securely replicate our data to [web server]. So in the event Phoenix NAP becomes a smoking crater, we would be back up and running in a short amount of time (minutes not hours).” Shon explained.

15 years ago I would have been right on the money. Multi-datacenter replication was so expensive that only the very top companies could afford it. Today, thanks to OpenStack, Rackspace, and most of all AWS, if your startup is taken out because your DC goes down, you’re not doing it right.

As it turns out, most of the security was there to prevent theft and non-destructive intrusion. Not destruction. Data and infrastructure are fairly easy to replicate today. Theft or worse, subtle infiltration is a much more significant risk for most. Again, replication and virtual hardware took much of the destruction risk away.

CC Bill, the emperor of porn payment processing, dedicates an entire floor of NAP to process transactions but they are no better off than the smaller startups housed there. Despite their massive scale and tenure in the digital space, they use a lot of the same data techniques that younger organizations use to mitigate destruction risk.

The Value of Data

Bits… they’re just a bunch of 0s and 1s. All theoretical, yet valuable. It’s a lot like how we value a person for their personality. It’s not the mass of the person that makes them valuable, it’s the synapses and electronic messages firing through the brain. The memory signature of a human being is quite like the data in this building. Really it’s just an extension of our own brain, isn’t it?

Perhaps that thought progression is what’s driving Trevor the astrophysicist, engineer, and CTO of HiringSolved to consider a possible future in neuroscience one day after he makes his buh-zillions.

 

Script Kiddie Danger

scriptkiddie

Not many people get to see this kind of thing. When I was a kid they took us to go see the factory where they made Coca Cola. Sure the assembly-line machinery was pretty cool, but nobody was carrying a taser.

They never take kids to things like data centers. Well, they never did in the 90s. There weren’t as many of them, in their defense. I have no idea where kids are field-tripping to these days, but if we really wanna trip them out we should take them to a data center.

On second thought, I’m not sure a group of rowdy kids could be trusted in a place like that. With things as fragile as they are, there isn’t much a kid couldn’t destroy.

Even with everything is locked away in black lockers like a goth Saved By the Bell.

Servers are just like your tower or laptop at home really, they need a power strip to plug into and an operating system to run their programming.

If a kid got in there and pressed the red button, they’d muck up the ongoing processes that the software company was performing. Right?

Wrong.

It does not work this way. Dennis the Menace could literally go into the datacenter, open the locker, plug in a Makita and start drilling holes in servers while our users happily went about their business.

Also, before Dennis could drill enough servers to make a difference, he would have been tazed in the head by one of the Kevlar-clad heroes.

If he turned everything off, yes, things would go down. HiringSolved would have to fire up the web server nodes until NAP security fixed the issue.

As a side note, HiringSolved’s entire cluster has in fact been completely powered off before. Not by a menacing kid, but by a clumsy ASU Computer Science intern and his stray elbow. Guess what happened? Nothing! HiringSolved powered back up to crawl sites and serve users like nothing happened.

We have a highly available redundant design. A datacenter being taken out is rare but if it happens we’re ready. If we weren’t, we would be morons.

Security Upon Security

After the security check and being processed, we walked through a glass door on the right that clicked to unlock and we went down a narrow hallway.

Halfway down the hallway there was a wall made of glass where we could see a bunch of tech security experts looking over their monitors right into our eyes as we passed.

They were seated on a series of graduated steps much like the bridge on a Starfleet C-class ship or some kind of secret nerd council or tribunal.

Their demeanor was very serious. A curly, long-haired ginger one I looked at gave me a stoic glare that sent chills through my soul.

I shuddered and kept walking, keeping my mouth shut so as not to set off some kind of security lip-reading software that detects stupid people.

When we got to the end of the hall we arrived at the most interesting part of the technology. The airlock room. Or whatever it was. It’s an airlock in the fashion that it was a room you entered and could not progress through until the doors on both sides had been locked.

Trouble in the Airlock
Airlock-Testing

There was a guy in there fumbling with the communication device on the wall. He seemed flustered and I couldn’t hear what he was saying through the door but it didn’t look good.

All of a sudden a green gas poured through hidden vents in the ceiling and in flew two automated drones with mounted machine guns and quadcopter flight.

They pointed their lasers at him and started firing. The portly young man in the death trap started trying to fight them off immediately and started coughing as the green gas started melting his skin. The gas billowed through the room until it was fully opaque. We couldn’t see anything.

Then *boom* we saw his disfigured bullet-ridden face smack against the glass door. He slumped to the ground and a giant spatula lowered from the ceiling to scrape his body from the floor and into a vacuum opening that had appeared on the far wall.

He was swiftly brushed into the vacuum where him and all the green gas flowed into the abyss. A pipe came out of the floor in the middle of the room and sprayed a solution everywhere. Squeegees dropped out of the ceiling to clean up the remaining specs of human. It all ended with a chirpy *ding* sound and the glass door slid open.

“They just murdered somebody,” I said.

“Shhh,” Shon said.

Shon badged first and motioned for me to follow.

I began to walk in and a piercing horn sound filled the air. Red lights flashed at the door’s entrance.

Shon whipped around and grabbed the badge on my neck to put it on the sensor. “You have to badge in too!”

But it was too late. We hit the deck as bullets went flying all around us.

What actually happened:

The befuddled gentleman in the enclosure finished whining into the speaker. We heard another voice come through our speaker and say, “Okay. You two can badge in and get in there with him now.”

“Alright. Back up,” we heard the loudspeaker say. By that time, Ian had caught up with us. I looked at him for some reciprocation and said “Dude, this place is badass.”  He nodded. It wasn’t his first time here, but he couldn’t deny the coolness either. (He tried).

“Yeah, your first time visiting one of these things is usually pretty intense. It gets less so when you’re running in and out to change hard drives all the time, but I have to admit it’s still pretty cool,” he said.

What I didn’t realize until after the trip is that the ‘airlock’ room we were in was actually measuring our weight down to a fraction of a gram and our weight while exiting would be compared to the original. Just another security measure to ward off theft.

That must be why the guards needed to know the brand and capacity of the drives we were bringing in as well as taking out. Hmm…

I exited the airlock totally ignorant of my weigh-in and entered the actual farm. The farm is a habitat for many a scurrying nerd. I spotted a silverback neckbeard, the duckbilled hipster, and even a water buffalo.

There are vents on the floors between each stack of servers. If left alone, servers can generate quite a bit of heat, actually. They’re kept to an appropriate temperature by a cycle of cool air that blows up from the floor on one side, then up and over the lockers to the other side where the hot air is vented out. So, if you get too cold, you can go to the hot side or vice versa.

I found this useful as I hadn’t dressed for either cold or hot weather. I was kind of in the middle (t-shirt with slacks). I found myself making the rounds once or twice.

Delicate Handling

Every part of the process was delicately and carefully handled. Removing a drive and replacing it is a serious undertaking for a business that relies on data. Each drive was handled like a newborn. It was clear that this server meant their livelihood. It was like defusing a bomb.

Ian added some nice-to-know stuff as I checked out the digs. Such as how the yellow housings that tracked along the tops of all the servers in the farm were the hamster highways where hamsters delivered written network codes from one side of the farm to the other.

Trevor explained how segments of each drive’s data was copied throughout the other drives, so if you lost one it was easy to recover from the other. With limited resources as a startup, this method serves very well and has prevented any major catastrophes at HiringSolved.

The guys joked about a marketing piece I wrote about our “thousands of servers” when in fact we have a much fewer number.

When it was time to go, I had soaked in the importance of data centers like this and the reason security is so important for them. When HiringSolved’s data does eventually need the capacity of thousands of servers, we’ll look back on this moment and think of how cute things were back then.

headshotAbout the Author: Christopher Murray is a growth hacker at HiringSolved. He’s also a Gangplank Chandler community member and volunteer where he supplies a weekly newsletter; edits blog posts; and handles community promotions. He started with HiringSolved in November 2014, and has immersed himself in the knowledge of sourcing, HR, and Recruiting. Christopher has a background in freelance writing, hacking, and marketing. His duties at HiringSolved include site content growth, writing patents & press releases, conducting email blasts, and charting their trajectory in the new media landscape.

Chandler Science Café at Hack Night, “Collective Genius for a Better World”

Please join us Wednesday, April 1, 2015, 7pm, at Gangplank, 260 S. Arizona Ave., Chandler, AZ 85225, for the Chandler Science Café at Hack Night, “Collective Genius for a Better World,” with Dennis Bonilla, NASA Contractor & Senior Digital Strategist – Valador, Inc.

Dennis Bonilla, Design Manager. Senior Digital Strategist. Community Manager. NASA Contractor.

Dennis has a decade of experience communicating complex concepts using video games, infographics, and interactive platforms to inform and empower communities. Dennis has created federal websites, designed visualization software for NASA, and worked on documents presented to the U.S. President. Dennis is also currently the community manager for http://LAUNCH.org, a sustainability program developed by NASA, Nike, the U.S. Department of State, and USAID.

Dennis will be discussing his work with the NASA Open Innovation Team, the NASA Office of the Chief Technologist, and the NASA Office of Human Capitol.

Science Cafés are live—and lively—events that take place in casual settings, are open to everyone, and feature an engaging conversation with a scientist about a particular topic. Chandler Science Cafe, first Wednesday of the month, 7pm, at Gangplank Chandler is presented in association with the Arizona SciTech Festival and Gangplank Labs Initiative.

Free, open to the public, all welcome.

My Week at GPAvondale

Guest post by Katie Hurst, Gangplank HQ Operations Director

I feel in love with Gangplank Avondale when I first saw the building. Hidden spaces under the stairs, a real bank vault…what’s not to love? The possibilities for creating a fun, open work environment were endless.

But it wasn’t until I returned nearly five months after the opening that I saw those opportunities being realized.

Last week, I decided to tag along with Gangplank Chandler Anchor Customer Systems, Inc., as they visited our west-side cousins for a few days. So we trekked our stuff (and some new IKEA desks!) the 42 miles from Downtown Chandler and set-up shop on the bottom floor of GPA. The drive isn’t that bad when you’ve got a driving buddy, by the way (Thanks Ed & Tim!).

There were so many things that blew me away about the West Valley. To begin, let’s talk about the food. You wouldn’t know it from the highway, but Avondale is home to a ton of great local restaurants, tucked away in the corners of strip malls. We enjoyed bento boxes at Goodyear’s Tomo Japanese Cuisine & Sushi and chicken shawarma at the Pita Kitchen. On Friday, the brownbag speaker was from local hot spot Caballero Grill and graciously provided amazing pork sandwiches for attendees. Of course, no trip to Gangplank Avondale would complete without a happy hour run to the Sonic next door for half priced slushies (Cherry Limeade ftw).

Even better than the food variety is the diverse array of projects happening in the West Valley. On Thursday night, GPAvondale hosted its first ‘After Hours’ event, inviting the community to hang out and work from 6:00-10:00pm. Around 25 people showed up to share what they were working on and collaborate. One group included Eric Torres and his game, Iconica: The World of Rynaga. From their website – “Set in the fictional World of Rynaga, Iconica blends turn-based strategy with a twist of chance for a unique card game experience that’s easy to learn, but a challenge to master.” The group hopes to meet monthly at GPAvondale to play Iconica and other RPGs, as well as work out any kinks. Other projects included an traffic stoplight Arduino project, a web series discussion and building a display case for a local invention.

The best part about GPAvondale? The people. Gangplank locations attract good people who want to engage with their local community and each other. I had a lot of fun (and laughs) talking with people about their projects, what ideas they have for Gangplank and things they’d like to learn more about. Of course, any group of folks that can put up with my constant running around and random dancing must be a pretty accepting bunch =)

Check out their next brownbag this Friday, ‘Storyrise: Stories of the Journey‘ at noon.

 

 

 

Gangplank Avondale Opens!

On Monday, March 5th, 2012 Gangplank Avondale opened it’s doors to innovators, techs, creatives, entrepreneurs, and the general public. The space is typically open from around 8:30am to 5:30pm. Gangplank Avondale provides an infrastructure (that is both social and physical) for creative collaboration, disruption, and the building of vibrant businesses, communities, and economies.

The space is now open for drop-in collaborative working, although March 20th from 4pm- 8pm will mark the Grand Opening event with remarks from the Mayor of Avondale, a founder of Gangplank, and the Gangplank Avondale Directors of Operations. Catering will be provided by West Valley restaurant, Caballero Grill. Take a tour of the space and get to know what Gangplank Avondale is all about!

Our model is based on human-powered collaborative working and the trading of social capital. We work in an open space that has very few physical barriers so that we can easily communicate, share, converse, and spread our thoughts and ideas with one another.

Join us on Fridays at noon for our brownbag presentations to hear leaders from the community fill us in on changes in their industry, best practices, new business models, changing laws, and insights into the successes and challenges of their careers. Open your mind and your mouth and let’s get conversations started!

Follow us on Twitter @gpavondale or on Facebook at http://www.facebook.com/GangplankAvondale.

Gangplank Avondale Demo

We are closing in on the opening date for our space on Central. Join us in the festivities at the Grand Opening with the Mayor of Avondale on March 20th from 4pm-8pm. The space will be open and free to the public and we welcome anyone to come in and see what collaborative working is all about!

Here are some photos of the demo work going down as we speak: