December 10, 2007 () From the start, Henry Malmgren was determined to get to the South Pole. After graduating from Texas Tech University in 1998 with a degree in MIS he applied for a job in the Antarctic every year before NSF contractor Raytheon finally hired him as a network engineer in 2001. Since then he has alternated between the Denver headquarters and the Amundsen-Scott South Pole Station, spending two summers and two winters there before finally working his way up to IT manager. Staying over is a commitment: Once the winter starts, there's no way to get in and out of the base until summer begins eight to nine months later. "I thought I would just do this for a single season, but somehow it always seemed too easy to keep coming back," he says.
Ok, I have to ask: How's the weather down there?
Right now it's pretty nice. We're at nice -64 degrees Fahrenheit with only about a 6.3 knot wind so it's what we would consider a really good day.
Whatever possessed you to work in Antarctica?
I had never traveled out of the U.S. at all until after I'd graduated from college. I had dated a girl who was an exchange student to Europe, and after hearing all of her stories, I knew I wanted to get a job out of the U.S. Somehow, the Antarctic job popped up on a job search, and I latched onto it as the perfect ticket to be able to travel for a while.
Is there really an official South Pole?
Absolutely. It looks like a barber pole. It's right outside our front door.
If you're on an ice sheet that moves about 30 feet a year, how do you know that the pole is really in the right spot?
We have a ceremony every New Year's day where we relocate the actual pole marker. Some years the United States Geologic Survey comes out and does an actual measurement with the sun, and other times we just use a surveyor-quality GPS to get a pretty close approximation. We actually put up a new marker, so you can see the line of previous markers stretching out away from the station.
The pole gets closer to the station every year. In about 20 years, it'll be right under our power plant.
What is your role at the station?My domain includes everything from the IT superstructure to the satellites to the telephones systems and handheld radios. Anything telecom or computer related I'm ultimately responsible for. I have a staff of about seven supporting about 250 to 270 people during the summer season, which is where we are right now. During the winter, which runs from about mid-February to mid-October, we drop down to just four [staff] supporting 60 to 70 people.
What is a typical work day like for you?
We work nine hours a day, minimum, six days a week. I spend the first couple of hours answering e-mails from the Denver folks. [Then] I go out and talk to the scientists to see if they're getting what they need and if there are any problems we can solve. I love the fact that, although I'm on the management side I get to stick my hands in the technology a whole lot. I very much feel like a player coach because I have five years' worth of experience with these systems while most of my employees are here for the first time.
What takes up most of your time?
Information security has really come to the forefront in our priorities. Right now keeping up with security vulnerabilities and patches and things like that is taking a good third of our time. That's a change from even two years ago.
What is your data center like?
We have a brand new station that we just completed [in 2005] and we have a full-on data center with raised floors and everything you would have anywhere else in the world. We have about 30 servers. We also have what we call the RF [radio frequency] building, which is about a kilometer away from the main station, and we have a backup emergency data center out there where we keep extra file servers and a SAN. That's where the satellite dishes are. If anything happens to the primary data center we can switch our operations out to there.
Do you work with an IT counterpart back in the States?
There's a whole team I can call on back in Denver and we depend on them quite a bit. If we have problems and we can't solve them down here we rely on our Denver staff to provide trouble shooting.
What's a little known fact about life at the South Pole Station?
People would be surprised to know how well we eat down here. We have a little greenhouse. We have enough greenhouse production with hydroponics to get us a salad every couple of days.
Is it tough to hire staff to work there?
We usually have a high number of applicants -- it took me four years of applying before I was hired. But there are some times when we just don't have enough candidates. It fluctuates with the state of the economy. Right now I'm in a huge competition for satellite communications guys with contractors in Iraq and Afghanistan. Those guys are able to pay so much more than we can. Of course the advantage down here is that nobody is shooting at you.
What technical challenges do you face?
Our biggest challenge is bandwidth. We only have it only 12 hours a day at anywhere from T-1 (1.54 Mbit/sec) to 3 Mbit/sec speeds. We also have a transponder that we can use to send 60 Mbit/sec unidirectional from the pole to the real world. We use that to upload scientific data. Our record was 94Gbytes out in one day.
We have three different satellites we use to provide our Internet. All of those are pretty ancient. We have a weather satellite, an old maritime communications satellite and an old NASA satellite, the first one that was launched back in 1981. The others were launched in 1976 or 1977.
Basically we're scavenging whatever we can find and we can only see each satellite for 3 to 4 hours a day. Other than that we're almost a typical network. We use Cisco gear, we've got land lines to all of the bedrooms, we've got fiber optic distributed throughout the building so if fiber to the desktop ever becomes a reality the building is prewired for that. So we are trying to be as future proof as possible.
What is the most interesting project you've worked on lately?
In the past year we put up a really cool system where we're using the Iridium satellite network. We have 12 modems mulitiplexed together and have a total of 28.8K connectivity 24 x 7. Nobody thought it would work. Nobody ever thought we would have 24 x 7 connectivity at the South Pole. Now that's our last resort. When our broadband satellites are down we switch to the Iridium system automatically.
What happens if the satellite link goes down on your end? Do you draw straws to see who goes out to the RF building to wiggle the antenna?
Everyone here has had the experience of walking out to the remote data center in -100 degree temperatures when it's pitch black outside. That's part of the adventure of coming down here -- you can have these extreme situations that you're faced with. If you can handle swapping a router out at -100 degrees here then you can handle it anywhere.
What scientific projects do you support? There are several really big science projects here and they all generate huge volumes of data. Our 800 pound gorilla right now is the South Pole Telescope, a 10-meter in diameter radio telescope that scans the cosmic microwave background. They're looking for things like dark matter, and it generates tons of data that they want to send back to the states as quickly as possible.
If we didn't have [satellite broadband] they would have to store the data for the nine-month winter. This way they can see the results much quicker. They can also analyze any issues with the telescope and correct them while the winter observing season is going on, rather than having to wait a full year.
How do you support the researchers and scientists?
Generally they provide their own equipment and we provide the back end. But when something breaks we do step forward and help them get it repaired.
Many scientists -- and I can't say that I blame them -- don't want to trust their data to anyone else. Our support is really just providing the communications they need, providing the technical expertise.
Do the conditions affect reliability and uptime at all?
It is incredibly dry [so] static electricity is a huge problem for us. We lose more laptops and hard drives to static electricity than anything else. Our biggest failures are things like power supplies and hard drives. We're at an altitude of 12,000 feet and with the thin air here, cooling fans don't push a lot of air. Anything heat related tends to need a lot of extra TLC.
Hard drives also have a problem with the high altitude. Most hard drive heads float on a cushion of air above the platter. We have fewer air molecules for hard drives to float on so we have more hard drive crashes than anywhere else.
What happens when something breaks?
Getting service is a little tough. We try to maintain at least a year's worth of spare parts.
What's the most fun --and least fun -- part of your job?
The most fun is working with the scientists. Everybody down here -- from the dishwasher to the scientists to the construction guys -- they all have interesting background stories. Nobody here is your average person.
The worst part is that I've done two full years down here and being away from your friends and family, especially for the holidays, can be really tough sometimes.
What unique management issues do you face?
We have to order and plan and do everything a year and a half in advance to get supplies on time. We get supplies by airplane, but those come through McMurdough station [and] it is supplied once a year by a cargo vessel.
It's also a physically demanding job. A lot of people who come down here immediately get sick from the altitude.
Other than the cold and the environment and the lack of fresh food and vegetables, sometimes it almost feels like I could be somewhere in the real world. But then I look out the window and see the South Pole in my front yard and realize that I have one of the best jobs in the world.
In many data centers in the U.S. heat density is becoming a problem. Surely that's not an issue for you?
You would think that at the South Pole cooling wouldn't be a problem but with the amount of heat we generate [in the data center] getting rid of it actually can be quite an issue. We try to pipe some that heat to other parts of the building to recover it. The data center in the old station just had a hole cut in the wall with a fan [to the outside] to cool the systems. Sometimes you'd be sitting there in a parka trying to get something done.
How do you address disaster recovery?
That's a big deal. Fire is a huge danger in the Antarctic because it is so dry and because liquid water is such a hot commodity. We try to make sure that all of our data is backed up independently and we run as many systems as possible in parallel in the two locations.
What happens if there is a fire?
We have a wet sprinkler system. That's pretty pioneering for the South Pole. The building is design to be modular and is broken into sections. If the main station ever goes out there's a survival pod with emergency communications, an emergency generator and an emergency kitchen that we can retreat to if necessary. And we do have a backup data center about a kilometer away from the station that allows the data to survive.
In the summer, the plan is if there is a fire we'll get people out of here. In the winter, we have to survive for four or five months, depending on how long it takes to get a plane in.
Do you work with any promising new technologies?
This is not the place to deploy emerging technologies. When you're on the cutting edge you require a lot of support and down here support is hard to get. It's just not the place to do it.
What have you learned from your time at South Pole Station that might help you in future endeavors?
If nothing else I would say diplomacy. Trying to work with a customer base that's everything from PhD scientists to a guy that could be an iron worker who has never been here before and just wants to know how to turned on a computer and e-mail his family. It's such an interesting experience to learn how to educate different kinds of people.
What's the most outrageous experience you had?
We have this tradition called the 300 Club. When the temperature drops below -100 we hike the sauna up to 200 degrees and stay in there as long as we can stand it. Then we run outside, naked, around the geographic pole and back inside so we get that total 300-degree change in temperature. That happens every year and it's absolutely amazing. Just the feel of that cold on your skin is like nothing else. People always wonder if you can feel the difference between 60 below and 100 below and the answer is absolutely.
Doesn't your skin freeze in those temperatures?
That's why you spend as much time as possible in the sauna. The trick is to pace yourself. You can't run because if you do you're going to get frost bite in your lungs. But you don't want to walk too slowly or you will lose all of your body heat and get frost bite in various sensitive spots. There's definitely a happy medium of a fast walk.
But even at that when everyone comes back inside, with all of the hacking and coughing going on the place sounds like a tuberculosis ward for the next couple of days.
With all the talk about global warming these days I have to ask: Is your real estate down there shrinking?
There is definite satellite photograph evidence about shrinking of the pack ice, and an acceleration of glacier flow rates leading to the ocean. A lot more studying needs to be done in the field to really quantify the rates.
When I went to the official South Pole web site and viewed the station through the Web cam, I noticed that the camera was pointing at a downward angle such that I couldn't see the top of the building. Was that just out of alignment?
No. We had to do that because at the beginning of the summer season the sun, instead of going up and over the horizon, basically circles the horizon at a certain elevation. In the early part of the year the sun is so low that if you point the camera straight at the building the sun would burn out the iris in the camera. Now that the sun is higher we've raised it so you might get a better view.
Time zones are usually determined by longitudinal distance from Greenwich. Since all longitudes converge at the pole, how do you decide what time it is there?
We could pick any time. It would be most convenient for us to go by Denver time, since that's where our headquarters is, but we use New Zealand standard time. All of our aid flights come from New Zealand and from a logistical standpoint it is much simpler to coordinate their flights with ours.
Describe your most interesting experience at the South Pole.
My first year down here we had a guy that ended up tearing a tendon in his knee. It wasn't a life threatening injury. but if we didn't repair it surgically he would not have been able to walk normally for the rest of his life. We ended up using this primitive videoconferencing system and we did remote telesurgery down here.
We had an orthopedic surgeon in the states watching and telling our doctor how to repair the tendon. The satellites weren't nearly as stable as they are now. We had several dropouts during the surgery and our guys were scrambling around at night to get the signal restored. We didn't want to leave this poor guy on the table with no doctor to guide us on what to do. But we restored the signal and [the doctor] finished successfully.
Do all communications go through the satellites?
Absolutely. The ice sheet that the station is on moves about 30 feet a year. We're 600 mile from the nearest base. Trying to lay cable on a moving target like that would be problematic.
What happens when those satellites finally stop working?
We expect to start using [other] satellites over the next few years because these are beyond their design lifetimes. Things are going to get worse for us before they get better, with a lot less bidrectional comminications. NSF is looking to lease time on some of the newer satellites. That will cost somewhere around $130 a minute.
Data centers consume a lot of power. Is power an issue?
We have the capability to produce about a megawatt of power, but when we do that it really stretches our fuel supplies. It's tough to keep generators running down here. They don't tend to like the altitude and we do have brownouts on occasion.
The limiting factor to the amount of power we can produce is the amount of fuel we can store. Every bit of fuel has to be flown in here, and there's only so much we can do in a four-month summer season of flights. Once the station is shut up for the winter there are no flights for the next eight to nine months.
Do you have backup power for the data center?
We don't have a backup data center generator but we do have backup UPS's. There is a single generator [for the entire facility] that is running at any one time but there are three here. One is running at all times, one is on hot standby, and one can be down for maintenance. If one generator fails, another can pick up within five or six seconds.
If we lose our generators, that's our heat. It wouldn't take more than four or five hours for the building to get really chilly.
How long will you keep coming down there?
Every time I come down here I feel lucky. Every year is different and I have a hard time imagining doing anything else. But my girlfriend is trying to convince me to do something closer to home. We're having ongoing discussions about that.
See a photo gallery of Malmgren at the South Pole.
More News and Discussion:
