Stories from the life of an IT specialist on a ship that passed around Antarctica

Carles Pina-Estani can not be called a typical explorer of the Arctic. This native of sunny Barcelona works as a programmer for Mendeley . This technology company is owned by Elsevier Scientific Publishing House and is based in London. Until this year, he had never slept on a ship. But when he was invited to participate in a three-month expedition around the Antarctic, he gladly jumped at the chance.



It all happened pretty quickly. Pina-Estani’s partner, Jen Thomas, who previously worked with the British Antarctic Survey , worked as a data expert on a research trip led by the newly established Swiss Polar Institute . SPI brings together researchers working beyond the polar circle or in other extreme conditions, conveys information to the general public about their existence, and facilitates researchers' access to these areas. The tour was paid for by the Swedish billionaire and adventurer Frederick Paulsen Jr. - and he even went along with everyone . This is not for you in the office to deal with technical support.







An IT specialist needed to be on board (in addition to two electronics and maintenance engineers). Fortunately, Thomas was familiar. And suddenly Pina-Estani discovered that he was aboard the Akademik Treshnikov , a Russian research vessel, embarking on an ambitious route known as the Antarctic Loop , through Cape Town in South Africa, Hobart in Tasmania and Punta Arenas in Chile.





The route "Antarctic loop"



During the brief stay of our hero in the role of a marine IT specialist, Akademik Treshnikov played with bad weather, below-average food, and telecommunications equipment that did not want to work properly. And although in any such journey we should expect difficulties, Pina-Estani faced a set of problems from the very beginning due to the too short time allotted for the preparation of the equipment. As he himself commented optimistically, “I had more opportunities to solve any problems.”







“Now I’m 35, and I’ve been working with computers for more than 20 years — in the afternoon, in the evenings, on weekends, and in the form of personal projects,” he told us. - And in this campaign I used everything that I know, and even the rare knowledge that I considered useless was useful to me. In the Antarctic, everything becomes useful. ”



When Pina-Estani's unusual work time came to an end this summer, and he returned to London at the end of July, he found time to talk with our editorial board and to try our inner nerd networker with stories about the life of IT in the open seas. At least from his story, technical support in the most extreme parts of the world is not too different from technical support in more familiar conditions - except for the lack of reliable communication and the ability to buy additional parts from Amazon or a local store. But the need for help is constantly present, emotions can go off-scale, and requirements vary from simple work with an email server to such things that no programming course can teach you.

Winch

One evening, about a month after the start of the campaign, one researcher cheerfully addressed Pina-Estani with what he called the "new ordeal." Before that, he mainly worked with the usual equipment that did not take him out of the comfort zone: computers, routers, hard drives and Raspberry Pi.



But this application for support was in a completely different area. The massive winch, a mechanism consisting of a cable and a crank arm, began to fail.



Prior to that, Pina-Estani did not even know what “winch” was. And this winch was responsible for launching the only CTD-profilograph on board that collected and analyzed the water. Typically, CTD was lowered to a depth of 1500 m, and this device was critical for the majority of 22 teams of researchers present on board.







It turned out that the problem lay in the software. The error interfered with the winch function by smoothly lowering the long cable into the water. Pina-Estani tried to debug, but the equipment manufacturer told him that it was impossible to remotely enter new parameters into the computer of the CTD winch. And since “Akademik Trshniki” was in the ocean, this caused minor difficulties.



The decision required the use of hacking skills and a willingness to challenge the elements:



“The temperature was from zero to -2 degrees, ocean spray was everywhere, the boat was rocking, and my hands were freezing. People asked me what I was doing there with the computer - the CTD was connected via a very short network cable. So you had to work outside, ”he tells me. “I got access to the winch computer that was running Windows CE from my Linux machine.” I saw the IP address on the boot screen using nmap, I discovered that it has a remote desktop server. I was very happy when, after pressing Enter, it turned out that I was able to change the parameters! ”







Pina-Estani wasn’t happy for long - it didn’t solve the problem. After the manufacturer reluctantly allowed him to reinstall the program, he had to wait until they moored in Hobart for restocking to download the firmware via WiFi in the hotel.



“We reinstalled everything, and the computer still didn't boot,” he says. - It was one of the worst moments of the whole expedition from the point of view of IT. The problem was that CTD was one of the most necessary items among scientific equipment - but this is a big winch, and I did not have any knowledge in this area. ”



Fortunately, this story has a happy ending. After reinstalling the update in another way, the winch ceased to be capricious. Soon, all scientists successfully collected their water samples, and Pina-E-Estani was finally able to carry his computer back to heat.

Network hack

Another memorable application for support came from a scientist about the apparatus, which she used to measure the reflectivity of the sea. She needed to remove data from the device, but she could do it only through a connection with a remote access point.



“Well, I say - yes, there are no problems, but where is the router? - he recalls. “He was not on the ship — it turned out that he was standing somewhere in Australia at all.”







The most distant points of this loop, which ran from Cape Town to Punta Arenas, can be 10,000 km from Australia, and at that moment, according to Pina-Estani, they were about 3,000 km from South Africa. But this did not stop him; he used his laptop as a remote access point, connecting to the equipment's FTP server to access the data. “But it was not a very convenient solution, because every time they needed data, they needed to get to me and my computer.”



Instead, Pina-Estani used a simple but preferred device for network hackers - a smartphone. “I figured out how to hack the network with an Android phone,” he explains. - It can raise the access point, and even without an external signal, the device can connect their laptop with equipment through the phone. In this way, scientists could get their data when they wanted, without my participation. ”

Mail system

At the time of dial-up, you could spend a lot of time watching the download process of a single email or page. What was annoying then, today's people accustomed to broadband access, are just crazy. But when a lot of people try to use an unreliable Internet connection via satellite to send and receive mail, do not wait. This can quickly raise social tensions.



“I've never seen so many people hitting their computers,” Pina-Estani recalls. - It was painful to see it. I can not stand the spectacle of a scientist standing in front of his computer in anticipation of the next step. My heart just bleeds. ”



With the exception of general disappointment, the accumulating queue of emails meant that people could not access some important permits and other documents related to the expedition. At some point, about 100 MB of letters got stuck in the system, so Ping-e-Estani began extracting them.







“It would be easier to return to the port and get them than to get them in the middle of the sea through Outlook,” he explains.



Of course, such a route change was impossible. Instead, he used the access to the remote server to download and archive all mail, and then sent these files to the ship using the rsync program, which copes well with unstable connections. He also wrote a script that controls the resumption of the interrupted download from the same place where it was interrupted.







“So I left this program running for 8-9 hours, and then I opened this big file in Thunderbird,” he said. “After that I was able to get all the necessary letters, including the permissions.”



But such a solution to the problem on the knee did not solve the main one - the ship received letters, but the expedition still did not have a reliable communication system.



“I knew how to fix it, but for this I needed several hours of reliable Internet,” said Pina-Estani. - During our three-day break after the first transition, Jen and I went to the hotel, where I [via the Internet] set up a new domain for the website and a webmail server, and then created each user. He became very popular. Everywhere, wherever I was on the ship, people used my webmail. Watching this was amazing. ”







In order for everything to continue to work without failures, Pina-i-Estoni needed to limit the size of sent emails to 200 Kb, which means another solution was needed to transfer larger files.



“I developed a system in which letters swung in pieces,” he recalls. - It was, in fact, a hack, because the mail protocols do not allow to separate the letters - they either download them or not. I bypassed it, and made it so that if the download was cut off by 20%, it resumed anew from the same place. ”







He also set up a queue system to work with a slow connection. Thanks to her, even if it took someone 5-10 minutes to receive mail, users immediately received notifications and knew that everything was working.



“I was very worried about the fairness of the system, so the letters were downloaded in the order received,” he said.

Ferry box

The latest Pina-Estany story happened in an area that any IT-specialist one way or another faces: data management. The task of managing the huge amount of data collected by the expedition in real time was one of the most difficult and ongoing tasks to which he applied his computer skills. The best example is Ferry Box, a Linux-based machine that is constantly used to measure salinity and temperature on the surface of the water.







Since access to each subsequent data set appeared every few days, it was very difficult for scientists to plan ahead. For quite a long time they could remain in the dark about important events, for example, the transition from one water to another, with excellent characteristics.







“To solve this problem, I figured out how to upload data to a website that scientists have had real-time access to,” says Pina-Estani. “In this way, they could make decisions about where to stop and, for example, take samples.”







This functionality allowed researchers to move from reactive to proactive data collection activities. It is easy to see how this affects science itself. And after a few such stories from Pina-Estani, it's easy to see how even the most extreme scientific research depends on modest IT workers.