As the founder of Simple Analytics, I always remembered the importance of trust and transparency for our customers. We are responsible for them so that they can sleep peacefully. The choice should be optimal in terms of confidentiality for both visitors and customers. So, one of the most important issue for us was the choice of server location.
Over the past few months, we have gradually moved our servers to Iceland. I want to explain how everything happened, and, most importantly, why. It was not an easy process, and I would like to share our experience. The article has some technical details that I tried to write in a clear language, but I apologize if they are too technical.
Why migrate servers?
It all started with the fact that our site was added to EasyList . This is a list of domain names for ad blockers. I asked why we were added, because we do not track visitors. We even obey the Do Not Track setting in the browser.
I wrote this comment on the pool request on GitHub :
[...] So if we continue to block good companies that respect user privacy, what's the point? I think this is wrong, you should not put each company on the list just because it sends a request. [...]
And got a response from @ cassowary714 :
Everyone agrees with you, but I do not want my requests sent to an American company (in your case, Digital Ocean [...]
At first I didn’t like the answer, but in a discussion with the community they pointed out that it was right. The U.S. government can actually access our user data. At that time, our servers really worked for Digital Ocean, they could just pull out our disk and read the data.
There is a technical solution to the problem. You can make a stolen (or disconnected for any reason) drive unsuitable for others. Full encryption will make access difficult in the absence of a key ( note: only Simple Analytics has a key ). You can still get small pieces of data by physically reading the server’s RAM. The server cannot work without RAM, so in this respect you have to trust the hosting provider.
This made me wonder where to move our servers.
New place
I began to search in this direction and came across a Wikipedia page with a list of countries that were noted for censorship and surveillance of users . There is a list of "enemies of the Internet" from the international non-governmental organization "Reporters Without Borders", which is located in Paris and stands for freedom of the press. A country is classified as an enemy of the Internet when it “not only censors news and information on the Internet, but also carries out almost systematic repressions against users.”
Beyond this list, there is an alliance called Five Eyes aka FVEY. This is a union of Australia, Canada, New Zealand, the UK and the USA. In recent years, documents have shown that they intentionally spy on each other's citizens and exchange information collected to circumvent legislative restrictions on espionage within the country ( sources ). Former NSA officer Edward Snowden described FVEY as "a supranational intelligence organization that does not obey the laws of its countries." There are other countries working with FVEY in other international cooperatives, including Denmark, France, the Netherlands, Norway, Belgium, Germany, Italy, Spain and Sweden (the so-called 14 Eyes). I could not find evidence that the 14 Eyes alliance was abusing intelligence.
After that, we decided that we would not be hosted in any of the countries on the list of “enemies of the Internet” and would definitely skip countries from the 14 Eyes alliance. The fact of collective surveillance is enough to refuse to store our customers' data there.
Regarding Iceland, the aforementioned Wikipedia page reads as follows:
Iceland’s constitution prohibits censorship, and there is a strong tradition of protecting freedom of expression that extends to the Internet. [...]
Iceland
During the search for a better country from the point of protection of privacy, Iceland appeared again and again. So I decided to study it carefully. Please keep in mind that I do not speak Icelandic, because of which I could have missed important information. Let me know if you have any information on the topic.
According to Freedom House's Freedom on the Net 2018 report, Iceland and Estonia scored 6/100 points in terms of censorship (the lower the better). This is the best result. Keep in mind that not all countries have been evaluated.
Iceland is not a member of the European Union, although it is a member of the European Economic Area and has agreed to follow consumer and business law similar to that of other member states. This includes the Electronic Communications Act (Electronic Communications Act 81/2003), which introduced data storage requirements.
The law applies to telecommunication service providers and provides for the storage of records for six months. It also states that companies can only provide information on telecommunications in criminal matters or on public safety, and that such information cannot be provided to anyone other than the police or prosecutors.
Although Iceland generally follows the laws of the European Economic Area, it has its own approach to protecting privacy. For example, the Icelandic Data Protection Act encourages the anonymity of user data. Internet service providers and hosters are not legally responsible for the content that they post or transmit. According to Icelandic law, the domain registrar ( ISNIC ) is responsible for the legality of using the .is domain. The government does not impose any restrictions on anonymous communication and does not require registration when buying SIM-cards.
Another advantage of moving to Iceland is the climate and location. The servers generate a lot of heat, and the average annual temperature in Reykjavik (the capital of Iceland, where most data centers are located) is 4.67 ° C, so this is a great place to cool the servers. For each watt for the operation of servers and network equipment, very little watt is spent proportionally for cooling, lighting and other overheads. In addition, Iceland is the world's largest producer of “clean” energy per capita and generally the largest producer of electricity per capita, with approximately 55,000 kWh per person per year. For comparison, the EU average is less than 6,000 kWh. Most hosters in Iceland receive 100% of their electricity from renewable sources.
If you draw a direct line from San Francisco to Amsterdam, you will cross Iceland. Simple Analytics has the majority of customers from the USA and Europe, so it makes sense to choose this geographical location. Additional advantages in favor of Iceland are laws protecting privacy and an environmental approach.
Server Migration
First, you had to find a local hosting provider. There are quite a lot of them, and it is really difficult to determine the best. We didn’t have the resources to try everyone, so we wrote several automatic scripts ( Ansible ) to configure the server so that we could easily switch to another host if necessary. We settled on the 1984 company with the motto, "Protecting Privacy and Civil Rights since 2006." We liked this motto, and we asked them some questions about how they will process our data. They reassured us, so we continued to install the main server. And they use electricity only from renewable sources.
However, during this process, we encountered several obstacles. This part of the article is quite technical. Feel free to move on to the next one. When you have an encrypted server, it is unlocked using a private key. This key cannot be stored on the server itself, that is, you need to enter it remotely when the server boots. Wait, what happens when you turn off the power? It turns out that all web page requests to the server will not be executed after a reboot?
That's why we added a primitive secondary server in front of the main server. It simply receives requests for viewing pages and sends them directly to the main server. If the primary server crashes, the secondary server will save the requests in its own database and will repeat them until it receives a response. Thus, after a power failure there is no data loss.
Let's get back to loading the server. When the encrypted main server loads, we need to enter the password. But we do not want to go to Iceland or ask someone to enter the server room there, for obvious reasons. For remote access to the server, the secure SSH protocol is usually used. But this program is available only during server or computer operation, and we need to connect before the server is fully loaded.
So we found Dropbear , a very small SSH client that can be run from disk in RAM for initial initialization (initramfs). And you can allow external connections via SSH. Now you don’t have to fly to Iceland to load our server, cheers!
Moving to a new server in Iceland took us a couple of weeks, but we are glad that we finally did it.
Store only necessary data
We live in Simple Analytics on the principle of "Store only the necessary data", collecting the minimum amount.
Web applications often practice soft data deletion . This means that the data is not actually deleted, but simply becomes inaccessible to the end user. We do not do this - if you delete your data, it will disappear from our database. We use hard removal. Note: they will remain in encrypted backups for a maximum of 90 days. In case of an error, we can restore them.
We have no delete_at fields ;-)
It is important for customers to know which data is stored and which is deleted. When someone deletes their data, we directly talk about it . The user and his analytics are deleted from the database. We also delete the credit card and email from Stripe (payment provider). We keep the payment history that is necessary for paying taxes and keep our log files and database backups for 90 days.
Question: if you only store a minimum of confidential data, why do you need all this protection and additional security?
Well, we want to be the best privacy-oriented analytics company in the world. We will do everything in our power to provide the best analytics tools without intruding on the privacy of your visitors. Even protecting huge volumes of anonymized information about visitors, we want to show that we take privacy very seriously.
What's next?
When we improved privacy, the speed of loading scripts embedded in web pages increased slightly. This makes sense because they used to be hosted on CloudFlare’s CDN, a collection of servers around the world that speed up downloads for everyone. Now we are thinking of raising a very simple CDN with encrypted servers that will only send our JavaScript and temporarily store web page requests before sending them to the main server in Iceland.