How to save up to half a million dollars in AWS?
Hello, Habr! I present to you the translation of the article “How to reduce your AWS costs? Save up to $ 500k with these guidelines! ” By George Batschinski.
In this article, we will detail how Back4App reduced its AWS expenses from $ 55,492 per month to $ 20,074 per month and saved $ 500,000 per year. We will describe practical steps based on our own experience of reducing costs in AWS and share tips that were useful to us.
AWS continues to innovate and dominate the cloud market. It has grown so much that it has become a key part of Amazon's strategy. I'm enthusiastic about AWS, but doubts about the complexity of the cloud and the cost led me to the idea of optimizing the use of the cloud. Despite the fact that AWS offers an ever-growing list of specialized services for various types of development, this is not always the most direct, fast or affordable way. Ideally, as soon as you go to the cloud, you need to immediately think about the architecture to reduce costs. This material will help accelerate cost optimization and contains real-life cases that we encountered during our cost-cutting venture.
A brief overview of what you will find in this article:
Here's what our AWS account looked like in January 2019. The total cost of the cloud is $ 55,492.
Here's what our AWS account looked like in May 2019. The total cost was $ 20,074.
First thing! Before you start doing anything, please make sure your company is eligible for AWS loans. Here's how you can get these loans:
So it was in our case! In 2016, our company grew quite fast. Our AWS bill began to increase and we had no idea or time to optimize the use of AWS. Since we did not have funding, we realized that we would either receive loans from AWS or we would disappear. We prepared our case for our account for managers in AWS, demonstrating how fast we are growing and the advantages for AWS in supporting us on our site. We showed him that we did not have funding and would be forced to go to Google Cloud (they offered us $ 100,000 in loans to migrate to AWS) if we did not receive loans from AWS. The response took about a week and AWS provided us with $ 100,000 for use over 12 months. It was a good deal for AWS as well! Since the loans expired, we have spent more than $ 750,000 with AWS.
If you intend to test a new service from AWS or an instance type, ask for loans from your account manager. The number of loans will depend on your monthly costs and the overall effort to launch POC. These loans are usually only valid for three months, and you need to make sure the speed of the POC. In our case, we were able to get $ 3,000 in loans to test the i3 family of instances.
If you founded your company or launched a startup, this is a great way to move forward. The amount of loans ranges from $ 1,000 to $ 100,000, depending on how you present your AWS case. There are several ways to get these loans.
One of the main advantages of Startup Accelerators in the bonuses that they provide. One of these bonuses is loans from cloud providers. The amount that you can receive will depend on the Accelerator in which you are represented. Some of the best, for example, YC, Alchemist, Wayra and 500 Startups will offer you $ 100,000 in AWS loans. In our case, we were accepted by Alchemist Accelerator and issued $ 100,000 in loans. Unfortunately, we were not able to take advantage of these loans, since before that we had already taken advantage of other loans. For example, if you accept two accelerators for $ 100,000 each, you will not be able to use $ 200,000, only $ 100,000.
Some WF / Angel Groups provide loans as part of the bonus program. In our case, we managed to get $ 10,000 of marketing loans, valid for two years from one of our investors. Unfortunately, we could not use them. Companies cannot use AWS Promotional Credits if they have previously received an equal or greater amount of loans.
All the leading players in the cloud market offer loans to startups. With AWS / Google, loans can go up to $ 100,000. At Azure, you can get $ 360,000 for three years.
The easy part is over, and now let's get down to the challenges! There is no magic pill to reduce costs in AWS, you need to work hard and follow a regular procedure to monitor costs! Reducing cloud costs should be your daily routine! Imagine how much you save in 365 days! The amount of time you invest depends on the size of your organization and your AWS costs. I spend about 30 minutes a day monitoring my account and prioritizing cost-cutting actions. In addition, Back4App engineers spend at least 1 hour per day on cost-cutting initiatives.
The first initiative we completed: creating a simple worksheet in Excel to prioritize cost reduction initiatives.
The main criteria by which we determine savings:
In our case, the greatest savings are the use of Elastic Compute Cloud (EC2), and this was the first subject that we started to understand.
The first point to be taken to reduce costs on EC2 is to turn off unused instances. There is nothing complicated here, and most of the work is quite straightforward. All you need to do is make a list of all the instances, load the excel file from the example, and define each. In our case, we divided our instances into 3 main categories:
Do not stop, but delete instances
Remove unused instances when possible! If you just stop them, you will continue to pay for using EBS!
This action is a little more complicated, and it is absolutely necessary to correctly select the instance size. Please ensure that you evaluate at least one week of data to increase the likelihood of a correct instance change. The approach by which we controlled this process was to divide instances into 3 main categories:
The way to verify historical data for an instance is extremely simple. In the AWS control panel, go to Services> EC2> Instances> Running Instances, select Instance> Monitoring. For example, you can check the CPU Utilization metric.
Reduce instance size sequentially
Reduce your instance size step by step. For example, do not reduce the size from t2.xlarge to t2.medium. First of all, reduce it from t2.2xlarge to t2.xlarge. Take a look at one week, check how many resources are being used and repeat the steps on a new one. By performing these steps, you will minimize the chances of errors or downtime.
Reserving instances can increase your savings by up to 70%. Please keep in mind that when you reserve an instance, you pay for using 24/7 throughout the entire reservation period. You will pay regardless of whether you use the instance or not.
There are two types of redundant capacities:
Standard - which you can sell on AWS Marketplace, you cannot convert to another type of instance, cheaper than convertible instances.
Convertible - You won’t be able to sell it on the AWS Marketplace, but you can convert it to an instance of a different type, more expensive than standard instances.
Convertible instances can be confusing
Convertible instances are very difficult! Each time you convert instances, you can only increase the size of the instance. Consider the following situation:
You acquire t3.medium, a convertible reserved instance.
You decide to convert t3.medium to m5.large instance.
You will certainly succeed! After using the m5.large instance for some time, you will want to return to t3.medium.
And this is already impossible!
You will not be able to reduce the consumption of the convertible instance, but only to increase the amount that you pay for it. So please, be sure to start with a small reserved instance and increase it over time. Do not start with a large instance, because you cannot reduce its consumption or sell it on AWS Marketplace'e. If you make a mistake with a convertible reserved instance, you are subscribed to a monthly payment of 12 or 36 months. At Back4App, we always first purchase t2.nano and increase it as needed.
I prefer standard Reserved instances over convertible for the following reasons:
Reserved instances are great, but buying reserved instances at AWS Marketplace is even better!
AWS Marketplace looks like a securities market! You can sometimes buy a standard Reserved instance with a price much more affordable than buying directly from AWS. The reason is simple; Companies sometimes own redundant instances that are no longer in use, but continue to pay for them. The reduction in the placement price greatly accelerates the purchase process.
Here I will give an example of what happened with Back4App. I was looking for 4 r4.large reserved instances. I managed to find instances on AWS Marketplace with an effective rate of 75% cheaper compared to On-Demand prices. AWS sold the same type of reserved instances, but with a 62% discount.
Another advantage of AWS Marketplace is that you can purchase instances with a flexible period of use. AWS itself provides only options for 12 and 36 months for the purchase of reserved instances. AWS Marketplace is much more flexible! You can purchase, for example, reserved instances for 15 months for the same price tag as for 36 months. Some companies prefer long-term obligations (36 months), and before the obligation expires, they sell them on AWS Marketplace. The new customer will only use the instance for the remaining period, for example:
The main difficulty is finding the right instances on the AWS Marketplace. A great way to simplify the search for instances is to look for them in several access zones, and not just in the one where you are located. Once you acquire an instance, you can modify it and assign it to another accessibility zone, or turn it into a regional instance. This is not at all difficult to do, but please follow the instructions below:
Following these steps will allow you to purchase Standard Reserved instances in any availability zone and modify it for your availability zone.
Important clarification for users outside the US jurisdiction. You will not be able to sell Standard Reserved Instances on the Marketplace unless you have a state legal entity. If your company finds outside the United States, you will be able to buy instances on Marketplace, but you will not be able to sell them.
I love Spot Instances! You can save up to 90% using them, and it will not require large investments! The only minus to Spot instances is that AWS can delete these instances in no time or after a two-minute warning. In this regard, this type of instances can only be used for fault tolerant or flexible loads.
Use SpotInst to Run Spot Instances
A safe way to run Spot instances and minimize the risk of downtime is to use the SpotInst platform. This service connects to your AWS account and manages the distribution of Spot instances. If the instance is turned off, the service will replace it with On Demand and prevent downtime. Rates are also quite interesting, they charge 20% of what was saved (between the difference in the price of On-demand and Spot).
A great scenario for Spot instances is to use a copy of servers for databases or for flexible application extensions. Use SpotInst to control the allocation of Spot instances, and most likely you will never encounter downtime associated with the use of this class of instances.
AWS Auto Scaling monitors your applications and automatically adjusts power to maintain stable, predictable power at the lowest cost. You can configure auto-scaling to automatically start and stop EC2 instances. Auto-scaling takes into account resource metrics such as CPU, API requests and on / off times.
To minimize costs, you must use automatic scaling + Spot instances + Reserved instances. Let's look at the real situation that we encountered in Back4App. Workloads change throughout the day. The number of requests that we receive is not uniform and has peak usage loads at certain hours. To solve this problem, we allocate Reserved Instances for the minimum consumption boundary, and scale with spot instances. The figure below helps you understand the configuration. Here's how much you can save:
Adjust scaling
Customize your vertical and horizontal scaling (Cloudformation)
The most expensive: Sao Paulo, Sydney and Hong Kong
Consider the following example:
The same configuration will be 102% more expensive in Sao Paulo compared to North Virginia.
Centralization of operations in one region also has its advantages.
At Back4App, we were unable to centralize all operations in North Virginia due to our business coverage and customer requirements. We decided to create a cluster in the USA, Europe and Asia. We centralized our operations in these clusters and expanded each region as much as possible.
AWS occasionally launches new types of instances. New generations are more powerful and cheaper.
The old generation of instances
C4.large
2 cores
3.75 GB of memory
$ 0.100 / hour
The new generation of instances
C5.large
2 cores
4 gb memory
$ 0.085 / hour
The new generation is 15% cheaper and 6% more powerful!
Allow enhanced network
Enable enhanced network for better instance performance!
EBS provides secure storage volumes for use with EC2 instances in the AWS cloud. Each EBS volume is duplicated within the availability zone to protect you from hardware failure, providing high availability and reliability. The cost of each gigabyte stored depends on the type of volume that you will use. The most common types are General Purpose SSD and Magnetic.
General Purpose SSD
$ 0.10 / GB-month
Magnetic
$ 0.05 / GB-month
Start tagging your instances and download this spreadsheet to make your life easier. For example, determine if the volume is Magnetic or SSD.
The second point to save on EBS is to determine the size of each attached volume. Excessive volumes and unused gigabytes are often used.
Stopping instances does not protect you from the costs of EBS! Your volume will continue to be stored and will be included in the payment! If you no longer need the instance and data, delete the instance! This action will reduce the costs of EC2 and EBS.
AWS allows consecutively increasing the size of EBS, but does not allow decreasing it. In general, it is better to have smaller volumes than large ones, and use EBS only when necessary.
Let's estimate how expensive it can be. Let's say you use 10 t3.small instances and each instance uses 50GB SSD EBS storage.
The monthly bill will be:
This simple example shows that EBS can go up to 32% of the cost of EC2 and make up a significant part of your account. This is a constant hidden value that needs to be controlled.
Take a look at the ephemeral vault
Check out the pros and cons of ephemeral storage. Of the benefits, you do not have to pay hidden costs for EBS, the delay will be lower and you can get discounts on Reserved instances. Of the minuses, it’s likely that you will not use all the internal storage (for which you continue to pay), and using Spot instances will become much more difficult.
Another action that will save money is the removal of disconnected and unused volumes.
Use S3 to backup
EBS is at least twice as expensive as S3! In fact, you should save backups to S3, and not to EBS. Please first make sure that your backups are saved incrementally and old versions are periodically deleted.
Few know that you can purchase reserved instances for Elastic Cache. The savings are quite significant:
The process is similar to acquiring EC2 instances. To purchase reserved node, you must do the following:
Other cost savings with Elastic Cache are pretty much the same as with EC2. You need to turn off unused nodes and reduce the size of the nodes themselves.
There is no AWS Marketplace for Elastic Cache
Please note that there is no Marketplace for Reserved Nodes! Before buying, be sure to use it for at least a year.
This cost line can get out of hand pretty quickly! It’s hard enough to control where and where the data is coming from, and tracking is even harder. In addition, there are several types of data transfer. For example, data transfer from the cloud, data transfer between regions, data transfer between access zones, S3 data transfer, etc.
I’ll tell you how Back4App lowered data transfer costs and how it tracked them.
Here are a few things you can do to lower data costs.
CDN (Content Delivery Network or Data Network) This simple action can reduce your data transfer costs by up to five times, depending on how much cache you hold.
This is a great example! The AWS calculator says that transferring data to the cloud is free for EC2 instances, but you also need to fill in the field for “Public IP / Elastic IP Data Transfer”. If you select a connection using a Public or Elastic IP address, or an Elastic Load Balancer within the Amazon network, you will pay interregional data transfer rates even if the instances are within the same availability zone. To transfer data within one access zone, you can easily avoid such costs (and improve network performance) by using your personal IP addresses where possible.
Using one availability zone instead of several, you can significantly save. Avoid or stop data transfer between regions.
One of the significant advantages of GraphQL is the reduction of data over-delivery and, as a result, the reduction in the total amount of data transferred. Replacing REST can be a daunting task, depending on how your application is structured; Also, it is worth assessing in advance the feasibility of such a transition. Another alternative is to use GraphQL sequentially for new products and features.
This is hardly applicable, since the required volumes for discounts are very large. This can be useful only to large businesses or large accounts.
You can save a fortune if you apply this action! It will also allow you to increase the speed of your applications and generally reduce latency - a win-win offer. I will talk about this a bit below.
There are two easy ways to track data costs.
They represent the collected data in different ways, and given both versions, you can better understand how the money is spent on data transfer.
To access the Cost Explorer report, go to My Billing Dashboard and select Cost Explorer. Select the Spend by Service View report and a new window will open. The prepared report for the last 6 months will open, and we will begin with it. Please note that in the column Service there are NO data bills! These costs are hidden by filters and you need to dig deeper to find them.
To display data transmission costs, you will need to configure the filters as follows:
After applying all the filters, the report will look like this:
Now you have sorted traffic information and where it comes from from S3 or EC2. If you want, you can apply a similar process and further break down costs by region and accessibility zone.
Another way to show traffic costs is with a billing report. Go to My Billing Dashboard and click Bills on the left.
The report will look something like this:
Note that there is a built-in splitting here showing the costs of EC2 data transfer. Click on Data Transfer and an analysis by region will appear.
Having selected a region, a breakdown by the type of transmitted data will appear (outgoing traffic, between availability zones, inter-regional).
Cloudfront's on-demand prices are rapidly decreasing as volume grows. Let's imagine the transmission of 100 Tb traffic. It will cost $ 0.060 / Gb, which is about 15% less compared to the same amount of data via ELB.
Here is the best way to reduce data costs! AWS prefers to use its CloudFront service and depending on the volume, you can reduce the price of outgoing CloudFront traffic to $ 0.02 / Gb. Most likely this will require a two-year commitment, and if you plan to use AWS for a long time, then this is a great offer. To make such an arrangement, you need to contact your account manager and ask him about CloudFront Private Contract.
Check if you are using this certificate. It costs $ 20 per day, and the same can be purchased from Cloudflare for $ 25 per month. With a Private & Volume Pricing contract, you will not be able to use other CDN (Cloud Delivery Network) services, and this action will not be available to you.
A few recommendations for reducing prices for S3 storage.
A great saving idea that few people are given is the use of the S3 VPC endpoint. Placing an S3 endpoint in your VPC (Virtual Private Cloud) gives any data access to S3 and gives it its own path inside the cloud, and AWS does not charge you as if it were public traffic. Depending on the volume, this technique can save up to several thousand dollars a month.
There are five storage models in total:
S3 standard
$ 0.023 / GB
Options:
S3 Infrequent Access
$ 0.0125 / GB
Options:
S3 One-Zone Infrequent Access
$ 0.010 / GB
Options:
S3 glacier
$ 0.004 / GB
Options:
S3 Glacier Deep Archive
$ 0.00099 / GB
Options:
S3 API pricing can be confusing
Please note that each time the storage class for objects changes, you will pay for the number of requests. It can be very difficult! I will share the situation that occurred in Back4App. One of our customers stores millions of small objects, and data is rarely in demand. At first, we kept all the objects in S3 Standard and decided to transfer them to S3 Glacier to save money. So far so good! But we did not take into account that AWS charges $ 0.05 for every 1000 calls to Glacier! In our case, the number of requests was huge, so the cost of moving did not pay off! Before moving objects between storage classes, it is necessary to consider the price for requests to S3 and conduct a payback analysis.
The easiest and most effective way to save on S3 is to simply delete old and unused data. Take a look at old backups and unused S3 bucket. Perform an analysis and, if possible, delete the data.
There are 4 types of technical support:
We tried the Business plan and were unsatisfied with the result. In our opinion, it is too expensive for the level of support and support that it provides. I can not say anything about the Enterprise level, since we did not use it. (The Enterprise level is suitable, for example, to large companies before launching major premieres or products.)
Engage your regional AWS manager
Use your regional manager and solution architect to the maximum! They work separately from the central support team and can provide much more suitable services. The AWS team is generally very skilled and helps a lot! The recommendations they provided on advanced security settings and the use of GDPR were simply the best! I was impressed by the technical awareness and the desire to help us achieve better results. Such support makes you think twice before changing your cloud provider.
It is difficult to overestimate the importance of calculating and knowing the daily costs. You must set up a detailed monitoring of your costs for yourself and take time every day to familiarize yourself with them. Imagine how much you can save in 365 days!
Daily monitoring will help detect suspicious trends and act immediately. Best practices regarding Monitoring and Management are listed below:
Changing the architecture is usually the most time-consuming exercise. It can take from weeks to months, depending on the size of your architecture.
Our experience with Back4App shows that you should try the simpler steps first. In general, successive changes will bring you significant savings in the short term. Make money on such changes and gain experience for rebuilding your architecture.
There is no magic formula to immediately reduce costs in AWS! It is necessary to develop a routine habit of checking expenses, setting priorities, and working hard to achieve your goals.
Here is a brief overview of the steps that helped us reduce our AWS score.
In this article, we will detail how Back4App reduced its AWS expenses from $ 55,492 per month to $ 20,074 per month and saved $ 500,000 per year. We will describe practical steps based on our own experience of reducing costs in AWS and share tips that were useful to us.
AWS continues to innovate and dominate the cloud market. It has grown so much that it has become a key part of Amazon's strategy. I'm enthusiastic about AWS, but doubts about the complexity of the cloud and the cost led me to the idea of optimizing the use of the cloud. Despite the fact that AWS offers an ever-growing list of specialized services for various types of development, this is not always the most direct, fast or affordable way. Ideally, as soon as you go to the cloud, you need to immediately think about the architecture to reduce costs. This material will help accelerate cost optimization and contains real-life cases that we encountered during our cost-cutting venture.
A brief overview of what you will find in this article:
- Apply for loans from AWS and get up to $ 100,000;
- Transfer data transfer to Cloud Front and save up to 75%;
- Use private IPs (not Elastic IPs) and stop paying for data transfer between access zones;
- Save up to 90% with Spot Instances;
- Save up to 75% by purchasing Reserved Instances at AWS Marketplace;
- Get reserved Elastic Cache nodes and save up to 77%;
- Reserve your data in S3, but not in EBS and save up to 75%;
- Configure auto-scaling for idle resources;
- Choose the North Virginia region and stay away from Sao Paulo;
- Upgrade to new generations of instances and save up to 20%;
- Locate the S3 endpoint in your VPC and enable the Enhanced Network on EC2.
Here's what our AWS account looked like in January 2019. The total cost of the cloud is $ 55,492.
Here's what our AWS account looked like in May 2019. The total cost was $ 20,074.
1. AWS loans
First thing! Before you start doing anything, please make sure your company is eligible for AWS loans. Here's how you can get these loans:
Ask AWS for loans
So it was in our case! In 2016, our company grew quite fast. Our AWS bill began to increase and we had no idea or time to optimize the use of AWS. Since we did not have funding, we realized that we would either receive loans from AWS or we would disappear. We prepared our case for our account for managers in AWS, demonstrating how fast we are growing and the advantages for AWS in supporting us on our site. We showed him that we did not have funding and would be forced to go to Google Cloud (they offered us $ 100,000 in loans to migrate to AWS) if we did not receive loans from AWS. The response took about a week and AWS provided us with $ 100,000 for use over 12 months. It was a good deal for AWS as well! Since the loans expired, we have spent more than $ 750,000 with AWS.
Credits for testing and launching Proof Of Concept
If you intend to test a new service from AWS or an instance type, ask for loans from your account manager. The number of loans will depend on your monthly costs and the overall effort to launch POC. These loans are usually only valid for three months, and you need to make sure the speed of the POC. In our case, we were able to get $ 3,000 in loans to test the i3 family of instances.
Startup Credits
If you founded your company or launched a startup, this is a great way to move forward. The amount of loans ranges from $ 1,000 to $ 100,000, depending on how you present your AWS case. There are several ways to get these loans.
Accelerators
One of the main advantages of Startup Accelerators in the bonuses that they provide. One of these bonuses is loans from cloud providers. The amount that you can receive will depend on the Accelerator in which you are represented. Some of the best, for example, YC, Alchemist, Wayra and 500 Startups will offer you $ 100,000 in AWS loans. In our case, we were accepted by Alchemist Accelerator and issued $ 100,000 in loans. Unfortunately, we were not able to take advantage of these loans, since before that we had already taken advantage of other loans. For example, if you accept two accelerators for $ 100,000 each, you will not be able to use $ 200,000, only $ 100,000.
Venture funds and groups of angels
Some WF / Angel Groups provide loans as part of the bonus program. In our case, we managed to get $ 10,000 of marketing loans, valid for two years from one of our investors. Unfortunately, we could not use them. Companies cannot use AWS Promotional Credits if they have previously received an equal or greater amount of loans.
All leading cloud providers provide loans for startups
All the leading players in the cloud market offer loans to startups. With AWS / Google, loans can go up to $ 100,000. At Azure, you can get $ 360,000 for three years.
2. Implement cost savings on AWS
The easy part is over, and now let's get down to the challenges! There is no magic pill to reduce costs in AWS, you need to work hard and follow a regular procedure to monitor costs! Reducing cloud costs should be your daily routine! Imagine how much you save in 365 days! The amount of time you invest depends on the size of your organization and your AWS costs. I spend about 30 minutes a day monitoring my account and prioritizing cost-cutting actions. In addition, Back4App engineers spend at least 1 hour per day on cost-cutting initiatives.
The first initiative we completed: creating a simple worksheet in Excel to prioritize cost reduction initiatives.
The main criteria by which we determine savings:
- Savings Potential - Highest Earnings from Savings
- Difficulty - How difficult is it to apply the effort and required work
- Owner - Who will perform the action
In our case, the greatest savings are the use of Elastic Compute Cloud (EC2), and this was the first subject that we started to understand.
3. EC2 - Elastic Compute Cloud
Turn off unused instances
The first point to be taken to reduce costs on EC2 is to turn off unused instances. There is nothing complicated here, and most of the work is quite straightforward. All you need to do is make a list of all the instances, load the excel file from the example, and define each. In our case, we divided our instances into 3 main categories:
- Instances that must work continuously
- Instances that can be stopped
- Instances that can be deleted
Do not stop, but delete instances
Remove unused instances when possible! If you just stop them, you will continue to pay for using EBS!
Reduce excess resources
This action is a little more complicated, and it is absolutely necessary to correctly select the instance size. Please ensure that you evaluate at least one week of data to increase the likelihood of a correct instance change. The approach by which we controlled this process was to divide instances into 3 main categories:
- Correct size
- Reduce Instance Size
- Increase Instance Size
The way to verify historical data for an instance is extremely simple. In the AWS control panel, go to Services> EC2> Instances> Running Instances, select Instance> Monitoring. For example, you can check the CPU Utilization metric.
Reduce instance size sequentially
Reduce your instance size step by step. For example, do not reduce the size from t2.xlarge to t2.medium. First of all, reduce it from t2.2xlarge to t2.xlarge. Take a look at one week, check how many resources are being used and repeat the steps on a new one. By performing these steps, you will minimize the chances of errors or downtime.
Use Reserved Instances
Reserving instances can increase your savings by up to 70%. Please keep in mind that when you reserve an instance, you pay for using 24/7 throughout the entire reservation period. You will pay regardless of whether you use the instance or not.
There are two types of redundant capacities:
Standard - which you can sell on AWS Marketplace, you cannot convert to another type of instance, cheaper than convertible instances.
Convertible - You won’t be able to sell it on the AWS Marketplace, but you can convert it to an instance of a different type, more expensive than standard instances.
Convertible instances can be confusing
Convertible instances are very difficult! Each time you convert instances, you can only increase the size of the instance. Consider the following situation:
You acquire t3.medium, a convertible reserved instance.
You decide to convert t3.medium to m5.large instance.
You will certainly succeed! After using the m5.large instance for some time, you will want to return to t3.medium.
And this is already impossible!
You will not be able to reduce the consumption of the convertible instance, but only to increase the amount that you pay for it. So please, be sure to start with a small reserved instance and increase it over time. Do not start with a large instance, because you cannot reduce its consumption or sell it on AWS Marketplace'e. If you make a mistake with a convertible reserved instance, you are subscribed to a monthly payment of 12 or 36 months. At Back4App, we always first purchase t2.nano and increase it as needed.
I prefer standard Reserved instances over convertible for the following reasons:
- They are cheaper
- They have higher liquidity - they can be sold at AWS Marketplace at any time.
- Discounted on AWS Marketplace
Use AWS Marketplace for reserved instances
Reserved instances are great, but buying reserved instances at AWS Marketplace is even better!
AWS Marketplace Can Be Much Cheaper
AWS Marketplace looks like a securities market! You can sometimes buy a standard Reserved instance with a price much more affordable than buying directly from AWS. The reason is simple; Companies sometimes own redundant instances that are no longer in use, but continue to pay for them. The reduction in the placement price greatly accelerates the purchase process.
Here I will give an example of what happened with Back4App. I was looking for 4 r4.large reserved instances. I managed to find instances on AWS Marketplace with an effective rate of 75% cheaper compared to On-Demand prices. AWS sold the same type of reserved instances, but with a 62% discount.
AWS Marketplace has flexible purchasing conditions
Another advantage of AWS Marketplace is that you can purchase instances with a flexible period of use. AWS itself provides only options for 12 and 36 months for the purchase of reserved instances. AWS Marketplace is much more flexible! You can purchase, for example, reserved instances for 15 months for the same price tag as for 36 months. Some companies prefer long-term obligations (36 months), and before the obligation expires, they sell them on AWS Marketplace. The new customer will only use the instance for the remaining period, for example:
- Total acquisition time: 36 months
- Initial Buyer: Uses instance for 21 months
- Second Buyer: Purchases it on the Marketplace and uses it for the remaining 15 months.
The main difficulty is finding the right instances on the AWS Marketplace. A great way to simplify the search for instances is to look for them in several access zones, and not just in the one where you are located. Once you acquire an instance, you can modify it and assign it to another accessibility zone, or turn it into a regional instance. This is not at all difficult to do, but please follow the instructions below:
- Step 1 - Go to the Reserved Instances Dashboard
- Step 2 - Click on Purchase Reserved Instances
- Step 3 - in the Offering Class, select Standard
- Step 4 - In the upper right corner of the popup window, select “Only show offerings that reserve capacity”
- Step 5 - Click Search
- Step 6 - Get an Instance from Availability Zone 1b
- Step 7 - Wait for the instance to become active
- Step 8 - Select the instance that was purchased
- Step 9 - Click on the Actions Button
- Step 10 - Click on Modify Reserved Instances
- Step 11 - Scope - Change Availability Zone or Region
- Step 12 - Click Continue
- Step 13 - Click on Submit Modifications
Following these steps will allow you to purchase Standard Reserved instances in any availability zone and modify it for your availability zone.
AWS Marketplace for companies outside the US
Important clarification for users outside the US jurisdiction. You will not be able to sell Standard Reserved Instances on the Marketplace unless you have a state legal entity. If your company finds outside the United States, you will be able to buy instances on Marketplace, but you will not be able to sell them.
Use Spot Instances
I love Spot Instances! You can save up to 90% using them, and it will not require large investments! The only minus to Spot instances is that AWS can delete these instances in no time or after a two-minute warning. In this regard, this type of instances can only be used for fault tolerant or flexible loads.
Use SpotInst to Run Spot Instances
A safe way to run Spot instances and minimize the risk of downtime is to use the SpotInst platform. This service connects to your AWS account and manages the distribution of Spot instances. If the instance is turned off, the service will replace it with On Demand and prevent downtime. Rates are also quite interesting, they charge 20% of what was saved (between the difference in the price of On-demand and Spot).
A great scenario for Spot instances is to use a copy of servers for databases or for flexible application extensions. Use SpotInst to control the allocation of Spot instances, and most likely you will never encounter downtime associated with the use of this class of instances.
Automatic scaling (Idle resources, Automation of start / stop EC2 instances)
AWS Auto Scaling monitors your applications and automatically adjusts power to maintain stable, predictable power at the lowest cost. You can configure auto-scaling to automatically start and stop EC2 instances. Auto-scaling takes into account resource metrics such as CPU, API requests and on / off times.
To minimize costs, you must use automatic scaling + Spot instances + Reserved instances. Let's look at the real situation that we encountered in Back4App. Workloads change throughout the day. The number of requests that we receive is not uniform and has peak usage loads at certain hours. To solve this problem, we allocate Reserved Instances for the minimum consumption boundary, and scale with spot instances. The figure below helps you understand the configuration. Here's how much you can save:
- Reserved Instances: At Back4App, we use a mix of 12/36 monthly standard reserved instances. Our median savings are 50% compared to on-demand prices.
- Automatic scaling: We start / stop instances based on the total number of API requests that each instance receives. We have three peak periods during the day, and the savings compared to idle resources that would work 24 hours at maximum load is approximately 75%.
- Spot instances: We have a mix of two families of instances that work on Spotinst for maximum uptime. Our savings are 55% compared to on-demand prices. This is our savings, taking into account the entire amount that we pay SpotInst. Using spot instances directly from AWS could save us about 70%, but would increase the risks. We like to play measuredly, and we decided to give back some of our savings to SpotInst.
Adjust scaling
Customize your vertical and horizontal scaling (Cloudformation)
Choose the Right AWS Region to Save
- Prices vary between regions! Choose the wrong one and you will overpay.
- The cheapest regions are North Virginia, Ohio, and Oregon.
The most expensive: Sao Paulo, Sydney and Hong Kong
Consider the following example:
- 4 x t3.medium instances
- 30 Gb EBS dedicated to each instance
- 1 Tb total traffic
The same configuration will be 102% more expensive in Sao Paulo compared to North Virginia.
- $ 221.65 / month in North Virginia
- $ 449.05 / month in Sao Paulo
Centralization of operations in one region also has its advantages.
- It is much easier to control the allocation of reserved instances.
- Inter-region traffic charges will not apply
At Back4App, we were unable to centralize all operations in North Virginia due to our business coverage and customer requirements. We decided to create a cluster in the USA, Europe and Asia. We centralized our operations in these clusters and expanded each region as much as possible.
Update previous generations of resources
AWS occasionally launches new types of instances. New generations are more powerful and cheaper.
The old generation of instances
C4.large
2 cores
3.75 GB of memory
$ 0.100 / hour
The new generation of instances
C5.large
2 cores
4 gb memory
$ 0.085 / hour
The new generation is 15% cheaper and 6% more powerful!
Allow enhanced network
Enable enhanced network for better instance performance!
4. EBS - Elastic Block Store
EBS provides secure storage volumes for use with EC2 instances in the AWS cloud. Each EBS volume is duplicated within the availability zone to protect you from hardware failure, providing high availability and reliability. The cost of each gigabyte stored depends on the type of volume that you will use. The most common types are General Purpose SSD and Magnetic.
General Purpose SSD
$ 0.10 / GB-month
Magnetic
$ 0.05 / GB-month
EBS Type - SSD x Magnetic
Start tagging your instances and download this spreadsheet to make your life easier. For example, determine if the volume is Magnetic or SSD.
- Choose Magnetic instead of SSD where possible.
- Replace SSD volume where possible.
EBS Size - Reduce Excessive Volumes
The second point to save on EBS is to determine the size of each attached volume. Excessive volumes and unused gigabytes are often used.
Delete instances - stopping will not affect EBS payment
Stopping instances does not protect you from the costs of EBS! Your volume will continue to be stored and will be included in the payment! If you no longer need the instance and data, delete the instance! This action will reduce the costs of EC2 and EBS.
AWS allows consecutively increasing the size of EBS, but does not allow decreasing it. In general, it is better to have smaller volumes than large ones, and use EBS only when necessary.
Let's estimate how expensive it can be. Let's say you use 10 t3.small instances and each instance uses 50GB SSD EBS storage.
The monthly bill will be:
- EC2 Monthly Cost = 10 x $ 15.23 = 152.30
- EBS Total capacity: 10 x 50 GB = 500 GB
- Price per GB: $ 0.10-month
- EBS Monthly Cost: $ 50.00
- Total cost: $ 182.30
This simple example shows that EBS can go up to 32% of the cost of EC2 and make up a significant part of your account. This is a constant hidden value that needs to be controlled.
Take a look at the ephemeral vault
Check out the pros and cons of ephemeral storage. Of the benefits, you do not have to pay hidden costs for EBS, the delay will be lower and you can get discounts on Reserved instances. Of the minuses, it’s likely that you will not use all the internal storage (for which you continue to pay), and using Spot instances will become much more difficult.
Remove unused volumes
Another action that will save money is the removal of disconnected and unused volumes.
Use S3 to backup
EBS is at least twice as expensive as S3! In fact, you should save backups to S3, and not to EBS. Please first make sure that your backups are saved incrementally and old versions are periodically deleted.
5. Elastic Cache Service - Reserved Nodes
Few know that you can purchase reserved instances for Elastic Cache. The savings are quite significant:
- 1 year reserve - 65% discount
- 3 years reserve - 77% discount
The process is similar to acquiring EC2 instances. To purchase reserved node, you must do the following:
- Go to Elastic Cache (not EC2 Reserved instances)
- Go to Reserved Nodes
- Click on the Blue “Purchase Reserved Node” button in the upper right
- Choose Redis or Memcached
- Select node type
- Choose your desired booking period
- Choose Offering Type (1 or 3 years)
- Click Next
Other cost savings with Elastic Cache are pretty much the same as with EC2. You need to turn off unused nodes and reduce the size of the nodes themselves.
There is no AWS Marketplace for Elastic Cache
Please note that there is no Marketplace for Reserved Nodes! Before buying, be sure to use it for at least a year.
6. Data Transfer
This cost line can get out of hand pretty quickly! It’s hard enough to control where and where the data is coming from, and tracking is even harder. In addition, there are several types of data transfer. For example, data transfer from the cloud, data transfer between regions, data transfer between access zones, S3 data transfer, etc.
I’ll tell you how Back4App lowered data transfer costs and how it tracked them.
How to reduce data transfer costs in AWS?
Here are a few things you can do to lower data costs.
Connect S3 endpoint with Cloudflare and other CDN services.
CDN (Content Delivery Network or Data Network) This simple action can reduce your data transfer costs by up to five times, depending on how much cache you hold.
Elastic IPs - Use your own IP addresses where possible
This is a great example! The AWS calculator says that transferring data to the cloud is free for EC2 instances, but you also need to fill in the field for “Public IP / Elastic IP Data Transfer”. If you select a connection using a Public or Elastic IP address, or an Elastic Load Balancer within the Amazon network, you will pay interregional data transfer rates even if the instances are within the same availability zone. To transfer data within one access zone, you can easily avoid such costs (and improve network performance) by using your personal IP addresses where possible.
Use One Availability Zone - Availability Zone
Using one availability zone instead of several, you can significantly save. Avoid or stop data transfer between regions.
Use GraphQL API
One of the significant advantages of GraphQL is the reduction of data over-delivery and, as a result, the reduction in the total amount of data transferred. Replacing REST can be a daunting task, depending on how your application is structured; Also, it is worth assessing in advance the feasibility of such a transition. Another alternative is to use GraphQL sequentially for new products and features.
Data Transfer Discounts
This is hardly applicable, since the required volumes for discounts are very large. This can be useful only to large businesses or large accounts.
Use CloudFront!
You can save a fortune if you apply this action! It will also allow you to increase the speed of your applications and generally reduce latency - a win-win offer. I will talk about this a bit below.
How to track data transfer costs?
There are two easy ways to track data costs.
- Cost explorer
- Accounts
They represent the collected data in different ways, and given both versions, you can better understand how the money is spent on data transfer.
To access the Cost Explorer report, go to My Billing Dashboard and select Cost Explorer. Select the Spend by Service View report and a new window will open. The prepared report for the last 6 months will open, and we will begin with it. Please note that in the column Service there are NO data bills! These costs are hidden by filters and you need to dig deeper to find them.
To display data transmission costs, you will need to configure the filters as follows:
- Click More Filters to expand them.
- Choose Usage Type Group
- Data transfer information appears
- Choose Data Transfer - Inter AZ
- Choose Data Transfer - Internet Out
- Choose Data Transfer - Region to Region Out
- Choose S3 Data Transfer Out
- Click Apply Filters
After applying all the filters, the report will look like this:
Now you have sorted traffic information and where it comes from from S3 or EC2. If you want, you can apply a similar process and further break down costs by region and accessibility zone.
Another way to show traffic costs is with a billing report. Go to My Billing Dashboard and click Bills on the left.
The report will look something like this:
Note that there is a built-in splitting here showing the costs of EC2 data transfer. Click on Data Transfer and an analysis by region will appear.
Having selected a region, a breakdown by the type of transmitted data will appear (outgoing traffic, between availability zones, inter-regional).
7. CloudFront
Cloudfront's on-demand prices are rapidly decreasing as volume grows. Let's imagine the transmission of 100 Tb traffic. It will cost $ 0.060 / Gb, which is about 15% less compared to the same amount of data via ELB.
Cloud Front Private Contract & Volume Prices
Here is the best way to reduce data costs! AWS prefers to use its CloudFront service and depending on the volume, you can reduce the price of outgoing CloudFront traffic to $ 0.02 / Gb. Most likely this will require a two-year commitment, and if you plan to use AWS for a long time, then this is a great offer. To make such an arrangement, you need to contact your account manager and ask him about CloudFront Private Contract.
Amazon CloudFront SSL-Cert-Custom
Check if you are using this certificate. It costs $ 20 per day, and the same can be purchased from Cloudflare for $ 25 per month. With a Private & Volume Pricing contract, you will not be able to use other CDN (Cloud Delivery Network) services, and this action will not be available to you.
8. S3 - Simple Storage Service
A few recommendations for reducing prices for S3 storage.
S3 VPC endpoint
A great saving idea that few people are given is the use of the S3 VPC endpoint. Placing an S3 endpoint in your VPC (Virtual Private Cloud) gives any data access to S3 and gives it its own path inside the cloud, and AWS does not charge you as if it were public traffic. Depending on the volume, this technique can save up to several thousand dollars a month.
Use the correct storage model and appropriate storage class.
There are five storage models in total:
S3 standard
$ 0.023 / GB
Options:
- Failover between Availability Zones
- Low latency and high throughput
- Resistant to events affecting the entire availability zone.
S3 Infrequent Access
$ 0.0125 / GB
Options:
- Ideal for long-term storage, backups and files in case of backup recovery
- Failover between Availability Zones
- For infrequent access, as the cost of accessing files is higher
S3 One-Zone Infrequent Access
$ 0.010 / GB
Options:
- One Availability Zone
- For infrequent access
- Great choice for backup backups
S3 glacier
$ 0.004 / GB
Options:
- Safe, reliable, and inexpensive type of storage for data archives
- Failover between Availability Zones
- Data upload options can take anywhere from a few minutes to several hours
S3 Glacier Deep Archive
$ 0.00099 / GB
Options:
- Lowest price per gigabyte and support for long-term storage and digital preservation for data that may be needed a couple of times a year.
- Unloading time within 12 hours
- Failover between Availability Zones
S3 API pricing can be confusing
Please note that each time the storage class for objects changes, you will pay for the number of requests. It can be very difficult! I will share the situation that occurred in Back4App. One of our customers stores millions of small objects, and data is rarely in demand. At first, we kept all the objects in S3 Standard and decided to transfer them to S3 Glacier to save money. So far so good! But we did not take into account that AWS charges $ 0.05 for every 1000 calls to Glacier! In our case, the number of requests was huge, so the cost of moving did not pay off! Before moving objects between storage classes, it is necessary to consider the price for requests to S3 and conduct a payback analysis.
Delete unused data in S3
The easiest and most effective way to save on S3 is to simply delete old and unused data. Take a look at old backups and unused S3 bucket. Perform an analysis and, if possible, delete the data.
9. Choose the AWS Technical Support Plan Wisely
There are 4 types of technical support:
- Basic - Free
- Developer - $ 29 / month or 3% of the bill every month
- Business - $ 100 / month or 10-3% of the account, depending on the volume
- Enterprise - $ 15,000 per month
We tried the Business plan and were unsatisfied with the result. In our opinion, it is too expensive for the level of support and support that it provides. I can not say anything about the Enterprise level, since we did not use it. (The Enterprise level is suitable, for example, to large companies before launching major premieres or products.)
Engage your regional AWS manager
Use your regional manager and solution architect to the maximum! They work separately from the central support team and can provide much more suitable services. The AWS team is generally very skilled and helps a lot! The recommendations they provided on advanced security settings and the use of GDPR were simply the best! I was impressed by the technical awareness and the desire to help us achieve better results. Such support makes you think twice before changing your cloud provider.
10. Monitoring and Management
It is difficult to overestimate the importance of calculating and knowing the daily costs. You must set up a detailed monitoring of your costs for yourself and take time every day to familiarize yourself with them. Imagine how much you can save in 365 days!
Daily monitoring will help detect suspicious trends and act immediately. Best practices regarding Monitoring and Management are listed below:
- Regular monitoring
- Using reports such as:
- Billing & Cost Management Report
- Monthly bills
- Cost Explorer - Monthly Service Costs
- Cost Explorer - Using RI
- Cost Explorer - RI Coverage
- Tag everything
- Measure average, track, report exceptions
- Relate your environment to AWS's shared responsibility model
- Use CloudWatch and Trusted Advisor to Monitor Costs
- Combine your accounts and get volume discounts
11. Architecture redesign
Changing the architecture is usually the most time-consuming exercise. It can take from weeks to months, depending on the size of your architecture.
Our experience with Back4App shows that you should try the simpler steps first. In general, successive changes will bring you significant savings in the short term. Make money on such changes and gain experience for rebuilding your architecture.
12. Conclusion
There is no magic formula to immediately reduce costs in AWS! It is necessary to develop a routine habit of checking expenses, setting priorities, and working hard to achieve your goals.
Here is a brief overview of the steps that helped us reduce our AWS score.
- Delete Unused EC2 Instances
- Reduce excess power
- Move data flow to CloudFront
- Switch to new generations of instances
- Use techniques like enhanced network
- Save backups in S3
- Reserve Instances at AWS Marketplace & Elastic Cache Nodes
- Maintain a healthy mix of Spot, Reserved, and On-Demand instances
- Automate autoscaling
- Use both S3 Standard and Glacier
All Articles