In this post, we will explore the Spotinst platform and review how it solves these challenges with Serverless Containers.<\/p>\n
Spotinst\u00a0<\/a><\/strong>is an\u00a0AWS Partner Network<\/a>\u00a0(APN) Advanced Technology Partner with the\u00a0AWS Container Competency<\/a>. Spotinst helps companies automate cloud infrastructure management and save on their AWS computing costs by leveraging\u00a0Amazon EC2 Spot Instances<\/a>\u00a0with ease and confidence.<\/p>\n To make the most cost-effective use of the cloud, users strive to achieve a flexible infrastructure that will scale up or down based on resource requirements such as CPU or memory utilization.<\/p>\n While natively scaling the application layer, Kubernetes does little to manage the underlying infrastructure layer.<\/p>\n As like most container orchestrators, it was designed with a physical data center in mind and assumes the capacity is always available as applications scale up and down.<\/p>\n Let\u2019s take a closer look at the challenges business face when building optimized, cost-effective, and highly available infrastructure on AWS.<\/p>\n With three available pricing models on AWS\u2014On-Demand, Reserved, and Spot\u2014we need to decide which model works best for every part of the workload.<\/p>\n Kubernetes does a great job with resilience and handling the interruption or replacement of servers. This makes it a great candidate to leverage Spot instances at a 70 percent discount compared to On-Demand prices.<\/p>\n Then again, it\u2019s important to maintain some persistent instances for certain workloads as well as cost predictability. That\u2019s where buying a pool of\u00a0Amazon EC2 Reserved Instances<\/a>\u00a0(RIs) ensures a baseline capacity that is cost effective.<\/p>\n The second and most complex challenge is choosing the right infrastructure size and type to satisfy the actual containers requirements.<\/p>\n Should we stick to one instance type? Should we mix families or instance sizes? What should we do when a GPU-based pod comes up to do a quick job a few times a day?<\/p>\n Traditionally, the answer to all of these questions has been, \u201cIt varies.\u201d As time goes on, clusters grow and more developers freely push new and differing containers into Kubernetes, resulting in constantly changing answers to the questions above.<\/p>\n Moreover, we need to figure out how to make sure it all keeps scaling and addressing changes in application requirements. To illustrate this point, consider that some containers require 0.2vCPU and 200MB of memory, and others require 8vCPU and 16GB of memory or more.<\/p>\n As a result, setting the right auto scaling policy based on simple CPU or memory thresholds rarely stays reliable in the long term.<\/p>\n The third and least tackled challenge is the containers utilization. This involves constantly adapting the limits and resource requests of the containers based on their consumption and utilization.<\/p>\n Even after solving for the first two issues and successfully using the right instance sizes for our containers, how do we know the container wasn\u2019t over-allocated resources?<\/p>\n Getting a pod of 3.8vCPU and 7.5GB of memory on a c5.xlarge instance is great as it puts the instance at nearly 100 percent allocation, but what if that pod only uses half of that? That would mean half of our instance resources are wasted and could be used by other containers.<\/p>\n Successfully solving for all of these issues can cut costs and free developer time by having infrastructure and containers that self-optimize in real-time based on application demands.<\/p>\n Spotinst built a DevOps automation platform that helps customers save time and money.<\/p>\n The core of the Spotinst platform is a sophisticated infrastructure scheduling mechanism that is data-driven. It allows you to run production-grade workloads on Spot instances, while utilizing a variety of instance types and sizes. Spotinst offers an enterprise-grade SLA on Spot Instances while guaranteeing the full utilization of RIs.<\/p>\n On top of this automation platform, Spotinst built\u00a0Ocean<\/a><\/strong>, a Kubernetes pod-driven auto scaling and monitoring service. Ocean automatically and in real-time solves the challenges we discussed earlier, while creating a serverless experience inside your AWS environment, and it does so without any access to your data.<\/p>\n Additionally, Ocean helps you right-size the pods in your cluster for optimal resource utilization.<\/p>\n Spotinst solves the pricing and instance sizing challenges by using dynamic container-driven auto scaling based on flexible instance size and life cycles.<\/p>\n There are three main components at play here: dynamic infrastructure scaling, pod rescheduling simulations, and cluster headroom. This is how they work.<\/p>\n Spotinst Ocean allows customers to run On-Demand, Spot, and Reserved Instances of all types and sizes within a single cluster. This flexibility means the right instance can be provisioned when it\u2019s needed, while guaranteeing minimum resource waste.<\/p>\n Whenever the Kubernetes Horizontal Pod Autoscaler (HPA) scales up, or a new deployment happens, the Ocean auto scaler reads the overall requirements of that event. These include:<\/p>\n After aggregating the sum of these requests, Ocean calculates the most cost-effective infrastructure to answer the request. This can be anywhere from a single t2.small to a combination of several M5.2xlarge, C5.18xlarge, P3.xlarge, R4.large, or anything in between.<\/p>\n The use of dynamic instance families and sizes helps gain a highly defragmented with high resource allocation cluster as it grows and scales up.<\/p>\n This dynamic infrastructure scaling ensures maximum resource allocation as we scale up and grow our clusters.<\/p>\n To make sure the allocation is maintained when Kubernetes scales the pods down as application demands decrease, Ocean runs pod rescheduling simulations every 60 seconds to figure out if the cluster can be further defragmented.<\/p>\n By considering all the requirements, Ocean checks for any expendable nodes. These can have their pods drained and the node removed from the cluster without having any of the pods go into a pending state, and get scheduled to other existing nodes in the cluster.<\/p>\n Figure 1 \u2013 Pod rescheduling simulation.<\/em><\/p>\n In\u00a0Figure 1<\/em>, you can see the pod rescheduling simulation when scaling down expendable nodes, considering pod and instance resources as well as other restrictions. In this example, we were able to reduce the cluster size by about 10 percent pre-scale versus post-scale CPU and memory.<\/p>\n Scaling the infrastructure when it\u2019s needed can achieve maximal node resource allocation, but may also slow the responsiveness of our application.<\/p>\n While high availability is critical, waiting for new nodes to spin up every time capacity is needed can lead to less-than-optimal latency and other performance issues. This is where the cluster headroom comes in. Headroom is a capacity of available units of work in the sizes of the most common deployments that stay available in order to allow Kubernetes to instantly schedule new pods.<\/p>\n Headroom is automatically configured by Ocean based on the size of each deployment in the cluster, and changes dynamically over time.<\/p>\n In practice, effective headroom means that more than 90 percent of scaling events in Kubernetes instantly get scheduled on the cluster into the headroom capacity, combined with a call for a headroom scaling event to be prepared for the next time Kubernetes scales.<\/p>\n In case an abnormal deployment happens that requires more than the available headroom, the cluster auto scaler handles the scaling events with a single scaling event of varying instance types and sizes to cover the entire deployment resource requirements. This helps Ocean achieve a 90 percent resource allocation in the cluster while staying more responsive than a traditional step scaling policy.<\/p>\n Once the pods are placed on the instances, the Ocean Pod Right-Sizing mechanism will start monitoring the pods for CPU and memory utilization. Once enough data is aggregated, Ocean starts pushing recommendations on how to right-size the pod.<\/p>\n For example, if a pod was allocated with 4GB of memory and over a few days its usage is between 2GB and 3.4GB , with an average of 2.9GB, Ocean Right-Sizing will recommend sizing the pod at ~3.4GB, which further helps\u00a0bin pack<\/a>\u00a0and defragment the cluster while keeping it highly utilized.<\/p>\n Figure 2 \u2013 Deployment with Amazon EKS best practices.<\/em><\/p>\n In\u00a0Figure 2<\/em>, you can see the deployment architecture of Ocean +\u00a0Amazon Elastic Kubernetes Service<\/a>\u00a0(Amazon EKS) where EKS managed the Kubernetes control pane and Ocean the worker nodes. The Amazon EKS architecture and security stays in place, and Ocean wraps around the worker nodes to dynamically orchestrate them.<\/p>\n Figure 3 \u2013 Deployment with any container orchestrator on AWS.<\/em><\/p>\n In\u00a0Figure 3<\/em>, you see the general Ocean architecture with the worker nodes connecting to one of the supported container orchestrators. On the left side, you can see the variety of container management control planes and on the right, Ocean wrapping around the worker nodes.<\/p>\n Deploying Spotinst Ocean for your existing Kubernetes implementation is simple and can be done in just a few minutes using your preferred deployment method, such as\u00a0Amazon CloudFormation<\/a>,\u00a0Terraform<\/a>,\u00a0Ansible<\/a>,\u00a0RESTful API<\/a>, or the\u00a0Spotinst UI<\/a>.<\/p>\n Ocean orchestrates within your AWS environment and leverages existing cluster components, including virtual private clouds (VPC), subnets,\u00a0AWS Identity and Access Management<\/a>\u00a0(IAM) roles, images, security groups, instance profiles, key pairs, user data, and tags.<\/p>\n In a nutshell, Ocean will fetch all of the existing configurations from your cluster to build the configuration for the worker nodes that it will spin up. Then, all you have to do is deploy the Spotinst Kubernetes Cluster Controller into your cluster so that Ocean can get the Kubernetes metrics reported to it.<\/p>\n Once this is done, Ocean will handle all infrastructure provisioning for you going forward. You should be able to give your developers complete freedom to deploy whatever they choose.<\/p>\n Let\u2019s walk through a set up so you can see how simple it is.<\/p>\n Now that we know what Ocean is and how to deploy it, let\u2019s look at some of the numbers and benefits of real usage scenarios, and how using dynamic infrastructure can help.<\/p>\n Example #1: Pod requires 6,500mb of memory and 1.5vCPUs \u2013 Different Instance Size<\/strong><\/p>\n Running five such pods on m5.large or m5.xlarge instances will cost $0.096 per hour per pod. However, going up to 2XL will allow further bin-packing and reduce the cost to $0.077 per hour per pod.<\/p>\n As seen in the chart above, by going a few sizes up, we can achieve better resource allocation and reduce the cost by 20 percent.<\/p>\n Example #2: Pod of 6500mb of memory and 1.8vCPUs \u2013 Different Instance Family<\/strong><\/p>\n Running replicas of such pods on c5.xlarge will only allow for one pod per instance, while an m5.xlarge will allow for two and reduce the cost from $0.17 per hour per pod to $0.096 per hour per pod.<\/p>\nChallenges<\/h3>\n
Pricing<\/h3>\n
Instance Sizing<\/h3>\n
Containers Utilization<\/h3>\n
Spotinst Automation Platform<\/h3>\n
Dynamic Infrastructure Scaling<\/h3>\n
\n
Pod Rescheduling Simulations<\/h3>\n
![\"Spotinst-Serverless-1\"](\"https:\/\/d2908q01vomqb2.cloudfront.net\/77de68daecd823babbb58edb1c8e14d7106e83bb\/2019\/10\/08\/Spotinst-Serverless-1.png\")
<\/a><\/p>\nCluster Headroom<\/h3>\n
Monitoring CPU and Memory Utilization<\/h3>\n
![\"Spotinst-Serverless-2\"](\"https:\/\/d2908q01vomqb2.cloudfront.net\/77de68daecd823babbb58edb1c8e14d7106e83bb\/2019\/10\/08\/Spotinst-Serverless-2.png\")
<\/a><\/p>\n![\"Spotinst-Serverless-3\"](\"https:\/\/d2908q01vomqb2.cloudfront.net\/77de68daecd823babbb58edb1c8e14d7106e83bb\/2019\/10\/08\/Spotinst-Serverless-3.png\")
<\/a><\/p>\nImplementation Walkthrough<\/h3>\n
\n
![\"Spotinst-Serverless-4\"](\"https:\/\/d2908q01vomqb2.cloudfront.net\/77de68daecd823babbb58edb1c8e14d7106e83bb\/2019\/10\/08\/Spotinst-Serverless-4.png\")
<\/a><\/p>\n\n
![\"Spotinst-Serverless-5\"](\"https:\/\/d2908q01vomqb2.cloudfront.net\/77de68daecd823babbb58edb1c8e14d7106e83bb\/2019\/10\/08\/Spotinst-Serverless-5.png\")
<\/a><\/p>\n\n
![\"Spotinst-Serverless-6\"](\"https:\/\/d2908q01vomqb2.cloudfront.net\/77de68daecd823babbb58edb1c8e14d7106e83bb\/2019\/10\/08\/Spotinst-Serverless-6.png\")
<\/a><\/p>\n\n
![\"Spotinst-Serverless-7\"](\"https:\/\/d2908q01vomqb2.cloudfront.net\/77de68daecd823babbb58edb1c8e14d7106e83bb\/2019\/10\/08\/Spotinst-Serverless-7.png\")
<\/a><\/p>\n\n
![\"Spotinst-Serverless-8\"](\"https:\/\/d2908q01vomqb2.cloudfront.net\/77de68daecd823babbb58edb1c8e14d7106e83bb\/2019\/10\/08\/Spotinst-Serverless-8.png\")
<\/a><\/p>\n\n
\n.<\/li>\nOptimization Data<\/h3>\n
![\"Spotinst-Serverless-9\"](\"https:\/\/d2908q01vomqb2.cloudfront.net\/77de68daecd823babbb58edb1c8e14d7106e83bb\/2019\/10\/08\/Spotinst-Serverless-9.png\")
<\/a><\/p>\n
![\"Spotinst-Serverless-10\"](\"https:\/\/d2908q01vomqb2.cloudfront.net\/77de68daecd823babbb58edb1c8e14d7106e83bb\/2019\/10\/08\/Spotinst-Serverless-10.png\")
<\/a><\/p>\n