Scaling Production Globally — The service mesh facelift (Part-1)
Achievers has been growing rapidly over the past two years, and we have been working hard to grow our microservices and scale our platform globally.
We have been running Istio in production for well over 18 months and we suffered and learned a lot during this process. I thought it would be interesting to share some stories about scaling around the world using Istio and Google Kubernetes Engine.
The Architecture
Before we start going over the issues, I wanted to take a moment to give some context on the architecture we wanted to achieve.
The challenge for the SRE team was to improve availability and scale to new regions while meeting strict data residency requirements. Kubernetes clusters already existed however, there was no communication between regions for failover. The chance of a cluster or single microservice going down in a region is higher than that of losing all our GCP sites entirely. It was our goal to boost availability for our customers even during a regional outage.
I kept a list of some of the service mesh requirements the team cooked up:
- Support HTTP2 and HTTP
- Authentication/authorization handling
- Communication authentication (mTLS)
- Locality failover
- Support for multi-cluster
- Traffic control and security policies
- Satisfy our data residency requirements
- Great observability tools and metrics
To create the globally distributed system the team desired, our Kubernetes clusters would need a service mesh facelift.
Migration
Great, now that we have our requirements I now can take you through the migration and challenges we encountered along the way.
When Achievers first started using Kubernetes in 2019, we used Kong as an API gateway. Since the team was already familiar with the Kong product, we decided to investigate their service mesh Kuma first. Kuma was simple to configure, but we felt it wasn’t mature enough at the time for our multi-cluster needs. Linkerd2 was also considered, however (again, at the time) multi-cluster was still new and there weren’t a lot of documented cases for locality. We ultimately chose Istio for the service mesh due to advancements in Envoy, multi-cluster, and their full gateway features. Our ingress requirements were simple, so we decided to scrap Kong altogether and use Istio for a gateway as well as a service mesh.
The process of enabling sidecars and virtual services alongside the legacy Kong gateway was a piece of cake. Once serious usage ramped up in production, a few issues came to light.
Second-class sidecar citizens
When migration was completed in late 2020, developers noticed pods intermittently crashed during startup. It became apparent that the issue was limited to services that required outbound calls on pod startup. Some containers in a pod attempted to start before the istio-proxy container was ready which caused outbound requests to fail. The issue with sidecar containers in Kubernetes is that you have no guarantee on the order of execution.
To solve this issue we set the holdApplicationUntilProxyStarts cluster configuration for all services. This allows for applications to delay their startup while the istio-proxy becomes ready to accept traffic.
Limits, Limits, LIMITS
Another side effect of the migration was horizontal pod autoscaling values became skewed due to the addition of the Istio sidecar container. Some pods would scale often or not at all. In other cases, Istio would push configuration updates to the envoy proxies and multiple services would scale together. Horizontal pod autoscaling uses the sum or average of all container CPU/Memory requests for a pod when determining when to scale. Adding a sidecar to a pod alters the total amount of CPU and memory, skewing the scaling values.
The solution was to review service CPU and memory usage with the sidecar enabled and then reconfigure targetCPUUtilizationPercentage on the Horizontal Pod Autoscaler.
503, the Envoy wet blanket error
After a few months in production, one engineer came to me asking for assistance troubleshooting an ongoing headache. Their application began seeing frequent503 UC upstream_reset_before_response_started errors once they migrated to Istio. Troubleshooting 503s can be tricky in Envoy because they are treated as a “blanket error” and can cover a wide range of issues. The SRE team jumped in to take a deeper dive into the problem.
Get Stefan Kolesnikowicz’s stories in your inbox
Join Medium for free to get updates from this writer.
First, we looked at Kubernetes for some help. Setting terminationGracePeriodSecond allows for extra time before fully terminating pods. The assumption was maybe the application needs more time to drain connections. The value was set pretty high to test with but still no success with reducing 503s.
Next, we thought maybe connection pooling on the destination rule was misconfigured and opted to tweak maxRequestsPerConnection to disable keepalives. We played with all the options available but determined Istio was configured correctly.
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: my-app-destinationrule
namespace: default
spec:
host: my-app
trafficPolicy:
loadBalancer:
simple: LEAST_CONN
connectionPool: # Timeouts and connection settings
tcp:
maxConnections: 10
connectTimeout: 1000ms
http:
http1MaxPendingRequests: 10
http2MaxRequests: 100
maxRequestsPerConnection: 10
maxRetries: 5
outlierDetection:
consecutiveErrors: 3
interval: 30s
baseEjectionTime: 60s
subsets:
- name: v1
labels:
version: "v1"At this point, it had to be in the application itself. The SRE team dug deeper and found Nodejs does not pass termination signals to child processes and therefore never receives a SIGTERM from Kubernetes.
process.on('SIGTERM', () => {
console.info('SIGTERM signal received.');
console.log('Closing http server.'); setTimeout(async () => {
await server.close();
console.log('Server Stopped!');
}, 10000);
});
This small code block, along with introducing an init process managed to stop the 503 errors and resolve the teams' reliability problem.
Troubleshooting Istio 503s can be overwhelming, to say the least.
ET can’t phone home!
With deprecation day for PodSecurityPolicies coming soon, the SRE team decided to upgrade Istio to benefit from the new Container Network Interface (CNI) feature. The Istio CNI plugin replaced the istio-init container that performs the same networking functionality but without requiring Istio users to enable elevated Kubernetes RBAC permissions. After upgrading Istio we began seeing some containers unable to communicate to the Google Metadata service causing pods to crash.
The CNI allows you to bypass a list of IP ranges to exclude from the Istio sidecar. Adding the excludeOutboundIPRanges annotation to the Kubernetes pod yaml resolved our container startup issue.
Out of our hands 🙏
As more services were deployed over the year, our Google Kubernetes Engine (GKE) clusters took on more load and resources. This brought a new GKE issue to light. Suddenly, a large number of failure alerts on gke-metric-agent pods began, and horizontal pod autoscaling intermittently stopped working for microservices. Upon investigation, we saw out-of-memory(OOM) events from GKE-metrics-agent pods. Since this deployment is managed by Google, increasing the pod limits was out of our direct control. Hoping this would be an easy fix, we reached out to Google Support asking them to bump the memory limits on the pod spec.
What we heard back was disheartening. The Google support team is unable to change limits on managed deployments. On top of that, we hit a bug. Larger type machine clusters running a vast number of pods will run into performance issues and crash the metrics server. GKE has a limit of 110 pods per worker, so we thought we were fine at around ~64 pods.
At this point, we had two options:
1. Drop pods per worker (and decrease machine type)
2. Investigate deploying metrics manually and disable Google monitoring API on the cluster
It was decided to drop the pods per worker node to hopefully stop the out-of-memory errors. We heard rumbles online about the magic number being 30 pods for a single worker. After dropping the number to 30 from 64, the metrics server stopped crashing but still received high CPU alerts. The downside of the workaround was that our node count jumped significantly. Even with the smaller machine type, it will now cost more for disks and traffic costs. The issue was eventually solved upstream by Google.
Conclusion
As a final note, throughout our journey, we learned that troubleshooting distributed systems can be overwhelming when you first start. It can take some time to understand how to visualize and leverage the abundance of metrics, and that’s why having proper observability is critical to the success of your multi-region service mesh.
Next month I’ll share more about how Achievers use metrics, monitoring, and overall observability of multi-cluster Istio to solve problems for engineers.
—
Have you had a similar experience as us?
We would love to hear your stories in the comments.
Oh, and if you like what you see? We are hiring A-Players to join the team!
