A lesson learned on GRPC Client side load balancing
Background This post walks through the steps I took to debug load balancing issues for a client/server written in Go, using gRPC and running in Kubernetes. This post starts by identifying the problem, attempting to fix it, realizing why that didn't actually fix it, and then finally understanding how gRPC load balancing was working with the Kubernetes service. It's kind of weird because rather than a nice issue/solution format, I walk you through my process. So if you're into that, great! If not, just go to the last part to figure out the main points. Anyways, I was given the task of writing a load tester for one of our microservices that takes in a request for a certain type of product recommendation and returns the products. We wanted to hit the service at ~60k requests/second. To do this, I built a load tester in Go that uses gRPC to talk to the service. The load tester is comprised of a client that is called locally from my laptop and port-forwards to the attacker p
