In our previous article, we implemented a server in Golang that gracefully reacts to termination signals. We presented the problems of the hello world version of the server, how to address it, but did not perform any verification. In this article, we’ll prepare an experiment on both the graceful server, and also the not-so-graceful one to contrast their behavior. The experiment is done in a macOS environment, but the same process will work in Windows and Linux.

Preparation

To simulate Kubernetes, we can use minikube. Firstly, we need to install it.

brew install minikube

Then we need to start it.

minikube start

Verify that everything is right by checking our new cluster. This command will download kubectl if it’s not installed yet, and return the list of pods in all namespaces.

minikube kubectl -- get po -A

We have to enable ingress addon in this experiment.

minikube addons enable ingress

At this point our environment setup is ready. We can checkout the source code we’re going to experiment with.

git clone git@github.com:xamenyap/go-server-demo.git

Inside the repository are 2 Docker files:

  • default.Dockerfile builds the image of the default HTTP server that stops abruptly upon receiving termination signals.
  • graceful.Dockerfile builds the image of the graceful HTTP server that handles termination signals correctly.

We can push these images to a remote Docker image registry, or we can push them to minikube docker environment. In this experiment, we choose the latter for the sake of simplicity. Open a new terminal window, and run this command to make use of minikube docker environment.

eval $(minikube docker-env)

Now, all docker commands, like docker build, will be executed in this minikube docker environment instead of our usual local docker environment. We can start building 2 Docker images for the 2 applications used in the experiment.

docker build . -f default.Dockerfile -t default-go-server:v1.0
docker build . -f graceful.Dockerfile -t graceful-go-server:v1.0

Next, we have to deploy our applications to minikube environment. In this experiment we can achieve it by applying the already provided file minikube.yaml presented in the demo.

minikube kubectl -- apply -f minikube.yaml

Under the hood, minikube.yaml provides the following:

  • A deployment definition for the default HTTP server
  • A deployment definition for the graceful HTTP server
  • A service definition to expose the default HTTP server externally
  • A service definition to expose the graceful HTTP server externally
  • An nginx ingress to route traffic to the services

This article will not cover the details of those, but it’s a good idea to understand what the file provides by checking Kubernetes documentation. In the next part, we’ll carry out the experiment.

Experiment

Before we can make any requests from our host machine to minikube cluster, we need to make a tunnel. Open a new terminal window and run this command.

minikube tunnel

Leave that terminal open, and use another terminal for the rest of the tutorial.

Our applications are ready to serve HTTP requests now, we can simulate a production environment by repeatedly sending them traffic. One tool we can use here is vegeta. We start with the default HTTP server.

echo "GET http://127.0.0.1/default/hello" | vegeta attack -duration=20s -rate=200 | tee results.bin | vegeta report

This script will run a load test against the default HTTP server in 20s with a rate of 200 requests/sec, then collect the data and print it in a format that we can examine. While waiting for the load test script, open another terminal window and run this command. Make sure that the command runs within 20s of load test.

minikube kubectl -- get po -lapp=default  -o name | head -n 1 | xargs minikube kubectl -- delete

This will delete the first pod of the default replica set that we initiated in the previous section. We do this to simulate the event when kubernetes sends a termination signal to our container. If we do everything correctly, what we get from the load test terminal window after 20 seconds will be similar to this.

Requests      [total, rate, throughput]         4000, 200.05, 197.84
Duration      [total, attack, wait]             20.198s, 19.995s, 203.121ms
Latencies     [min, mean, 50, 90, 95, 99, max]  15.429ms, 270.654ms, 202.03ms, 205.95ms, 206.171ms, 5.206s, 5.206s
Bytes In      [total, mean]                     44556, 11.14
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           99.90%
Status Codes  [code:count]                      200:3996  502:4
Error Set:
502 Bad Gateway

We have 3996 requests succeeding, and 4 requests failing because of the deleted pod while it was still processing them. There is a chance that some requests failed due to exceeding the maximum number of open files, which we can simply ignore. The key point here is the 4 requests with HTTP status code 502, which indicates that the response is malformed and the nginx ingress does not know how to interprete it. Depending on our environment, sometimes we can observe more or fewer than 4 failing requests, even none. Repeating this test a few times and surely we’ll catch some 502.

We can proceed to do the same test against the graceful HTTP server. Initiate the load test by the command below.

echo "GET http://127.0.0.1/graceful/hello" | vegeta attack -duration=20s -rate=200 | tee results.bin | vegeta report

While the load test is running, delete the first pod of the graceful relica set to simulate the event when kubernetes sends a termination signal to the container. Make sure the following command runs within 20s of the test.

minikube kubectl -- get po -lapp=graceful  -o name | head -n 1 | xargs minikube kubectl -- delete

If we do things correctly, the output may look like below.

Requests      [total, rate, throughput]         4000, 200.05, 198.04
Duration      [total, attack, wait]             20.198s, 19.995s, 203.616ms
Latencies     [min, mean, 50, 90, 95, 99, max]  200.672ms, 202.644ms, 201.899ms, 205.801ms, 205.969ms, 207.177ms, 239.141ms
Bytes In      [total, mean]                     44000, 11.00
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:4000
Error Set:

As can be seen in the report, all requests succeeded even though we delete a pod during the execution of the load test. The HTTP server is working correctly, handling in-flight requests even when it receives the termination signal.