Now I’ve migrated off Azure I’m looking for a log aggregation solution to replace Application Insights. The ELK Stack is popular, but the default Elasticsearch helm chart alone is much too large for my cluster. Can I decrease Elasticsearch’s footprint and have it run in my cluster?

The Helm Chart

Elastic offer a number of helm charts. Installing the Elasticsearch chart should be a simple as this:

helm install elasticsearch elastic/elasticsearch

However my cluster is tiny so the pods never get scheduled. This chart and my cluster don’t match because the chart:

  • Needs too much CPU
  • Needs too much Memory
  • Needs too many nodes

The following steps will describe my experience reducing the chart to a footprint I can run inside my cluster.

Reducing Memory

The chart configuration defaults the JVM heap size to 1GB with the parameter esJsOptions=-Xmx1g -Xms1g. We can override this, but to what? We can test on our local using the docker image directly:

MEM=96 && \
    docker rm -f elasticsearch && \
    docker run -d --name elasticsearch -p 9200:9200 \
        -e "discovery.type=single-node" \
        -e "ES_JAVA_OPTS=-Xms${MEM}m -Xmx${MEM}m" \
        elasticsearch:7.10.1

We’re expecting one of two outcomes based on different values of MEM. Either the container comes up and we can make REST calls to it:

curl -s "http://localhost:9200/_cluster/health?pretty" | grep status
  "status" : "green",

Or, the container will throw:

fatal error in thread [main], exiting
java.lang.OutOfMemoryError: Java heap space
...

Using this approach I can get the container up with as low as -Xms75m -xmx75m.

Using the Open Source Image

The default container uses Elastic’s commercial image.

These images are free to use under the Elastic license. They contain open source and free commercial features and access to paid commercial features.

We can instead use the Open Source Image. Doing so lets us reduce the heap further, and using the above approach I can raise docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2 with -Xms47m -Xmx47m.

Running in the Cluster with Helm

Now we know how low we can set the heap it’s time to have it run in the cluster via the chart. We can immediately pass --set replicas=1 to solve the node problem, but additional errors around cluster.initial_master_nodes and the need to set discovery.type=single-node prove that the chart does not support single-node. Before adding support ourselves we check PRs to see if anyone has the same idea, and they do!

Forking and cloning that repo we can see the branch uses a values.yml so we’ll use that rather than --set on helm install.

Now we can look at reducing the resources available to the pod.

resources:
  requests:
    cpu: "100m"
    memory: "512M"
  limits:
    cpu: "1000m"
    memory: "512M"

Immediately I needed to drop resources.requests.cpu down to 10m due to an existing kubernetes issue.

Also, we’ll need more memory than just what we set for heap size and Elasticsearch Heap size settings says:

It is normal to observe the Elasticsearch process using more memory than the limit configured with the Xmx setting.

Running the OSS image with -Xms47m -Xmx47m we can inspect the memory usage:

kubectl top pod -l app=elasticsearch-master
NAME                     CPU(cores)   MEMORY(bytes)   
elasticsearch-master-0   5m           215Mi

The above suggests we could set resources.limits.memory=250mi comfortably.

Reducing CPU

Whilst the command above shows CPU at 5m, it needs a lot more than that to start the container. Using the default image I saw CPU usage around 800m. I wanted to benchmark the effect of decreasing resources.limits.cpu so I first wrote a script to do it locally against the (default) container. The following script waits for the container to start listening on 9200 and outputs how many seconds it took:

MEM=96 && \
    docker rm -f elasticsearch && \
    docker run -d --name elasticsearch -p 9200:9200 \
        -e "discovery.type=single-node" \
        -e "ES_JAVA_OPTS=-Xms${MEM}m -Xmx${MEM}m" \
        elasticsearch:7.10.1 && \
    START=$(date +%s) && \
    (docker logs -t -f elasticsearch &) | grep -m1 starting && \
    echo $(($(date +%s)-START))

I then converted the docker run command to docker-compose with composerize and converted the docker-compose to a Deployment with kompose. After setting resources.limits.cpu in the resulting Deployment I converted the benchmark to work with kubectl:

kubectl apply -f elasticsearch-deployment.yaml && \
    POD=$(kubectl get pod -l app=elasticsearch -o jsonpath="{.items[0].metadata.name}") && \
    echo ${POD} && \
    kubectl wait --for=condition=ready pod ${POD} && \
    START=$(date +%s) && \
    (kubectl logs -f ${POD} &) | grep -m1 starting && \
    echo $(($(date +%s)-START))
Host CPU Limit Startup Time
localhost none 38s
cluster none 37s
cluster 500m 68s
cluster 250m 142s

As can be seen, reducing the CPU limit slows the start-up time. In my case I chose to not limit CPU so start-up could be quick.

The final values.yaml looks like this:

singleNode: true
antiAffinity: "soft"
esJavaOpts: "-Xmx47m -Xms47m"
image: docker.elastic.co/elasticsearch/elasticsearch-oss
resources:
  requests:
    memory: "100Mi"
    cpu: "10m"
  limits:
    memory: "250Mi"
    cpu: "800m"
volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 1Gi

I install the chart using the (currently unmerged) branch mentioned above.

helm install elasticsearch ./helm-charts/elasticsearch --values ./path-to-my/values.yaml

Increasing Stability

On default image, and at the lower limit of heap space, making REST calls will constantly fail with HTTP Status Code 429. To reduce this I’ll bump the heap to 96m instead of 75m.

{
   "error":{
      "root_cause":[
         {
            "type":"circuit_breaking_exception",
            "reason":"[parent] Data too large, data for [<http_request>] would be [76010944/72.4mb], which is larger than the limit of [75707187/72.1mb], real usage: [76010944/72.4mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=752/752b, model_inference=0/0b, accounting=0/0b]",
            "bytes_wanted":76010944,
            "bytes_limit":75707187,
            "durability":"TRANSIENT"
         }
      ],
      "type":"circuit_breaking_exception",
      "reason":"[parent] Data too large, data for [<http_request>] would be [76010944/72.4mb], which is larger than the limit of [75707187/72.1mb], real usage: [76010944/72.4mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=752/752b, model_inference=0/0b, accounting=0/0b]",
      "bytes_wanted":76010944,
      "bytes_limit":75707187,
      "durability":"TRANSIENT"
   },
   "status":429
}

Conclusion

The open source image will run with a heap size of 47m and uses about 215m on the cluster. The standard image will run at 75m heap - which results in a lot of 429s - so instead running at 96m heap results in about 314m on the cluster.