Kubernetes offers a wide range of functionalities to manage containerized applications and create complex distributed systems. One such utility is to automatically scale the number of replicas in a deployment when the workload increases. In this post we’ll see a simple example of how to use Kubernetes’ Horizontal Pod Autoscaler to dynamically adjust the number of workers that are consuming a task queue in a Flask + Celery app.

Write the Celery app

Celery is a framework to build software based on task queues. We’ll deploy a system composed by a Flask frontend web server that schedules tasks when receiving connections, a Redis server that acts as queue broker, and a set of workers that consume the tasks asynchronously. The number of workers will be decided by Kubernetes based on the workload at each time.