Image for post
Image for post

We have a Jenkins instance that is running jobs in Docker containers on its host.

Eventually, we’ve faced with an issue when the current AWS Ec2 instance t2.2xlarge (8 CPU, 32 RAM) during peak workload periods was too overloaded — not enough CPU time, not enough memory.

So, the first solution could be to proceed with its vertical scaling for example up to c5.9large, and proceed running builds on the master-host, or by moving some jobs to external workers.

At this time, we have three such workers — Android-builds are running on a PC in our office with Android studio…

Image for post
Image for post

We have a PHP application running with Kubernetes in pods with two dedicated containers — NGINX и PHP-FPM.

The problem is that during downscaling clients get 502 errors. E.g. when a pod is stopping, its containers can not correctly close existing connections.

So, in this post, we will take a closer look at the pods’ termination process in general, and NGINX and PHP-FPM containers in particular.

Testing will be performed on the AWS Elastic Kubernetes Service by the Yandex.Tank utility.

Ingress resource will create an AWS Application Load Balancer with the AWS ALB Ingress Controller.

Для управления контейнерами на Kubernetes…

Image for post
Image for post

Besides the Apache Bench and JMeter there is another utility — Yandex Tank.

It’s used by our QA team and now it’s time for me to take a closer look on it to test one issue with our application running on a Kubernetes cluster.

In this post a short overview of its capabilities and configuration.

In contrast to the Apache Bench, Yandex.Tank displays response codes statistics and is much more simple in running and configuration the JMeter, plus it has a nice Autostop feature for a case when “Huston, we have a problem” (с)


See Modules.

The Yandex Tank core…

Image for post
Image for post

Locales is a set of environment variables that are used to determine how to display data and time (for example, first of the week), symbols encoding (for example, how to display cyrillic symbols), default files order when one executing the ls command, and so on.

Those variables are:

  • LANG: Determines the default locale in the absence of other locale related environment variables
  • LANGUAGE: List of fallback message translation languages
  • LC_CTYPE: Character classification and case conversion
  • LC_NUMERIC: Numeric formatting
  • LC_TIME: Date and time formats
  • LC_COLLATE: Collation (sort) order
  • LC_MONETARY: Monetary formatting
  • LC_MESSAGES: Format of interactive words and responses
  • LC_PAPER: Default paper…

Image for post
Image for post

We are using to collect our Kubernetes cluster logs (also, there is a local Loki instance).

Logs are collected and processed by a Fluentd pod on every WorkerNode which are deployed from a DaemonSet in its default configuration, see the documentation here — logzio-k8s.

The problem we faced is that those pods are consuming too much CPU — up to 3000 millicpu, while our WorkerNodes has only 4 cores, e.g. 4000 millicpu.

So, to solve this issue I’ve decided to search for similar log collectors and the second thing to do is was to able to deploy them with…

Image for post
Image for post

Active alerts sending frequency via Alertmanager is configured via the repeat_interval in the /etc/alertmanager/config.yml file.

We have this interval set to 15 minutes, and as result, we have notifications about alerts in our Slack each fifteen minutes.

Still, some alerts are such a “known issues”, when we already started the investigation or fixing it, but the alert is repeatedly sent to Slack.

To mute those alerts to prevent them to be sent over and over they can be disabled by marking them as “silenced”.

An alert can be silenced with the Web UI of the Alertmanager, see the documentation.


Image for post
Image for post

ArgoCD helps to deliver applications to Kubernetes by using the GitOps approach, i.e. when a Git-repository is used as a source of trust, thus all manifest, configs and other data are stored in a repository.

It can b used with Kubernetes manifest, kustomize, ksonnet, jsonnet, and what we are using in our project — Helm-charts.

ArgoCD spins up its controller in the cluster and watches for changes in a repository to compare it with resources deployed in the cluster, synchronizing their states.

Some additional features which are really useful are SSO with SAML, so we can integrate it with our…

Image for post
Image for post

For the authentification and authorization, Kubernetes has such notions as User Accounts and Service Accounts.

User Accounts — common user profiles used to access a cluster from the outside, while Service Accounts are used to grant access from inside of the cluster.

ServiceAccounts are intended to provide an identity for a Kubernetes Pod to be used by its container to authenticate and authorize them when performing API-requests to the Kubernetes API-server.


Default ServiceAccount

Every Kubernetes Namespace has its own default ServiceAccount…

Image for post
Image for post

We’d like to have the ability to add a DNS-record on the AWS Route53 when a Kubernetes Ingress resource is deployed and point this record to the URL of an AWS Load Balancer which is created by the ALB Ingress controller.

To achieve this, the ExternalDNS can be used which will make API-requests to the AWS Route53 to add appropriate records.

AWS installation is described in its documentation>>>.


AWS set up

IAM Policy

First, need to create an IAM policy. For the…

Arseny Zinchenko (setevoy)

DevOps engineer.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store