Design and implementation of cloud infrastructure for optAd360, based on Kubernetes clusters and the IaC approach

Optad360 is a dynamically developing Polish company dealing with implementing solutions supporting and managing the monetization of websites, video content, and applications in terms of displayed ads using proprietary software. The modern approach to programmatic advertising and AI tools has resulted in optAd360 experiencing very intensive growth in the global online advertising markets, handling 1200 terabytes of advertising per month.

How it started

Optad360, in less than five years of activity, has acquired over 3500 websites and mobile applications for cooperation, in which it records 11 billion page views per month. That gives you 4200 ads per second for 24 hours, seven days a week. Therefore, the scale of operation is enormous, and plans for the future assume doubling the number of ads displayed around the world as early as 2022.

Since the company's inception, the Client's infrastructure has been operating in the AWS environment. However, some systems, such as Amazon Elastic Kubernetes Service (EKS), have not been updated. It was also impossible to make business decisions based on data analysis because the AWS Elasticsearch, Logstash, and Kibana (ELK) stack did not collect them correctly. Rapid implementation of subsequent solutions and systems required excellent stability and agility for optimal functioning. This initiated the process of implementing changes.

"We need to manage a massive amount of data, and with more and more customers, we've had to automate our ability to scale and acquire new customers. Rapid growth meant we needed a better test environment and DevOps. Hosters and AWS helped us build stable systems but can also scale quickly – explains Jarosław Wisłocki, co-founder and  Head of Technology Development at optAd360.

After conducting an in-depth audit of the Client's existing infrastructure, we concluded that the best solution would be to create an entirely new environment based on microservices and the IAC approach.

Infrastructure implementation

When planning the implementation of the new infrastructure for optAd360, we focused on the use of AWS EKS for the deployment of all applications, mi. In: Redis, MySQL/Postgres SQL. We created the entire infrastructure in the IaC approach, using Terraform and Helm tools. The assumption of the project was to ensure the separation of environments by their intended purpose – for technical and commercial employees.

We launched the Kubernetes cluster as a service managed by AWS, while AMAZON KMS is responsible for encrypting secrets. In  the Helm greyhound cluster, we launched m.in.:

AWS LB controller – which creates loadbalancers to handle incoming traffic

Metrics Server – aggregates metrics from cluster API

Kubernetes Dashboard – to preview mirror operations

Data Dog – for collecting data about clusters and applications

Cluster Autoscaler – for automatic scaling of nodes in the mirror depending on demand

The infrastructure has been divided into three standard environments – dev/stage/prod. Each environment has a separate, dedicated AWS account. In addition, we have also set up a tools environment that includes an S3 bucket with a  Terraform state file for all environments. We have configured Open VPN, which connects all environments and provides secure access to the network. The environment's design assumed access to the dev/stage/prod infrastructure for many developers at different permission levels. For this purpose, we used properly configured IAM roles.

To ensure security and guarantee the rapid delivery of advertising content worldwide, we have configured Amazon CloudFront. Thanks to the network of Amazon endpoints scattered around the world, we have managed almost to eliminate the problem of delays in content delivery, transferring 1.2 petabytes of data per month to recipients around the world.

Summary

For optAd360, we have implemented a completely new environment based on Kubernetes clusters and the IaC approach. Thanks to the latest solutions and best practices, it was possible to reduce the time necessary for onboarding new optAd360 customers by 50%, i.e., 2 to 4 days. Using Amazon CloudFront, we have guaranteed instant and secure transfer of a massive amount of content to recipients worldwide and the technologies used to create an excellent foundation for further intensive technological development of the company in the coming years.