Nationwide call center, based on Amazon Web Services
At the request of one of our Customers, we created a call center system with a nationwide range, capable of handling up to 50,000 calls per hour, whose aim was to provide the information indicated by the Customer. The first working version of the system was made available after several hours from the start of the project, using Amazon Web Services with particular attention to AWS Connect and AWS Lambda services.
Problem
Our client's organization, along with subordinate structures, faced the challenge of implementing nationwide IT solutions for all residents of the country. The Customer needed to immediately launch a fully automated call center with high efficiency, which could replace manual information activities, performed so far by employees of the structures subordinate to the Employer, and other activities so far impossible to perform due to time, personnel and technological limitations.
The key requirements of the client came down to the following functions:
- Calling the number base transferred daily with a specific message according to a specific pattern of behavior and repeating,
- Automation of the process of sending behavioral instructions to selected persons
- Possibility of fast implementation of the solution (24h) and changing the paths within just a few hours,
- Reporting and data analytics on the basis of calls made and responses collected from patients.
Implementation
Hostersi undertook a difficult task of creating a call center system based on the infrastructure and powerful tools of Amazon Web Services with particular emphasis on AWS Lambda and AWS Connect. Thanks to the availability of AWS tools, we completed the implementation within a dozen or so hours, refining and increasingly automating it in subsequent iterations. This allowed for instant offloading of the client's employees and outreach activities on an unprecedented scale. The designed call center also allowed for additional identification of people who could potentially be interested in the information provided by the client.
Description of the system architecture and operation
The solution was created entirely based on Amazon Web Services and "serverless" architecture, i.e. built on ready-made components and cloud services operating in the PaaS model. Mechanisms "serverless" enabled the implementation and deployment of the solution in just over an hour, and obtaining full automation of flows and processes in the application took several weeks. After about 2-3 weeks, the system became "self-service" and its control was reduced to a random check of the accuracy of the processed data of people to whom phone calls with specific information were made. The automat, created on the basis of AWS Connect, also asked the caller questions during the voice message, and the options selected by the caller were recorded and entered into the database, being an element of information in generated reports. It's worth mentioning that changing the content of messages and their implementation in the production environment took a few minutes, also changes in the logic of their presentation is usually a matter of several minutes.
Performance
In the first weeks of the project launch, it was performance that proved to be one of the key challenges for the entire system. High expectations for the number of connections made very quickly led to a situation where we reached the limits of the standard AWS account limits for concurrent AWS Lambda function calls and concurrent connections for Amazon Connect. These limits, as an AWS Advanced Consulting, we were able to increase to the following values:
- 5,000 - limit of concurrent Lambda function calls
- 2,700 - limit of concurrent connections for Amazon Connect
By increasing the limits, the application was able to handle 2,700 simultaneous connections. However, the demand was not that high and in practice the system worked on average with 400-800 simultaneous connections. With this number of simultaneous connections and the length of the scheme defined in Amazon Connect, we serviced on average about 10 000 - 20 000 people per hour. In our case, the deployed infrastructure allowed us to serve a maximum of 50,000 people per hour.
Implementation of microservices as a feature of AWS Lambda
Microservices were implemented as AWS Lambda functions based on events ("event-driven") or run according to a specific schedule ("scheduled", "scheduledriven"). The source of events was usually the "contact flow" of the Amazon Connect service or the Step Functions service. AWS Lambda functions executed the key business logic of the system including:
- They controlled the execution of Amazon Connect service's "contact-flow", i.e. the process of calling customers, based on data retrieved from the database (DynamoDB)
- Recording customer responses from Amazon Connect, which were the basis for further data processing in the form of generated reports or were associated with additional customer service, e.g. making another call
- Process the "raw" data provided by the customer and transform it into the form required by the call mechanism
- Generated reports on the basis of collected responses from the clients
- The client did not raise any objections or doubts related to the use of AWS Lambda and the "serverless" approach.
Security
The whole solution has been very "hermetically" embedded in AWS cloud environment - there are no external endpoints, which should be particularly protected against attacks. As the solution is based on "serverless" architecture, the entire effort of managing the physical infrastructure, virtualization layer, and operating system updates rests with the cloud provider. All data at rest ("at rest"), as well as during the process of copying or transferring ("in transit") from the Employer's local data center to AWS are encrypted. Access to the cloud environment itself, to the so-called AWS account, is particularly protected - only selected persons with appropriate permissions can access it, but also only after prior two-factor authentication ("MFA") using a physical or virtual device.
Metrics and problem detection
A number of metrics and alerts based on them have been prepared for the services used by the application. A dashboard was created in Amazon Cloudwatch service, which allowed the observation of all major metrics in one place. Below is an example of metrics used for the Connect service thanks to which it is possible to observe the number of current connections and to prepare an alarm in case of exceeding the limit set for AWS account.
Immediate response to problems with the operation of applications and the power of used services was possible thanks to the configuration of alarms in Amazon Cloudwatch service and sending notifications about their occurrence via Amazon SNS service. An example of such an alert would be a case where AWS Lambda function was not executing properly and its subsequent calls ended in an error.
Infrastructure diagram
Results
Thanks to the competence of Hostersi team and the use of Amazon Web Services environment, the first working version of the system was launched within several hours from the start of the project. The launched infrastructure allows to serve 50 000 people per hour. The entire solution is highly secure, because it is completely embedded in AWS cloud. The infrastructure is based on the so-called "serverless", which means that the entire effort of managing the physical infrastructure, virtualization layer, operating system updates lies with the cloud provider. For our customer, this means no additional activities and commitment of resources for the proper operation of the infrastructure, because all actions happen completely automatically.
Read also:
44-200 Rybnik
Poland
REGON: 240692928
National Court Register (KRS): 275333
