Moving towards event-driven architectures with serverless event aggregators

Event-driven architectures are gaining popularity because they help deliver benefits by decoupling services, increasing scalability, adding flexibility, and increasing agility for the developer organization. Over time, companies must find ways to integrate (merge) multiple processes and applications (usually owned by different teams).

Connected applications work in specific scenarios where synchronous communication is required (e.g., API calls). They can support critical business functions but, over time, can become cumbersome when dependencies between applications extend beyond the original context. For example, with increasing complexity, changes in one application may also require changes in closely related applications. These closely related applications usually start to introduce bottlenecks and reduce flexibility when looking at the overall modernization effort. Simple changes become complex and, in some cases, block upgrade efforts altogether or delay them significantly. In such situations, it is worth looking at the advantages of event-driven architectures.

In event-driven architectures, each application triggers an event every time something changes. The dependent applications are then responsible for listening and deciding what action to take. This approach allows you to separate the sources and targets of events. Reducing dependencies between applications also enables teams to operate more independently and gives them the flexibility required for further upgrade work.

Today, you can use Amazon EventBridge to orchestrate your events. In this choreography, Amazon EventBridge rules match incoming events from event sources. It directs them to targets based on defined trigger patterns. However, if you have complex processes where multiple conditions are analyzed before an event is triggered, you will need additional logic to organize events on a large scale. This is what is referred to as event aggregators.

In this article, the authors will discuss how you can transform your existing environment with event-driven architectures. This includes how to build 'serverless event aggregators' when creating complex triggers.

Scenario discussion

To explain the value of the serverless event aggregator approach, you will use a simplified version of a popular e-commerce scenario. This example scenario concerns an online business where customers use your site to buy your products. In this case, multiple teams (and back-end systems/applications) are looking for different parts of the process. They all have to perform tasks based on the information they receive.

When your customers create an order:

• you check and approve their account for enough credits to cover the cost of the product AND
• you check that the stock for the requested product is sufficient.

And if BOTH processes are successful:

• you send a message to your customer stating "Order sent" or
• if there is a problem (incorrect order), you take further action.

In the past, two basic patterns could be observed for similar scenarios:

• Polling mechanism: this is set up for each successive application to check that the previous processes have been completed continuously.
• Fixed scheduling: a fixed schedule is established for each process, making it easier to follow the sequence.

While these integration approaches have worked well in some use cases, they have inadvertently introduced tight coupling between applications. This led to static architectures that became bottlenecks in modernization efforts. This also directly impacted customers, where complex synchronous architectures (such as API operations) required all steps to be completed before a response could be made available.

Creating Serverless Event Aggregators

You can implement asynchronous controls using an event aggregator to address the issues described in the previous scenario. This decision decouples applications, potentially protecting the client environment while all steps are processed.

This serverless event aggregator solution can be built using DynamoDB, AWS Lambda, and Amazon EventBridge. In this solution, the authors use the functionality of DynamoDB and AWS Lambda streams to aggregate events, as well as event-based routing from Amazon EventBridge. This solution consists of four main steps:

Step 1: Update DynamoDB from event sources

It all starts with the event sources (in this example, the 'checking account' and 'in-store' applications). When the source applications are launched, they update the DynamoDB table via the UpdateItem API.

The screenshot below shows the 'Events' table created in DynamoDB and the updated events from the source applications ('checking account' and 'in-store') with their order ID.

Using DynamoDB as an event store, you can add more event sources or update them as your business needs change. This will provide greater flexibility and scalability, allowing you to include more teams and processes in your event-driven architecture.

Step 2: use a DynamoDB stream to trigger an event

When your applications (event sources) update a DynamoDB table, your DynamoDB table will trigger a Lambda function using an integrated Lambda and DynamoDB trigger mechanism.

You can implement this integration using DynamoDB streams. When you enable a DynamoDB stream in your DynamoDB table, it will be seen as a 'trigger' in the AWS Lambda function. This tutorial can create a DynamoDB stream and AWS Lambda trigger. The image below shows the Lambda console after installation.

Step 3: Aggregate your events

You will now build your event aggregator using the Lambda function to aggregate all your events. This will include all the aggregation business logic. In our scenario, this function receives the order ID you created as input. It then scans the DynamoDB table for all events related to the same order. It then generates a custom EventBridge event with a list of all events for the order using the PutEvents API.

Below is an example of the event aggregator of this Lambda function written in Python:

import json

import boto3

from boto3.dynamodb.conditions import Attr

dynamodb = boto3.resource('dynamodb')

client = boto3.client('events')

def lambda_handler(event, context):

print(json.dumps(event))

order = int(event['Records'][0]['dynamodb']['NewImage']['OrderID']['N']) print(order)

table = dynamodb.Table('Events')

table_scan = table.scan(FilterExpression=Attr('OrderID').eq(order))

event_status = {} for item in table_scan['Items']:

event_status[item['EventName']] = order print (json.dumps(event_status))

response = client.put_events(

Entries=[

{

'Source': 'my-event',

'DetailType': 'events list from dynamodb',

'Detail': 'json.dumps(event_status)', '

EventBusName': 'default',

},

])

Step 4: Filter and target your events using Amazon EventBridge

When the aggregator function is run, it will create an event that contains an aggregated view of the state of multiple events. For example, in the generated event in the table below, you can see that both the 'in stock' and 'checking account' events have been updated with the value OrderID: 7. This means that both checks for Order 7 have been completed and you can now proceed with further events.

{

"version": "0",

"id":"711ef30b-582a-cda5-c132-bceb46bc5c6a",

"detail-type": "events list from dynamodb",

"source": "my-event",

"account":"123456789098",

"time":"2022-10-13T13:03:02Z",

"region":"eu-west-1",

"resources":[

],

"detail":{

"In Stock":7,

"Account Check":7

}

}

When EventBridge receives this event, it will be processed in EventBridge rules to capture the specific state of both events ("account check" and "in stock"). If both pass the tests, the Lambda function 'Invoice processing' will be triggered.

Below is the configuration of the EventBridge rule defined for the scenario in question.

{

"source": ["my-event" ],

"detail-type": ["events list from dynamodb" ],

"detail": {

"Account Check": [{

"exists": true

}],

"In Stock": [{

"exists": true

}]

}

}

EventBridge rules use event patterns to select and match incoming events and trigger specific targets for processing.

Amazon EventBridge supports several comparison operators. The Amazon EventBridge event patterns provide a full list of supported operators.

In the example pattern, the authors used the 'And' operator to select the events 'checking account' and 'in stock' "exists: true." When the pattern conditions are met, EventBridge will trigger the 'Invoice processing' target.

The screenshot below shows how the Lambda function "InvoiceProcessing" is configured to trigger with the matched event.

With this step, the authors have completed the scenario in question. They collected events from multiple event sources and aggregated them as desired using the "Event Aggregator." Once all the defined conditions were met, they organized them via Amazon EventBridge to run the defined target application, "Invoice processing." Due to the flexible nature of this solution, components can be added/removed and changed without changing the architecture.

Applications

With serverless event aggregators, scalability, flexibility, and control can be achieved by decoupling applications that require complex triggering mechanisms. As with any modernization project, your journey starts with understanding your business needs and current capabilities. Bringing your teams together and establishing how your applications interact will be a valuable step towards moving to an event-driven architecture.

In this article, the authors explored how to approach application transformation with event-driven architectures using serverless event aggregators. They applied a simplified scenario to an e-commerce application historically tightly coupled with back-end processing systems. The transformation broke up the tight coupling of the dependent processes and enabled independent and asynchronous event processing to be completed. This allows you to remove bottlenecks and scale your modernization efforts more effectively.

With DynamoDB integration, you can have as many event sources as you need, a growing solution, and complete control over the Lambda Aggregator function. These two integrations will provide the flexibility to respond to changing business requirements. Once events are aggregated, AWS EventBridge can help integrate the solution with more than 35 targets. These targets include many SaaS applications such as Datadog, Onelogin, and PagerDuty. With pre-built integrations, you don't have to manage any integration configuration. This will increase developer flexibility, reducing the need for coordination between different teams and SaaS applications.

To learn more about event-driven architectures, try the 'Building event-driven architectures on AWS workshop', check out Serverless Land for the latest blogs, videos ,and training resources, and explore the AWS Serverless Application repository to discover ready-made applications for your upgrade needs.