How to create a python-based webhook listener on AWS

Webhooks are a powerful mechanism for real-time data exchange between different applications or services. They allow one system to send automated, instant notifications to another when specific events occur, eliminating the need for constant polling.

We've just launched Simple Python Webhook Cloud Component on Github. Take a look at the full components source code on GitHub. In this article we explain the concept of a webhook and then we explain how to deploy the MakeOps cloud component into your AWS environment.

Webhooks are Everywhere

Webhooks are frequently used to glue different services or systems together. There has been a rise in companies that want to build their own customizations into existing tools to improve their company productivity and efficiency.

Webhooks are versatile tools that can streamline operations across various industries and use cases. In e-commerce, a webhook could trigger when a customer places an order, instantly notifying the warehouse management system to begin the fulfillment process. For SaaS platforms, webhooks can alert account managers when a client's usage approaches their plan limits, enabling proactive outreach. In IoT scenarios, a smart home system might use webhooks to notify a security service when a sensor detects unexpected movement. Development teams often implement webhooks to automate their CI/CD pipelines, triggering build and deployment processes when code is pushed to a repository. Financial applications can leverage webhooks to send real-time alerts for unusual account activity or to update account balances across integrated systems. These examples illustrate how webhooks serve as the digital nervous system for modern, interconnected applications, enabling real-time responsiveness and automation across diverse business processes.

Implementing webhooks involves setting up an endpoint in the receiving application to accept incoming HTTP POST requests from the sending application. This simple yet powerful approach has become a cornerstone of modern API integrations and microservices architectures.

From today, we've released a MakeOps Cloud Component to make it easier to deploy a simple python-based webhook handler into your existing or new AWS environment.

What are Cloud Components?

MakeOps Cloud Components are pre-built stacks that perform common tasks on AWS. Things that we've seen customers try to build themselves. We use AWS CDK (written in Typescript) to create reusable infrastructure as code samples that can be integrated into your product or service.

Architecture for a Python Webhook Listener on AWS

The output of deploying the cloud component is an architecture as seen below:

The architecture flow is as follows:

• User or External Application accesses the webhook via API Gateway
• The API gateway has an integration to AWS Lambda, when a webhook is received by API gateway, the python lambda function code is triggered.
• The python webhook trigger function runs and returns a value to API Gateway
• The user receives the response from the webhook
• During the function trigger, any logs are sent to Amazon CloudWatch (Note the logs are set to only be retained for 2 weeks to save costs)

Deployment - Pre-requisites

You'll need to have the following setup before you can deploy this component:

• AWS CLI Installed & Configured
• AWS CDK Installed & Configured
• PNPM or NPM installed with v20.14 or greater
• Docker installed and running
• Git Installed
• Optional HTTPie installed and configured - to test the webhook
• Optional AWS SAM installed and configured - to view logs

Deployment

• Clone the repo and change to the component directory.

$ git clone git@github.com:MakeOps/components.git
$ cd components/simple-python-webhook

• Update the code you want to run when a webhook is received in code/handler.py

def webhook_handler(event, _context):
    '''Webhook Event Handler Function'''

    # For an example event see example.json
    print(json.dumps(event))

    ## ======= YOUR CODE ========

    response_payload = {'version': '0.1.0'}

    ## ==== END OF YOUR CODE ====

    return {
        'statusCode': 200,
        'body': json.dumps(response_payload),
        'headers': {
            'Content-Type': 'application/json'
        }
    }

Save the file.

• Install the dependencies using npm or pnpm

$ pnpm install
$ # OR
$ npm install

• Deploy the stack to your AWS environment

$ cdk deploy --all

Accept any prompts about the IAM changes

AWS Python Lambda Webhook Usage

When the stack finishes deploying you should see that there are two variables that are output in your command line:

[...]
Outputs:
SimplePythonWebhookStack.APIEndpoint = https://pwsw41taz9.execute-api.eu-west-1.amazonaws.com
SimplePythonWebhookStack.LoggingOutput = /aws/lambda/SimplePythonWebhookStack-WebhookFunction59DCB58D-UelSZjxMj0I0
Stack ARN:
arn:aws:cloudformation:eu-west-1:375479154925:stack/SimplePythonWebhookStack/b626bf50-540f-11ef-aab5-063ed5e6995f
[...]

• APIEndpoint is the endpoint for your webhook, this is what you register with a third-party
• LoggingOutput is where the logs for your webhook python function are sent

Triggering the Python-based Webhook

You may use any tool to do this but we prefer httpie. You may use Postman or Curl.

$ http POST https://pwsw41taz9.execute-api.eu-west-1.amazonaws.com name=makeops
HTTP/1.1 200 OK
Apigw-Requestid: cGMi_haHjoEEPmw=
Connection: keep-alive
Content-Length: 20
Content-Type: text/plain; charset=utf-8
Date: Tue, 06 Aug 2024 17:18:23 GMT
{
    "version": "0.1.0"
}

Viewing the Logs from our Python Webhook

To view the logs you can navigate to Amazon CloudWatch Logs to find the log group mentioned in LoggingOutput above.

Alternatively (and in our opinion better) is to use SAM CLI:

$ sam logs -t --cw-log-group /aws/lambda/SimplePythonWebhookStack-WebhookFunction59DCB58D-UelSZjxMj0I0

This should show you some of the most recent logs from your webhook.

Clean Up

If you're done playing around with this component, delete it using the cdk destroy operation.

$ cdk destroy --all

Next Steps

Once you've got the webhook deployed you can start to update the python code inside the handler.py file.

As a starting point, try connect the webhook endpoint into an application that emits webhooks and the get an event to trigger. You can view the logs to get an idea of the structure of the webhook event, then create code to handle the event in a customized way.