Related Articles

Mastering IoT and API Integration with AWS: The Wildcard Advantage

In the dynamic world of IoT, agility and security are paramount. Building on my previous articles about Smart IoT Solutions, AWS Serverless APIs, and High Availability, I engineered a solution that is robust, scalable, and secure. In this post, I walk you through how wildcards in MQTT topics simplify device management while ensuring a seamless data flow—from AWS IoT Core, through Lambda and DynamoDB, to a public API endpoint.

The Wildcard Advantage in AWS IoT

Using a wildcard in a topic like bestsense/poc/*/lab allows you to group devices effortlessly. The asterisk substitutes for any single-level value, enabling dynamic device identification and data aggregation without hardcoding individual IDs. This approach simplifies management, enhances scalability, and reduces configuration overhead.

End-to-End Architecture with AWS Services

I designed this solution using several key AWS services:

Critical CLI Steps You Shouldn't Miss

1. Creating the IoT Policy

The IoT policy ensures devices can connect, publish, subscribe, and receive on the necessary topics. For example:

bash
cat << 'EOF' > iot-policy.json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": ["iot:Connect", "iot:Publish", "iot:Subscribe", "iot:Receive"], "Resource": [ "arn:aws:iot:ap-southeast-2:XXXXXXXXXXXX:client/*", "arn:aws:iot:ap-southeast-2:XXXXXXXXXXXX:topic/bestsense/poc/*/lab" ] }] } EOF aws iot create-policy \ --policy-name "poc_policy" \ --policy-document file://iot-policy.json

2. Setting Up the IoT Rule

This rule listens to topics like bestsense/poc/+/lab and triggers the Lambda function:

bash
aws iot create-topic-rule \ --rule-name "poc_rule" \ --topic-rule-payload '{ "sql": "SELECT * FROM '\''bestsense/poc/+/lab'\''", "description": "Process sensor data", "ruleDisabled": false, "awsIotSqlVersion": "2016-03-23", "actions": [{ "lambda": { "functionArn": "arn:aws:lambda:ap-southeast-2:XXXXXXXXXXXX:function:Air-net-PoC" } }] }'

3. Creating the Lambda Function for IoT Data Processing

Before creating the DynamoDB table, set up a Lambda function that processes IoT data and writes it to the table:

bash
cat << 'EOF' > Air-net-PoC.py import json import boto3 from datetime import datetime dynamodb = boto3.resource('dynamodb') table = dynamodb.Table('poc_data') def lambda_handler(event, context): try: timestamp = str(datetime.utcnow().isoformat()) response = table.put_item( Item={ 'id': timestamp, 'raw_data': event } ) return {'statusCode': 200, 'body': json.dumps('Data stored successfully')} except Exception as e: return {'statusCode': 500, 'body': json.dumps(str(e))} EOF zip Air-net-PoC.zip Air-net-PoC.py aws lambda create-function \ --function-name "Air-net-PoC" \ --runtime python3.9 \ --handler Air-net-PoC.lambda_handler \ --role arn:aws:iam::XXXXXXXXXXXX:role/poc_lambda_role \ --zip-file fileb://Air-net-PoC.zip

4. Creating the DynamoDB Table

A simple, on-demand DynamoDB table to store the processed data:

bash
aws dynamodb create-table \ --table-name poc_data \ --attribute-definitions AttributeName=id,AttributeType=S \ --key-schema AttributeName=id,KeyType=HASH \ --billing-mode PAY_PER_REQUEST

5. Creating the Lambda Function for API Access

Finally, create a Lambda function to expose the data via API Gateway. Sensitive details have been masked:

bash
cat << 'EOF' > mobile--poc.py import json import boto3 from decimal import Decimal dynamodb = boto3.resource('dynamodb') table = dynamodb.Table('poc_data') def lambda_handler(event, context): try: response = table.scan(Limit=10) return { 'statusCode': 200, 'headers': { 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json' }, 'body': json.dumps(response['Items'], default=str) } except Exception as e: return { 'statusCode': 500, 'body': json.dumps(str(e)) } EOF zip mobile--poc.zip mobile--poc.py aws lambda create-function \ --function-name "mobile--poc" \ --runtime python3.9 \ --handler mobile--poc.lambda_handler \ --role arn:aws:iam::XXXXXXXXXXXX:role/poc_lambda_role \ --zip-file fileb://mobile--poc.zip

Ready to implement this in your lab environment, download the resources below.

🚀 Ready to Build This?

Get all the resources you need to build this in your AWS lab!

Download from My Github

Securing Public API Access with Reverse Proxy (Note)

In my simulation, data is generated via a Python script, so I did not implement a reverse proxy. However, in a production environment where real data is exposed publicly, setting up a reverse proxy using Nginx or Apache—with SSL/TLS, a Web Application Firewall (WAF), rate limiting, and IP whitelisting—is highly recommended to safeguard your API.

PM1

Very fine particles

Loading...

PM2.5

Fine particles, like bacteria and smoke

Loading...

PM4.25

Slightly larger particles

Loading...

PM10

Larger particles, like dust and pollen

Loading...

Bringing It All Together

This solution demonstrates how AWS services can be orchestrated to build an efficient, secure IoT-to-API pipeline. The wildcard mechanism simplifies handling diverse device data, while the comprehensive setup—including IoT policies, rules, Lambda functions, and DynamoDB—ensures your data is processed and stored seamlessly. Although my simulation does not include a reverse proxy, I strongly recommend one for any production environment to secure your public endpoints.

Embrace the power of wildcards. Embrace the future. With this approach, your IoT journey becomes both streamlined and secure.