Stream IoT Cellular Data Directly into AWS Kinesis Firehose with Soracom Funnel
Written by: Mark Reichenbach
Published: February 9, 2023
In this blog, I’m going to show you how to easily stream real-time data from connected cellular IoT devices into the AWS cloud service Kinesis using our cloud resource adapter service, Soracom Funnel.
Many people have asked me about streaming sensor data directly to the cloud without first going through AWS IoT Core, thinking the process would be difficult or complex. Fortunately, it is relatively simple to connect streaming devices to the AWS Kinesis firehose directly using our connector resource, Soracom Funnel. Funnel is a simple and cost-effective way to eliminate the need for certificate management on your devices (which is often a huge headache) and offload it to the Soracom service.
One of the main benefits of Funnel is that it performs encryption in the cloud, sending your data directly to a cloud service securely. It provides a greatly simplified configuration for applications that leverage the cloud and covers most of the forwarding protocol configuration required for certain cloud services, allowing you to specify the forwarding destination and credentials for your data.
The following diagram shows how Funnel can feed into multiple cloud services. This blog will only be covering one method for streaming into the AWS Kinesis Firehose. You can adapt this process to connect to other services as well.
Sounds simple so far, right? Let’s begin to set this unique connector up.
Before We Begin
For the purposes of this blog, we will assume the following:
- The user has a Soracom account (if not, learn how to open a Soracom account)
- The user has a Soracom Air SIM and a device that can be used for testing (i.e., smartphone/tablet, mobile router, cellular IoT device, etc.)
- The user has an AWS account.
If you do not have a Soracom account, please follow the Quick Start Guide to create an account and set up your payment information.
If you already have an account or have completed the setup laid out above, please complete the SIM registration.
Connect Soracom Funnel to AWS Kinesis Firehose
First, we must create a Kinesis Firehose in AWS.
- Log into your AWS account, search for Kinesis and create a new Kinesis Firehose from the tool. The step-by-step instructions are here.
- Then create and name an S3 Bucket where you will store the streaming data.
Before you connect the Kinesis Firehose to the S3 Bucket, you must create an IAM role in AWS that allows you to save the data to the S3 bucket.
- Create the IAM users that will access Kinesis Firehose. On the AWS IAM Management Console, select an existing IAM role or create a new IAM user and attach the policy. We will attach the role and allow Full Access to the services.
Your Amazon Kinesis Firehose cloud service should now be set up.
Before you go any further, you will want to test with sample data before connecting Funnel to Kinesis.
- There is a test with pre-populated sample data from Kinesis Firehose to S3 Bucket that you can use to verify that you have access to the bucket.
- This shows it’s working on the test data.
Setting up Funnel in the Soracom Console
The next step I will cover is the creation of a new SIM group in the Soracom console to configure it to send all of the user data into the Soracom Funnel service. But first, you will need to register your Cloud Service credentials in Soracom access manager, this is where you register your AWS credentials, to allow Soracom Funnel to send the data to AWS Kinesis Firehose.
- Create a new group to send to Funnel. I called mine (marks_kenisis_firehose)
(note, I used the Soracom ARC service, which allows me to create a virtual SIM that allows me to connect to the Soracom cellular network with my Mac using Wireguard. This way, I can just open a terminal window and enter the data to send through the network, emulating a cellular device connection.)
- Create a Soracom Funnel
- Add AWS destination endpoint https://firehose.us-west-2.amazonaws.com/(your destination endpoint here) to Funnel
- Remember, in order to send your IoT data into AWS Kinesis Firehose, you have to create a new role in Soracom Access Manager with credentials taken from AWS IAM. This will allow you to connect to AWS Kinesis Firehose, with the credentials for this role.
- Send test commands to Funnel using my mac (I am using Soracom Arc from my Mac)
- Verify if Funnel responds correctly. You should see the following:
- Above shows that Funnel has responded correctly with a successful upload via HTTP
- Now it’s time to check in the S3 Bucket to see if you received the message Hello Soracom Funnel in the S3 bucket.
It’s Working! The data is in the AWS S3 bucket and includes the simID, identifying the individual IoT device.
Conclusion
The benefit of the Sorcom Funnel service is to provide cloud developers with a simple path for their data into AWS Kinesis Firehose. This means the network can bypass AWS IoT, which adds additional costs, if those developers only need to stream data into an S3 bucket for storage and additional analysis. If users don’t require any of the AWS IoT core services, it is a simpler and less costly path.
………………
Got a question for Soracom? Whether you’re an existing customer, interested in learning more about our product and services, or want to learn about our Partner program – we’d love to hear from you!
