How to integrate InfluxDB with AWS

Now that InfluxDB Cloud is on AWS US East (us-east-1), based in northern Virginia, InfluxDB Cloud runs on three AWS regions, after initially launching in AWS Oregon and later AWS Frankfurt.

This brings InfluxDB Cloud’s effortless scaling, flexible usage-based pricing, AWS marketplace integration, and a broad range of AWS connectivity points to customers that want to manage their time series data in the eastern United States.

Where InfluxDB Cloud runs

Where does InfluxDB run now? Here’s a list of all the locations:

As you can see, InfluxDB Cloud is broadly distributed across North America and Europe:

Integrate InfluxDB Cloud with AWS

InfluxDB Cloud has a broad range of integration points with AWS. Before we dive in, it’s helpful to think of our time series data platform as doing three main things:

1. Accumulating time series data by ingesting it from numerous locations
2. Analyzing that data, for visualization, anomaly detection and alerting
3. Acting on that data, by sending to alerting systems, custom applications or third-party tools

With this framework, there are a number of ways in which InfluxDB integrates with AWS; AWS technologies are marked with an AWS logo, InfluxDB technologies are in purple, and other technologies are in gray.

To accumulate time series data, InfluxDB provides a number of options for data ingest, but I’d like to focus on two: Telegraf and Flux. Telegraf, our open source server agent for collecting metrics, can pull data from the following AWS services:

• AWS CloudWatch: Import CloudWatch metrics with the CloudWatch Telegraf plugin from 97 different AWS services, including S3, VPC and DynamoDB. InfluxDB provides more powerful and flexible dashboarding than what’s offered natively than CloudWatch. The easiest way to set this is up is with our AWS CloudWatch Monitoring Template; here’s a tutorial on how to quickly import templates into your InfluxDB Cloud instance.
• AWS EC2, AWS Elastic Container Service (ECS) and AWS Fargate: Monitor performance metrics inside your EC2 virtual machines, ECS containers, and Fargate containers, using the Telegraf AWS ECS plugin or Telegraf’s generalized plugins for system monitoring and Docker monitoring. The Telegraf ECS plugin is Fargate compatible, and uses the Amazon ECS v2 metadata and stats API endpoints to gather stats on running containers in a task.
• AWS Elastic Kubernetes Services (EKS): Inventory the components of your Kubernetes clusters running on AWS, and track the performance of each pod and container, using the Telegraf Kubernetes and Kubernetes Inventory plugins.
• AWS Kinesis: Ingest data from AWS Kinesis for analysis, visualization, and alerting, using the Telegraf Kinesis consumer plugin. You can also use Telegraf to publish data to Kinesis as well.
• AWS synthetic monitoring: Track the response times of all your AWS API HTTP endpoints.
• AWS Programmatic Billing: Import your Programmatic Billing data into InfluxDB to better understand AWS spend.
• IoT data collection & analysis: If you’re pulling data from IoT devices and sensors into AWS, use the Telegraf OPC-UA plugin, Modbus plugin, MQTT plugin, AMQP plugin, JSON plugin, HTTP listener plugin, Kinesis consumer, DDS plugin, and other IoT plugins.

Using Flux, our data scripting language purpose-built for time series data, you can pull data directly from SQL data stores, such as AWS RDS Postgres, RDS MySQL, RDS MariaDB, RDS SQL Server, and AWS Athena. The specific Flux function to do this is called sql.from(), and is part of the Flux SQL package. This function lets you enrich time series data to provide additional context. Here are a few examples:

• For IoT users, Flux lets you combine time series data from sensors with information from relational tables like manufacturer, model number, device age or mileage.
• For real user monitoring (RUM), you can use Flux to look up customer IDs and connect customer name, pricing plan, and firmographics like company name and location.
• In IT infrastructure monitoring, Flux can resolve server IDs to software version and configuration parameters.

In fact, Flux lets you enrich from much more than just RDS — from just about any relational database:

In addition to Telegraf and Flux, InfluxDB Cloud can also ingest data from Vector, Apache NiFi, OpenHAB, JMeter, Node-RED, and Jaeger. All in all, a broad range of options for data ingest, for making observability reality.

The analyze piece of the equation is handled by InfluxDB itself. Our main vehicle for this is Flux. Beyond data enrichment, Flux provides a broad range of functions for statistical analysis and forecasting, such as:

• Calculating percentiles to detect SLA compliance failures
• Windowing and aggregating data to pick out insights from noisy data sets
• Forecasting with Holt-Winters to predict outages and capacity issues
• Geographically tracking monitoring metrics to better determine which regions are experiencing problems
• Sophisticated anomaly detection, using techniques like Median Absolute Deviation (MAD) and Balanced Iterative Reducing & Clustering (BIRCH), to help you find and fix problems faster and reduce your mean time to resolution (MTTR)
• Machine learning using Naive Bayes Classification

To act on time series data, you can use Flux running in InfluxDB Tasks to send alerts to a broad range of endpoints: Amazon Simple Email Service (SES) and other email services such as Twilio SendGrid, Mailjet, and Mailgun; IT alerting systems such as PagerDuty, Atlassian Opsgenie, Sensu, and soon ServiceNow and BigPanda; messaging services such as Microsoft Teams, Slack, Discord, Telegram; and general-purpose solutions such as Pushbullet, MQTT, and webhooks.

You can also use Flux to pull data into your code running on AWS Lambda, EC2, or other services, using our client libraries for JavaScript/Node.js, Python, Go, Arduino, Java, Scala, C#, Ruby, PHP, and Kotlin. These client libraries were previously provided by our open source community; they are now fully maintained and supported by InfluxData.

How to access InfluxDB Cloud on AWS Virginia

Here’s a step-by-step guide on how to get InfluxDB Cloud on AWS us-east-1:

First, visit cloud2.influxdata.com and sign up for a free InfluxDB Cloud account.

Once you hit the Create Account button, check your email for a message that looks like this:

Click Verify Your Email, and you’ll see the following page asking you where to store your data.

Once you select AWS, choose US East (N. Virginia), and you’ll use that instance of InfluxDB Cloud.

There are a few more steps that are self-explanatory, but one thing I want to point out is the InfluxDB Cloud URL for AWS East US:

https://us-east-1-1.aws.cloud2.influxdata.com

This is the endpoint of the InfluxDB Cloud API that you’ll be using when you interact with the InfluxDB API, InfluxDB client libraries, Influx CLI, and Telegraf. Here is our documentation page with a complete list of InfluxDB Cloud URLs if you are running on a different region.

Customized code snippets for AWS US East

Speaking of our documentation, here’s a handy trick to customize all our code samples to use the AWS US East URL. In the InfluxDB Cloud documentation, click on the gear icon in the upper-right corner:

Then specify that you’re running in AWS US East (Virginia):

And now code samples in our documentation, such as on this page, will have the correct InfluxDB Cloud URL already inserted so you have one less step to worry about. See the first line of the curl command below:

Expanding our reach

Even though InfluxDB Cloud is available on six regions, we’re still looking to expand our footprint to best serve you. However, we can’t simply deploy across the combined ~75 regions offered by Amazon Web Services, Microsoft Azure, and Google Cloud.

The reason: InfluxDB Cloud is available as an instant-on, always available cloud service; you don’t spin up and maintain a single-tenant virtual machine. This means that for each region, we’re always running a fleet of InfluxDB instances, ready to accept your incoming data, run your queries and tasks, display your dashboards, and fire off your alerts. And THAT means each region incurs a significant monthly cloud bill that needs to be offset by revenue — just like any business.

So, we need YOUR help in telling us where to expand next by filling in this form. It’s literally five fields, anonymous, and will take you all of 30 seconds to complete:

AWS US East is just part of the story

Expanding into AWS US East is just one of the many recent improvements to InfluxDB Cloud. In this video from the InfluxDays North America 2020 Virtual Experience, our VP of Engineering, Ryan Betts, goes into more detail about all we’ve done recently and some of our roadmap:

https://www.youtube.com/watch?v=OOwlGfJrCic&feature=emb_logo

Conclusion

Here’s more about our partnership with AWS. If you’d like to check out InfluxDB Cloud for yourself, sign up for a free InfluxDB Cloud account. Once you do, feel free to ask any questions in our ever-helpful InfluxDB community and community Slack channel. Enjoy!