On Telegraf. Q&A with Samantha Wang

Telegraf has great community support, and developers contribute plugins to the project all the time. In fact, we recently recognized our 1,000th Telegraf contributor!

Q1.  What is the open source project Telegraf?

Telegraf is InfluxData’s data collection agent. It’s plugin-based and has more than 300 plugins, many of which came from subject matter experts in our community. These plugins allow developers to collect and transform data from hundreds of data sources. If there’s a plugin that a developer needs, but doesn’t exist, Telegraf is open source, so you can write your own. Of course, if you want to contribute that back to the community we’re happy to have it. Telegraf also does more than simply manage data ingest. It has plugins for processing and aggregating data, so developers can clean and transform data before it ever hits the datastore. It also has output plugins so developers can send data to one or more sources once they’re done transforming it. It’s an incredibly flexible tool.

Q2. What are the main benefits of using Telegraf?

Where do I even start? First of all it’s very lightweight. It requires a minimal memory footprint, so developers can run it almost anywhere. It’s also compatible with pretty much any architecture, too, so it’s really flexible in that sense as well. Plus, being open source, if you have some sort of bespoke architecture, you can always compile your own version of Telegraf if you need to do so. 

I think one of the biggest benefits is the plugin library though. Telegraf has great community support, and developers contribute plugins to the project all the time. In fact, we recently recognized our 1,000th Telegraf contributor! The 300+ plugins really give Telegraf a lot of its power because they allow the agent to handle virtually any type of data you can throw at it. For developers, this means Telegraf has the flexibility to meet the needs of developers no matter what they’re working on, from large enterprise solutions to at-home side projects. That makes it incredibly nimble and also incredibly powerful.

Q3. What kind metrics and events from databases, systems, and IoT sensors is Telegraf collecting?

There’s not a finite group of metric or event types that Telegraf can handle. One of the main goals is to make Telegraf capable of handling any type of time series data. Each plugin collects different types of measurements. But to answer the question, when it comes to database information some commonly collected metrics include reads and writes, system performance, and dropped metrics. When it comes to systems, that can include things such as CPU or memory usage, or IoT sensors that track things like humidity, temperature, power, voltage, and more.

Q4. Telegraf is a server-based agent. What does it mean in practice?

It means that Telegraf gives developers a lot of flexibility for how and where they run it. Simply put, Telegraf is an application that runs on a server. That can be in a physical system or a virtual machine. We refer to it as a data collection agent because its job is to read data from the clients and sources producing data. 

Q5. Let’s talk about coverage: How does it work to collect metrics from endpoints?

Telegraf reads directly from endpoints. The specific way it does this depends on the plugin, the data source, and the plugin configuration. Typically, users can designate a port or URL for an endpoint that they want to collect data from in the plugin configuration. 

Q6. Now let’s look at flexibility: How does it work to handle different cases, e.g. whether you need metrics on the edge, or in a centralized manner?

We designed Telegraf to use it in a wide variety of use cases, whether that’s tinkering around at home with a Raspberry Pi or integrating it into a major enterprise environment. Telegraf is lightweight and provides robust scheduling, so it’s ideal for collecting metrics on the edge. Telegraf can also collect data on edge devices and send that data to a messaging queue or directly to InfluxDB. 

Telegraf functions just as well in large-scale environments. It can handle collecting thousands of metrics per second from thousands of devices. When working at an enterprise scale, we recommend spreading out the data collection load across multiple Telegraf instances. Large enterprise environments often have 10,000-15,000 Telegraf agents collecting metrics and sending data to InfluxDB. 

Q7. If we consider integration, how does Telegraf’s plugin system allow new inputs and outputs to be added?

We designed Telegraf to be a plugin-based agent, so it’s easy to add new plugins to the binary. Every new minor release includes new features, and oftentimes that includes new plugins, too. Sometimes these plugins originate from requests from within the Telegraf open source community that users request in our GitHub repository. And sometimes community members contribute the plugins they build back to the open source project. New plugins are like additional building blocks for Telegraf; they aren’t necessarily required, but they’re another tool that allows Telegraf to easily integrate new data sources with InfluxDB. 

Developers can also run external plugins with Telegraf using Execd. These plugins run an external program as a long-running daemon. They can use this functionality to write an external program in any language and then use Execd to write the metrics in or out with Telegraf. 

Q8. How can Telegraf transform data? 

Telegraf’s main job is as a collection agent. But if developers can clean data as it comes in, then they have less processing to do in their datastore. So the sooner users can clean data, the easier and more efficient that data is to work with. Telegraf has four plugin categories, two of which are processors and aggregators. These plugins give developers ways to clean and transform their data before it hits their datastore. Processors let developers add, remove, or alter tags for incoming metrics, for example. Aggregators apply some sort of mathematical operation to incoming metrics, like mean, minimum, maximum, and standard deviation. Users can always check out our docs for more information on the full capabilities of these plugins.  

Q9. Is it possible to download and use Telegraf as standalone, or do you need to have InfluxDB already installed?

Telegraf needs to connect to a datastore, so it can’t completely stand alone. However, that datastore doesn’t have to be InfluxDB. Obviously, we recommend using Telegraf with InfluxDB, and the fact that we have a single API that works seamlessly across the entire InfluxDB platform really helps to streamline using them together, but developers can use Telegraf without InfluxDB. Users need an input data source to connect to and a source where they write the output, but the input and output sources can be whatever they want or need them to be, such as Kafka, Prometheus, or an HTTP endpoint.

Q10: How can people learn more about configuring and using Telegraf?

We recently released Data Collection with Telegraf, a free self-paced course through InfluxDB University. Developers can learn the basics of Telegraf, how to configure it to transform and process data, and we cover troubleshooting tips for avoiding common mistakes like a pro. Learners who pass the course will earn a digital badge they can add to their LinkedIn to validate their skills. 

……………………………………………..

Samantha Wang 

Product Manager

Samantha Wang is currently a Product Manager at InfluxData, creator of InfluxDB, the leading time series platform. Samantha is currently responsible for Telegraf, InfluxData’s open source data collection agent. Previous to InfluxData, she worked in product and engineering roles at Gracenote (a Nielsen Company) and Verizon. Samantha holds a degree in Industrial & Systems Engineering from the University of Southern California.

Sponsored by InfluxData.