Custom Beats vs. Scripts

Elastic Stack Beats
1

Working on a project that will be querying a number of data sources and inserting the resulting data into ES. The data sources include: Bing API, Twitter, Instagram, Facebook, and more social media platforms.

Thoughts on the following options, pros/cons/etc:

Option 1: Develop custom beats for each of these data sources and output directly to ES via https

Option 2: Write a number of scripts in $language to query these data sources and output directly to ES via https

2

apibeat sounds like a good idea, with modules for each site maybe?

Otherwise Logstash has a http_poller input.

3

I'd consider option 3: Write a number of scripts in $language to query these data sources and dump to files that are consumed by Filebeat. It's simple, language-agnostic, and gives you a natural buffer that helps with outages on the receiving end.

4

@DefensiveDepth I would also check what kind of plugins are already available for Logstash.

5

This topic was automatically closed after 21 days. New replies are no longer allowed.