Custom Beats vs. Scripts


(Josh Brower) #1

Working on a project that will be querying a number of data sources and inserting the resulting data into ES. The data sources include: Bing API, Twitter, Instagram, Facebook, and more social media platforms.

Thoughts on the following options, pros/cons/etc:

Option 1: Develop custom beats for each of these data sources and output directly to ES via https

Option 2: Write a number of scripts in $language to query these data sources and output directly to ES via https


(Mark Walkom) #2

apibeat sounds like a good idea, with modules for each site maybe?

Otherwise Logstash has a http_poller input.


(Magnus B├Ąck) #3

I'd consider option 3: Write a number of scripts in $language to query these data sources and dump to files that are consumed by Filebeat. It's simple, language-agnostic, and gives you a natural buffer that helps with outages on the receiving end.


(ruflin) #4

@DefensiveDepth I would also check what kind of plugins are already available for Logstash.


(system) #5

This topic was automatically closed after 21 days. New replies are no longer allowed.