How to ingest RSS and physical device data into ElasticSearch?

Elastic Stack Elasticsearch
1

In v1.0 of a platform, we had a Windows service read from SQL Server settings to determine which RSS feeds request data from with filters and on a schedule. The feed locations and filters were all based on specific account-related settings.

In working on the planning stages of a v2.0, I am going to be using the ELK stack as the foundation. As such, I am looking for suggestions on how to accomplish the same types of activities. For example, I have been reading about Beats and how it may be better than using LogStash for this.

We will also be consuming data from physical devices as well, where they may report to the server, or the server may need to make a request to the devices.

Can someone please lead me in the right direction for these requests?

2

This sounds like something you would need to build. Logstash can make HTTP calls, but it cannot handle all the rest.

What data?

3

In saying LogStash can handle HTTP calls - can these be based on database parameters that are sent to it or is Beats a better candidate?

Data from physical sources may include things like a geo-location, some other identifiable fields and measurements. I suppose this could be shipped as a JSON object. Some devices can ship data, others we would need to request the data.

I have been thinking of re-writing the Windows service using Task/Async architecture and using NEST to link up to ES, however, I would like to know some other ES-based options.

4

It might be easier to keep this in your other thread - Can someone please explain the Beat architecture - Can a Beat be used to pull data from a list of sources?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.