Logstash HTTP polling performance

Elastic Stack Logstash
1

Hello,

In my use case I need to periodically query huge population of devices(~1M) via HTTP REST API and store status of each device in elasticsearch.

I consider to use http_poller but I worry about perfomance and scalability of this solution. Does anyone have more experience with this input type ?

2

I suspect Logstash uses one thread per input plugin, so using one http_poller per device clearly won't scale. I don't think a pure Logstash solution is appropriate here. You need something else to do the HTTP polling, but Logstash could still read the results of that polling and process it further.