Logstash doesn't load data until interrupted

Elastic Stack Logstash
1

My cURL command in the input below invokes an https endpoint that will stream data until interrupted. My config below works but no data is loaded to Elasticsearch until I interrupt Logstash. Logstash keeps the data in memory until I do Ctrl^C to kill the pipeline.

input {
   exec { 
      command => 'curl -X GET "https://feed.test.com/1.0/json/A98F049" -H "api_key:zBAHA"'
      interval => 600000000 
      condec => "json_lines"
   }
filter {
   json { source => message}
}
}
output {
  stdout {
    codec => rubydebug
  }
elasticsearch {
   hosts => "localhost:9200"
   index => "abc"
  }
}

My screen shows some stats like this which update in real time showing time since start and bytes received; bytes xfered stays zero:

% Total % Received % Xferd Average Speed Time Time Time Current
0 0 0 0 0 0 0 0--:--:-- 0:00:28 --:--:-- 887

but only after killing the pipeline, the data shows up on the screen and in Elasticsearch. How can I index the data as it arrives continuously? Changing the interval value didn't make a real change.

2

The Exec Input Plugin doesn't support streaming. It waits for the process to complete and then passes the output through to the codec to generate Events.

3

Thank you for clearing the dust. It has been a long road to Exec Input. I had started with http and http_poller as shown here: How to set up input for https curl and header key? without success. Which plugin is best-practice for streaming that supports continuous indexing?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.