Elasticsearch input plugin not working

Elastic Stack Logstash
1

Hi all,

I'm trying a simple pipeline where I use elasticsearch to get data and send them again to elastic but updated.

For some reason the pipeline stops...

Here the config I'm using:

input{
  	elasticsearch {
      hosts => "localhost"
      index => "dp_api-2017.04.28"
      docinfo => true
      query => '{"query": {"bool": {"must": [{"match": {"class": "DPAPINTERNAL"}},{"range": {"@timestamp": {"from": "now-10h","to": "now"}}}],"must_not": { "match": {"tags": "enriched"  }}}}}'
      tags => ["to_enrich"]
      scroll => "2m"
      codec => "plain"
    }
}
filter{
  if "to_enrich" in [tags]{
    elasticsearch { #prendo i dpapi logs
      hosts => "localhost"
      index => "*"
      query_template => "/home/vittorio/Documents/offline-pipelines/conf.d-2/elastic-queries/matching-requestaw.json"
      fields => { "request" => "new_key" }
      add_tag => ["enriched", "output_splunk"]
    }
  }
}

output{
	if "enriched" in [tags] {
    elasticsearch {
        hosts => "localhost"
        index => "%{index_name}-%{+YYYY.MM.dd}"
        document_id => "%{[@metadata][_id]}"
        action => "update"
      }
  }
}

There's no error from logstash in debug mode..

2

here is part (debug mode) of the shell result:

[2017-05-02T18:24:57,784][DEBUG][logstash.outputs.elasticsearch] Found existing Elasticsearch template. Skipping template management {:name=>"logstash"}
[2017-05-02T18:24:57,785][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x654e1406 URL://localhost>]}
[2017-05-02T18:24:57,790][INFO ][logstash.filters.elasticsearch] New ElasticSearch filter {:hosts=>["localhost"]}
[2017-05-02T18:24:57,855][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-05-02T18:24:57,866][INFO ][logstash.pipeline        ] Pipeline main started
[2017-05-02T18:24:57,877][DEBUG][logstash.agent           ] Starting puma
[2017-05-02T18:24:57,878][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2017-05-02T18:24:57,879][DEBUG][logstash.api.service     ] [api-service] start
[2017-05-02T18:24:58,106][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2017-05-02T18:24:58,167][DEBUG][logstash.inputs.elasticsearch] closing {:plugin=>"LogStash::Inputs::Elasticsearch"}
[2017-05-02T18:24:58,207][DEBUG][logstash.pipeline        ] Input plugins stopped! Will shutdown filter/output workers.
[2017-05-02T18:24:58,271][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2017-05-02T18:24:58,272][DEBUG][logstash.pipeline        ] Pushing shutdown {:thread=>"#<Thread:0x160a7fa7 run>"}
[2017-05-02T18:24:58,272][DEBUG][logstash.pipeline        ] Pushing shutdown {:thread=>"#<Thread:0x2cbcd052 sleep>"}
[2017-05-02T18:24:58,273][DEBUG][logstash.pipeline        ] Pushing shutdown {:thread=>"#<Thread:0xc0a9d16 sleep>"}
[2017-05-02T18:24:58,273][DEBUG][logstash.pipeline        ] Pushing shutdown {:thread=>"#<Thread:0x635717cb sleep>"}
[2017-05-02T18:24:58,273][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0x160a7fa7>
[2017-05-02T18:24:58,300][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0x2cbcd052>
[2017-05-02T18:24:58,301][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0xc0a9d16>
[2017-05-02T18:24:58,301][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0x635717cb>
[2017-05-02T18:24:58,301][DEBUG][logstash.filters.elasticsearch] closing {:plugin=>"LogStash::Filters::Elasticsearch"}
[2017-05-02T18:24:58,302][DEBUG][logstash.outputs.stdout  ] closing {:plugin=>"LogStash::Outputs::Stdout"}
[2017-05-02T18:24:58,302][DEBUG][logstash.outputs.elasticsearch] closing {:plugin=>"LogStash::Outputs::ElasticSearch"}
[2017-05-02T18:24:58,304][DEBUG][logstash.outputs.elasticsearch] Stopping sniffer
[2017-05-02T18:24:58,304][DEBUG][logstash.outputs.elasticsearch] Stopping resurrectionist
[2017-05-02T18:24:58,625][DEBUG][logstash.outputs.elasticsearch] Waiting for in use manticore connections
[2017-05-02T18:24:58,626][DEBUG][logstash.outputs.elasticsearch] Closing adapter #<LogStash::Outputs::ElasticSearch::HttpClient::ManticoreAdapter:0x3c52d27e>
[2017-05-02T18:24:58,627][DEBUG][logstash.pipeline        ] Pipeline main has been shutdown
[2017-05-02T18:25:00,877][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2017-05-02T18:25:00,877][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2017-05-02T18:25:00,878][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping
[2017-05-02T18:25:00,880][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}
[2017-05-02T18:25:00,881][DEBUG][logstash.pipeline        ] Closing inputs
[2017-05-02T18:25:00,881][DEBUG][logstash.inputs.elasticsearch] stopping {:plugin=>"LogStash::Inputs::Elasticsearch"}
[2017-05-02T18:25:00,881][DEBUG][logstash.pipeline        ] Closed inputs
3

Please somebody take a look at this.

4

is it normal maybe that when it finishes process the data it stops the pipeline?

5

Hello Vittorio,

Are you try to add port number to your elasticsearch in the output plugin?

6

hello @Baco, thanks for your answer!

I'm not sure I've understood your question, have you asked if I tried to put port number in the output or are you asking if I'm trying to add port number?

If you refer to the first question, yes I tried to do "localhost:9200"

7

Yes, my question was if you did "localhost:9200"

Sorry, but I will continue to look the configuration.

8

Is the elasticsearch input even supposed to keep Logstash alive? I suspect it's designed to fire off the query once, send the results down the pipeline, and then shut down Logstash.

9

Hi @magnusbaeck thanks for the answer.

I think you are right, that is strange though.... What about I want to query elasticsearch every 5m for example and execute the same pipeline again?

10

Since the elasticsearch input lacks a schedule option (unlike the jdbc input) you can always run Logstash via cron. But yes, it would be more convenient if this was built in.

11

How do I run logstash via cron? Using the jdbc input plugin?

12

What part are you finding difficult? I don't have time to explain cron in general but if there are any Logstash-specific issues I can chip in.

13

Thanks but i don't need you to eplain cron in general, but as you said, start logstash via cron is something new for me

14

Okay, but what's difficult about starting Logstash from cron? I'm sure you've started Logstash from a terminal before and I don't see why cron would be materially different.

15

Oh I see, you mean setting up a cronjob... I was thinking something inside logstash for instance using some plugins

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.