Elasticsearch input plugin not working

(VittorioP) #1

Hi all,

I'm trying a simple pipeline where I use elasticsearch to get data and send them again to elastic but updated.

For some reason the pipeline stops...

Here the config I'm using:

  	elasticsearch {
      hosts => "localhost"
      index => "dp_api-2017.04.28"
      docinfo => true
      query => '{"query": {"bool": {"must": [{"match": {"class": "DPAPINTERNAL"}},{"range": {"@timestamp": {"from": "now-10h","to": "now"}}}],"must_not": { "match": {"tags": "enriched"  }}}}}'
      tags => ["to_enrich"]
      scroll => "2m"
      codec => "plain"
  if "to_enrich" in [tags]{
    elasticsearch { #prendo i dpapi logs
      hosts => "localhost"
      index => "*"
      query_template => "/home/vittorio/Documents/offline-pipelines/conf.d-2/elastic-queries/matching-requestaw.json"
      fields => { "request" => "new_key" }
      add_tag => ["enriched", "output_splunk"]

	if "enriched" in [tags] {
    elasticsearch {
        hosts => "localhost"
        index => "%{index_name}-%{+YYYY.MM.dd}"
        document_id => "%{[@metadata][_id]}"
        action => "update"

There's no error from logstash in debug mode..

(VittorioP) #2

here is part (debug mode) of the shell result:

[2017-05-02T18:24:57,784][DEBUG][logstash.outputs.elasticsearch] Found existing Elasticsearch template. Skipping template management {:name=>"logstash"}
[2017-05-02T18:24:57,785][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x654e1406 URL://localhost>]}
[2017-05-02T18:24:57,790][INFO ][logstash.filters.elasticsearch] New ElasticSearch filter {:hosts=>["localhost"]}
[2017-05-02T18:24:57,855][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-05-02T18:24:57,866][INFO ][logstash.pipeline        ] Pipeline main started
[2017-05-02T18:24:57,877][DEBUG][logstash.agent           ] Starting puma
[2017-05-02T18:24:57,878][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2017-05-02T18:24:57,879][DEBUG][logstash.api.service     ] [api-service] start
[2017-05-02T18:24:58,106][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2017-05-02T18:24:58,167][DEBUG][logstash.inputs.elasticsearch] closing {:plugin=>"LogStash::Inputs::Elasticsearch"}
[2017-05-02T18:24:58,207][DEBUG][logstash.pipeline        ] Input plugins stopped! Will shutdown filter/output workers.
[2017-05-02T18:24:58,271][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2017-05-02T18:24:58,272][DEBUG][logstash.pipeline        ] Pushing shutdown {:thread=>"#<Thread:0x160a7fa7 run>"}
[2017-05-02T18:24:58,272][DEBUG][logstash.pipeline        ] Pushing shutdown {:thread=>"#<Thread:0x2cbcd052 sleep>"}
[2017-05-02T18:24:58,273][DEBUG][logstash.pipeline        ] Pushing shutdown {:thread=>"#<Thread:0xc0a9d16 sleep>"}
[2017-05-02T18:24:58,273][DEBUG][logstash.pipeline        ] Pushing shutdown {:thread=>"#<Thread:0x635717cb sleep>"}
[2017-05-02T18:24:58,273][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0x160a7fa7>
[2017-05-02T18:24:58,300][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0x2cbcd052>
[2017-05-02T18:24:58,301][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0xc0a9d16>
[2017-05-02T18:24:58,301][DEBUG][logstash.pipeline        ] Shutdown waiting for worker thread #<Thread:0x635717cb>
[2017-05-02T18:24:58,301][DEBUG][logstash.filters.elasticsearch] closing {:plugin=>"LogStash::Filters::Elasticsearch"}
[2017-05-02T18:24:58,302][DEBUG][logstash.outputs.stdout  ] closing {:plugin=>"LogStash::Outputs::Stdout"}
[2017-05-02T18:24:58,302][DEBUG][logstash.outputs.elasticsearch] closing {:plugin=>"LogStash::Outputs::ElasticSearch"}
[2017-05-02T18:24:58,304][DEBUG][logstash.outputs.elasticsearch] Stopping sniffer
[2017-05-02T18:24:58,304][DEBUG][logstash.outputs.elasticsearch] Stopping resurrectionist
[2017-05-02T18:24:58,625][DEBUG][logstash.outputs.elasticsearch] Waiting for in use manticore connections
[2017-05-02T18:24:58,626][DEBUG][logstash.outputs.elasticsearch] Closing adapter #<LogStash::Outputs::ElasticSearch::HttpClient::ManticoreAdapter:0x3c52d27e>
[2017-05-02T18:24:58,627][DEBUG][logstash.pipeline        ] Pipeline main has been shutdown
[2017-05-02T18:25:00,877][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2017-05-02T18:25:00,877][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2017-05-02T18:25:00,878][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping
[2017-05-02T18:25:00,880][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}
[2017-05-02T18:25:00,881][DEBUG][logstash.pipeline        ] Closing inputs
[2017-05-02T18:25:00,881][DEBUG][logstash.inputs.elasticsearch] stopping {:plugin=>"LogStash::Inputs::Elasticsearch"}
[2017-05-02T18:25:00,881][DEBUG][logstash.pipeline        ] Closed inputs

(VittorioP) #3

Please somebody take a look at this.

(VittorioP) #4

is it normal maybe that when it finishes process the data it stops the pipeline?


Hello Vittorio,

Are you try to add port number to your elasticsearch in the output plugin?

(VittorioP) #6

hello @Baco, thanks for your answer!

I'm not sure I've understood your question, have you asked if I tried to put port number in the output or are you asking if I'm trying to add port number?

If you refer to the first question, yes I tried to do "localhost:9200"


Yes, my question was if you did "localhost:9200"

Sorry, but I will continue to look the configuration.

(Magnus Bäck) #8

Is the elasticsearch input even supposed to keep Logstash alive? I suspect it's designed to fire off the query once, send the results down the pipeline, and then shut down Logstash.

(VittorioP) #9

Hi @magnusbaeck thanks for the answer.

I think you are right, that is strange though.... What about I want to query elasticsearch every 5m for example and execute the same pipeline again?

(Magnus Bäck) #10

Since the elasticsearch input lacks a schedule option (unlike the jdbc input) you can always run Logstash via cron. But yes, it would be more convenient if this was built in.

(VittorioP) #11

How do I run logstash via cron? Using the jdbc input plugin?

(Magnus Bäck) #12

What part are you finding difficult? I don't have time to explain cron in general but if there are any Logstash-specific issues I can chip in.

(VittorioP) #13

Thanks but i don't need you to eplain cron in general, but as you said, start logstash via cron is something new for me

(Magnus Bäck) #14

Okay, but what's difficult about starting Logstash from cron? I'm sure you've started Logstash from a terminal before and I don't see why cron would be materially different.

(VittorioP) #15

Oh I see, you mean setting up a cronjob... I was thinking something inside logstash for instance using some plugins

(system) #16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.