Hi ELKers,
I would like to ask about your help on the following issue, which it is that I cannot get input from elasticsearch into logstash.
My configuration is simple as that in the logstash-simple.conf:
input {
elasticsearch {
hosts => "localhost:9200"
query => '{ "query": { "match_all": {} } }'
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "test_logstash"
}
}
And the command that i use is the following:
bin/logstash --log.level debug -f config/logstash-simple.yml
When I run the logstash it seems that cannot get data from Elasticsearch for some reason and terminates itself. A sample output is:
Blockquote
[2020-01-20T10:29:07,007][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2020-01-20T10:29:07,008][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2020-01-20T10:29:07,098][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-01-20T10:29:07,150][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2020-01-20T10:29:07,154][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2020-01-20T10:29:07,161][DEBUG][logstash.outputs.elasticsearch] Found existing Elasticsearch template. Skipping template management {:name=>"logstash"}
[2020-01-20T10:29:07,161][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["//localhost:9200"]}
[2020-01-20T10:29:07,167][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2020-01-20T10:29:07,286][INFO ][logstash.pipeline ] Pipeline main started
[2020-01-20T10:29:07,316][DEBUG][logstash.agent ] Starting puma
[2020-01-20T10:29:07,320][DEBUG][logstash.agent ] Trying to start WebServer {:port=>9600}
[2020-01-20T10:29:07,331][DEBUG][logstash.api.service ] [api-service] start
[2020-01-20T10:29:07,371][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-01-20T10:29:07,373][DEBUG][logstash.inputs.elasticsearch] closing {:plugin=>"LogStash::Inputs::Elasticsearch"}
[2020-01-20T10:29:07,374][DEBUG][logstash.pipeline ] Input plugins stopped! Will shutdown filter/output workers.
[2020-01-20T10:29:07,418][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2020-01-20T10:29:07,419][DEBUG][logstash.pipeline ] Pushing shutdown {:thread=>"#<Thread:0x1f26c632 sleep>"}
[2020-01-20T10:29:07,419][DEBUG][logstash.pipeline ] Pushing shutdown {:thread=>"#<Thread:0x14dcc3d2 sleep>"}
[2020-01-20T10:29:07,420][DEBUG][logstash.pipeline ] Pushing shutdown {:thread=>"#<Thread:0x34c17ed1 run>"}
[2020-01-20T10:29:07,421][DEBUG][logstash.pipeline ] Pushing shutdown {:thread=>"#<Thread:0x55162334 sleep>"}
[2020-01-20T10:29:07,422][DEBUG][logstash.pipeline ] Shutdown waiting for worker thread #Thread:0x1f26c632
[2020-01-20T10:29:07,461][DEBUG][logstash.pipeline ] Shutdown waiting for worker thread #Thread:0x14dcc3d2
[2020-01-20T10:29:07,461][DEBUG][logstash.pipeline ] Shutdown waiting for worker thread #Thread:0x34c17ed1
[2020-01-20T10:29:07,463][DEBUG][logstash.pipeline ] Shutdown waiting for worker thread #Thread:0x55162334
[2020-01-20T10:29:07,463][DEBUG][logstash.outputs.elasticsearch] closing {:plugin=>"LogStash::Outputs::Elasticsearch"}
[2020-01-20T10:29:07,464][DEBUG][logstash.outputs.elasticsearch] Stopping sniffer
[2020-01-20T10:29:07,464][DEBUG][logstash.outputs.elasticsearch] Stopping resurrectionist
[2020-01-20T10:29:08,152][DEBUG][logstash.outputs.elasticsearch] Waiting for in use manticore connections
[2020-01-20T10:29:08,152][DEBUG][logstash.outputs.elasticsearch] Closing adapter #LogStash::Outputs::ElasticSearch::HttpClient::ManticoreAdapter:0x1c6336d9
[2020-01-20T10:29:08,153][DEBUG][logstash.pipeline ] Pipeline main has been shutdown
[2020-01-20T10:29:10,311][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2020-01-20T10:29:10,311][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2020-01-20T10:29:10,311][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping
[2020-01-20T10:29:10,312][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] PeriodicPoller: Stopping
[2020-01-20T10:29:10,314][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
[2020-01-20T10:29:10,315][DEBUG][logstash.pipeline ] Closing inputs
[2020-01-20T10:29:10,316][DEBUG][logstash.inputs.elasticsearch] stopping {:plugin=>"LogStash::Inputs::Elasticsearch"}
[2020-01-20T10:29:10,316][DEBUG][logstash.pipeline ] Closed inputs
Notes:
- I run the elasticsearch and the logstash on the same machine.
- When i try to give an input from stdin{} instead of the elasticseach it works fine and the new index "text_logstash" is created.
- A simple curl query to localhost:9200 works fine, I am getting the response back.
Your help is appreciated!!