Logstash is not connecting with ElasticSearch and I don't find any error

Alvaro_Sanz_Garrigue · April 27, 2017, 8:23am

Hell I'm new with elasticSearch and logstahs and I'm having some problems.

I'm Trying to connect Logstash and ES with a simple conf file but nothing happens in ES and I can get the error from the logs.

input {

 	file{
 		path => ["/Users/asanzgarrigues/Desktop/logstash-plain.log"]
 		start_position => "beginning"
 		type => "logs"
 	}
 }
 # The filter part of this file is commented out to indicate that it is
 # optional.
 # filter {
 #
 # }
 output {
 elasticsearch { 
 	 hosts => ["http://localhost:9203"]
 	 action => "index"
 	 index => "prueba"
 	 } 
     stdout { codec => rubydebug }
 }

The ES port is 9203 and I change it in the Elasticsearch.yml configuration file.

When I executed logstahs with:

C:\Users\asanzgarrigues\Desktop\elastic\logstash-5.3.0\bin>logstash -f prueba.conf

I get the following response:

Sending Logstash's logs to C:/Users/asanzgarrigues/Desktop/elastic/logstash-5.3.0/logs which is now configured via log4j2.properties
[2017-04-27T10:13:59,271][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9203/]}}
[2017-04-27T10:13:59,271][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9203/, :path=>"/"}
[2017-04-27T10:13:59,458][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x78fe632b URL:http://localhost:9203/>}
[2017-04-27T10:13:59,458][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-04-27T10:13:59,521][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-04-27T10:13:59,536][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>[#<URI::HTTP:0x5fd8227c URL:http://localhost:9203>]}
[2017-04-27T10:13:59,536][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-04-27T10:14:00,593][INFO ][logstash.pipeline ] Pipeline main started
[2017-04-27T10:14:00,749][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

The execution with --debug is too big but it finalizes in a loop like this:

[2017-04-27T10:18:56,055][DEBUG][logstash.inputs.file ] _globbed_files: /Users/asanzgarrigues/Desktop/logstash-plain.log: glob is: ["/Users/asanzgarrigues/Desktop/logstash-plain.log"]
[2017-04-27T10:19:00,764][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-04-27T10:19:05,766][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-04-27T10:19:10,774][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-04-27T10:19:11,125][DEBUG][logstash.inputs.file ] _globbed_files: /Users/asanzgarrigues/Desktop/logstash-plain.log: glob is: ["/Users/asanzgarrigues/Desktop/logstash-plain.log"]
[2017-04-27T10:19:15,781][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-04-27T10:19:20,788][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-04-27T10:19:25,793][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-04-27T10:19:26,196][DEBUG][logstash.inputs.file ] _globbed_files: /Users/asanzgarrigues/Desktop/logstash-plain.log: glob is: ["/Users/asanzgarrigues/Desktop/logstash-plain.log"]

If the complete debug log is necessary i can post it in various posts

I need your help, I'm blocked with it.

Thank you!

Ruthu_Ramya · April 27, 2017, 9:02am

@Alvaro_Sanz_Garrigue,
Try these options

check if ES is working at 9203, check http://localhost:9203 in browser and verify
can you try to touch the log file.
Ex: delete the last line in the log file while Logstash is active
add a sample filter and check
output to a file so you are sure that some data is parsed and ready to be shipped out of logstash

Regards,
Ruthu

zegdene · April 27, 2017, 2:16pm

Hi ruthu ,
can you me explain the path to the log file ?? i have the same error
Thank you

Alvaro_Sanz_Garrigue · April 27, 2017, 3:22pm

Thanks for your tips, but it still not working.

1.check if ES is working at 9203, check http://localhost:92032 in browser and verify
I have checked it and it seems that it is working correctly.

2. can you try to touch the log file.
I did it
3. add a sample filter and check
4. output to a file so you are sure that some data is parsed and ready to be shipped out of logstash
The new .conf file is the following:

> input {
> 	file{
> 		path => ["/Users/asanzgarrigues/Desktop/logstash-plain.log"]
> 		start_position => "beginning"
> 		type => "logs"
> 	}
> }

> filter {
> grok {
>     match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{LOGLEVEL:log_level} \]\[%{DATA:info}\]%{GREEDYDATA:message}"  }
>   }
>  }
> output {

> 	file{
> 		path => ["/Users/asanzgarrigues/Desktop/prueba.json"]
> 		}
>     stdout { codec => rubydebug }
> }

The log format is the following:

[2017-04-26T15:05:18,794][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9203/]}}

When I execute again logstash now appears the following but nothing is written in the destination file.

C:\Users\asanzgarrigues\Desktop\elastic\logstash-5.3.0\bin>logstash -f prueba.conf
Sending Logstash's logs to C:/Users/asanzgarrigues/Desktop/elastic/logstash-5.3.0/logs which is now configured via log4j2.properties
[2017-04-27T17:58:21,118][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-04-27T17:58:21,899][INFO ][logstash.pipeline        ] Pipeline main started
[2017-04-27T17:58:22,087][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Now I'm totally lost hahaha. I supose that I'm missing something during the configuration or installation. But I just download ES and logstash zips from elastic.co and cjange the elastic port configuration and execute the commands bin\elasticsearch.bat and bin/logstash -f logstash.conf indicated in https://www.elastic.co/downloads/elasticsearch
and
https://www.elastic.co/downloads/logstash

Sorry if my english is too bad. And Thank you for help @Ruthu_Ramya

Alvaro_Sanz_Garrigue · April 28, 2017, 8:18am

I finally make it works. I was doing it in the enterprise laptop which is restricted. I create a Ubuntu VM where I have the whole control, and there with the same configuration it finally works.
So I guess that in my enterprise computer there were permissions issues or something like that which denies logstash to work as expected.

You can close this topic.
Thank you

Ruthu_Ramya · April 28, 2017, 9:01am

@zegdene, try as in the sample posted here. Thats the right one.
You can use stdout if you want to see the results on the console, refer: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-stdout.html

system · May 26, 2017, 9:11am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.