Logstash not sending logs to elastic


(Muthu Kumar) #1

[2018-03-09T09:46:31,839][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2018-03-09T09:46:31,840][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-03-09T09:46:31,843][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-03-09T09:46:31,850][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.23.210.35:9200"]}
[2018-03-09T09:46:32,050][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x3e92988d@/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
[2018-03-09T09:46:32,061][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}


(Muthu Kumar) #2

Can any one help me on this?


(Robert Cowart) #3

I don't see any error here. Are you sure the data is being collected by logstash and isn't being discarded. You should add a stdout output to your config and ensure that data is actually making it to the output.


(Muthu Kumar) #4

Yes Robert..my config file now is as below

input {
file {
path => "/apps/appname/HUB/logs/2018-02-02/Default.log"
start_position => "beginning"
ignore_older => 0
}
}
output {
stdout {}
file {
path => "/tmp/output.txt"
}
}

But while executing below command
bin/logstash -e logstash-syslog.conf

[2018-03-09T10:25:00,225][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/apps/logstash/logstash-6.2.1/modules/fb_apache/configuration"}
[2018-03-09T10:25:00,230][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/apps/logstash/logstash-6.2.1/modules/netflow/configuration"}
[2018-03-09T10:25:00,454][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-03-09T10:25:00,545][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.1"}
[2018-03-09T10:25:00,592][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-03-09T10:25:00,595][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 3, column 1 (byte 76) after ", :backtrace=>["/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:90:in execute'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/apps/logstash/logstash-6.2.1/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

Error i am getting.please help.

The above test case to read a file using logstash and write it in one file


(Robert Cowart) #5

When specifying a pipeline config file you must use -f not -e


(Muthu Kumar) #6

ya i used the below command

bin/logstash -f logstash-syslog.conf

But no output.txt is written in target folder

Output as below

[2018-03-09T10:59:13,687][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/apps/logstash/logstash-6.2.1/modules/fb_apache/configuration"}
[2018-03-09T10:59:13,692][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/apps/logstash/logstash-6.2.1/modules/netflow/configuration"}
[2018-03-09T10:59:13,932][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-03-09T10:59:14,028][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.1"}
[2018-03-09T10:59:14,079][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-03-09T10:59:14,289][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-03-09T10:59:14,456][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x6cf4f4a4@/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
[2018-03-09T10:59:14,467][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}


(Muthu Kumar) #7

Do we need to keep on feed the data to the input file or it can read the old values also from that file?


(Robert Cowart) #8

Add sincedb_path => "/dev/null" to your file input so it will start over reading the file each time you start logstash. Otherwise it will read from where it last read which is probably the EOF, and the reason you get no data.


(Muthu Kumar) #9

thanks robert..for your help..i am able to see logs in kibana now :slight_smile:


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.