[2018-03-09T09:46:31,839][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2018-03-09T09:46:31,840][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-03-09T09:46:31,843][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-03-09T09:46:31,850][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.23.210.35:9200"]}
[2018-03-09T09:46:32,050][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x3e92988d@/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
[2018-03-09T09:46:32,061][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
Can any one help me on this?
I don't see any error here. Are you sure the data is being collected by logstash and isn't being discarded. You should add a stdout output to your config and ensure that data is actually making it to the output.
Yes Robert..my config file now is as below
input {
file {
path => "/apps/appname/HUB/logs/2018-02-02/Default.log"
start_position => "beginning"
ignore_older => 0
}
}
output {
stdout {}
file {
path => "/tmp/output.txt"
}
}
But while executing below command
bin/logstash -e logstash-syslog.conf
[2018-03-09T10:25:00,225][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/apps/logstash/logstash-6.2.1/modules/fb_apache/configuration"}
[2018-03-09T10:25:00,230][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/apps/logstash/logstash-6.2.1/modules/netflow/configuration"}
[2018-03-09T10:25:00,454][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-03-09T10:25:00,545][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.1"}
[2018-03-09T10:25:00,592][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-03-09T10:25:00,595][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 3, column 1 (byte 76) after ", :backtrace=>["/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/agent.rb:90:in execute'", "/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/apps/logstash/logstash-6.2.1/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
Error i am getting.please help.
The above test case to read a file using logstash and write it in one file
When specifying a pipeline config file you must use -f not -e
ya i used the below command
bin/logstash -f logstash-syslog.conf
But no output.txt is written in target folder
Output as below
[2018-03-09T10:59:13,687][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/apps/logstash/logstash-6.2.1/modules/fb_apache/configuration"}
[2018-03-09T10:59:13,692][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/apps/logstash/logstash-6.2.1/modules/netflow/configuration"}
[2018-03-09T10:59:13,932][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-03-09T10:59:14,028][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.1"}
[2018-03-09T10:59:14,079][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-03-09T10:59:14,289][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-03-09T10:59:14,456][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x6cf4f4a4@/apps/logstash/logstash-6.2.1/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
[2018-03-09T10:59:14,467][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
Do we need to keep on feed the data to the input file or it can read the old values also from that file?
Add sincedb_path => "/dev/null" to your file input so it will start over reading the file each time you start logstash. Otherwise it will read from where it last read which is probably the EOF, and the reason you get no data.
thanks robert..for your help..i am able to see logs in kibana now 
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.