Hi,
I have installed logstash on my elastic machine and pointed to one of the log file as input, i have started logstash service, i don't see any logs coming into elastic either index in elastic, when i tail logstash log file looks everything fine.
logstash log file
[2019-02-28T22:35:18,311][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2019-02-28T22:35:22,278][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch ssl_certificate_verification=>false, template_name=>"loadtesttemaplate", hosts=>[//localhost:9200], index=>"loadtest-%{+YYYY.MM.dd}", manage_template=>false, id=>"36c65c8187301c6a89bc9244fd69799a8a5b1dad38e50222e3e434e76cf0ec63", document_type=>"loadtest", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_7b56ecc4-666d-4d9e-8ca9-3753ddcfbaae", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2019-02-28T22:35:22,332][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-02-28T22:35:22,919][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-02-28T22:35:23,172][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-02-28T22:35:23,256][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
_[2019-02-28T22:35:23,262][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document type {:es_version=>6}
[2019-02-28T22:35:23,299][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-02-28T22:35:23,797][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_10d87552dcaaf428d07d4ce3882b5665", :path=>["/root/benchmark"]}
[2019-02-28T22:35:23,873][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xde916db run>"}
[2019-02-28T22:35:23,968][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-02-28T22:35:23,992][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-02-28T22:35:24,403][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
here are the config files on elasticsearch machine for logstash
cat 02-beats-input.conf
input {
file {
path => "/root/benchmark"
}
}
cat 10-syslog-filter.conf
filter {
dissect {
mapping => { "message" => "%{topic} %{records_pushed} %{records_sec} %{avg_latency} %{max_latency} %{percentile_50} %{percentile_95} %{percentile_99} %{percentile_99_9}" }
}
}
cat 30-elasticsearch-output.conf
output {
elasticsearch {
hosts => ["localhost:9200"]
ssl_certificate_verification => false
manage_template => false
document_type => "loadtest"
index => "loadtest-%{+YYYY.MM.dd}"
template_name => "loadtesttemaplate"
}
}