Logstash not able to create index on Elastic

Elastic Stack Logstash
1

Hi,

I have installed logstash on my elastic machine and pointed to one of the log file as input, i have started logstash service, i don't see any logs coming into elastic either index in elastic, when i tail logstash log file looks everything fine.

logstash log file

[2019-02-28T22:35:18,311][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.4"}
[2019-02-28T22:35:22,278][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch ssl_certificate_verification=>false, template_name=>"loadtesttemaplate", hosts=>[//localhost:9200], index=>"loadtest-%{+YYYY.MM.dd}", manage_template=>false, id=>"36c65c8187301c6a89bc9244fd69799a8a5b1dad38e50222e3e434e76cf0ec63", document_type=>"loadtest", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_7b56ecc4-666d-4d9e-8ca9-3753ddcfbaae", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2019-02-28T22:35:22,332][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-02-28T22:35:22,919][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-02-28T22:35:23,172][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-02-28T22:35:23,256][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
_[2019-02-28T22:35:23,262][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document type {:es_version=>6}
[2019-02-28T22:35:23,299][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-02-28T22:35:23,797][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_10d87552dcaaf428d07d4ce3882b5665", :path=>["/root/benchmark"]}
[2019-02-28T22:35:23,873][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xde916db run>"}
[2019-02-28T22:35:23,968][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-02-28T22:35:23,992][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-02-28T22:35:24,403][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

here are the config files on elasticsearch machine for logstash
cat 02-beats-input.conf

input {
file {
path => "/root/benchmark"
}
}

cat 10-syslog-filter.conf

filter {
dissect {
mapping => { "message" => "%{topic} %{records_pushed} %{records_sec} %{avg_latency} %{max_latency} %{percentile_50} %{percentile_95} %{percentile_99} %{percentile_99_9}" }
}
}

cat 30-elasticsearch-output.conf

output {
elasticsearch {
hosts => ["localhost:9200"]
ssl_certificate_verification => false
manage_template => false
document_type => "loadtest"
index => "loadtest-%{+YYYY.MM.dd}"
template_name => "loadtesttemaplate"
}
}

2

Are you appending lines to /root/benchmark?

If you want to see what filewatch is doing then enable --log.level trace (debug is not enough).

3

yes, i am appending lines manually to file, does it fetch appended lines like filebeat?

4

Yes. Try enabling --log.level trace and see if filewatch notices the new lines.

5

it's not finding any logs from the file i have created manually and also am keep amending lines into that file, given permissions as expected and group and user to that file is logstash, still not getting any data from that file.

if i point to logstash log file then it's collecting all the logs from it.

6

--log.level trace will show if it finds the file, how big it thinks the file is etc.

7

Thanks looks like it's working.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.