No indexes, no error logs

Elastic Stack Logstash
1

I've been trying to push logs with Logstash into 3-node elasticsearch cluster. I start logstash, logstash logs all fine, but no logs appear in ES. I've 3 nodes, both data and master-eligible nodes, and one among them is an injest node.

My logstash pipeline postfix.conf:

output {
elasticsearch {
hosts => [ "elk00-pco.easydns.vpn:9200", "elk01-pco.easydns.vpn:9200", "elk02-pco.easydns.vpn:9200" ]
user => "logstash_logs"
password => "logstash_logs."
ssl => true
cacert => "/etc/logstash/ca.pem"
index => "logstash-postfix-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}

User logstash_logs has a role under the same name, which has privileges monitoring, manage_index_templates, and all privileges to logstash-* indices.

The logstash log (some lines omitted):

[2017-12-22T13:17:43,018][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-12-22T13:17:44,673][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://logstash_system:xxxxxx@elk00-pco.easydns.vpn:9200/, https://logstash_system:xxxxxx@elk01-pco.easydns.vpn:9200/, https://logstash_system:xxxxxx@elk02-pco.easydns.vpn:9200/]}}
[2017-12-22T13:17:44,677][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@elk00-pco.easydns.vpn:9200/, :path=>"/"}
[2017-12-22T13:17:45,423][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://logstash_system:xxxxxx@elk02-pco.easydns.vpn:9200/"}
[2017-12-22T13:17:45,440][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["https://elk00-pco.easydns.vpn:9200", "https://elk01-pco.easydns.vpn:9200", "https://elk02-pco.easydns.vpn:9200"]}
[2017-12-22T13:17:45,441][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2, :thread=>"#<Thread:0x1f40dc3d@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290 run>"}
[2017-12-22T13:17:45,574][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://logstash_system:xxxxxx@elk00-pco.easydns.vpn:9200/, https://logstash_system:xxxxxx@elk01-pco.easydns.vpn:9200/, https://logstash_system:xxxxxx@elk02-pco.easydns.vpn:9200/]}}
[2017-12-22T13:17:45,579][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@elk00-pco.easydns.vpn:9200/, :path=>"/"}
[2017-12-22T13:17:45,631][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"https://logstash_system:xxxxxx@elk00-pco.easydns.vpn:9200/"}
[2017-12-22T13:17:45,867][INFO ][logstash.pipeline ] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2017-12-22T13:17:47,404][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://logstash_logs:xxxxxx@elk00-pco.easydns.vpn:9200/, https://logstash_logs:xxxxxx@elk01-pco.easydns.vpn:9200/, https://logstash_logs:xxxxxx@elk02-pco.easydns.vpn:9200/]}}
[2017-12-22T13:17:47,408][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_logs:xxxxxx@elk00-pco.easydns.vpn:9200/, :path=>"/"}
[2017-12-22T13:17:47,590][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://logstash_logs:xxxxxx@elk00-pco.easydns.vpn:9200/"}
[2017-12-22T13:17:47,617][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_logs:xxxxxx@elk01-pco.easydns.vpn:9200/, :path=>"/"}
[2017-12-22T13:17:47,891][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://logstash_logs:xxxxxx@elk01-pco.easydns.vpn:9200/"}
[2017-12-22T13:17:47,897][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_logs:xxxxxx@elk02-pco.easydns.vpn:9200/, :path=>"/"}
[2017-12-22T13:17:48,039][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://logstash_logs:xxxxxx@elk02-pco.easydns.vpn:9200/"}
[2017-12-22T13:17:48,049][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-12-22T13:17:48,057][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-12-22T13:17:48,072][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["//elk00-pco.easydns.vpn:9200", "//elk01-pco.easydns.vpn:9200", "//elk02-pco.easydns.vpn:9200"]}
[2017-12-22T13:17:48,276][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250, :thread=>"#<Thread:0x37340b02@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290 run>"}
[2017-12-22T13:17:48,554][INFO ][logstash.pipeline ] Pipeline started {"pipeline.id"=>"main"}
[2017-12-22T13:17:48,577][INFO ][logstash.agent ] Pipelines running {:count=>2, :pipelines=>[".monitoring-logstash", "main"]}
[2017-12-22T13:17:48,579][INFO ][logstash.inputs.metrics ] Monitoring License OK

All checks out fine. I did get this configuration working once. Then, I ran _all index delete, reconfigured x-pack authentication, and now, no logs/indices are showing up in Kibana. How to track down the issue?

Thanks.

2

How do you know Logstash has events to process and send to ES? What does your input configuration look like?

3

@magnusbaeck

How do you know Logstash has events to process and send to ES?

Before deleting all the index, I'd run logstash with the same pipeline config and logs had ended up in ES.

Input config:

input {
file {
type => "postfix" #to differentiate between postfix for mx-caprica, esmtp.
path => "/usr/share/logstash/email_qmgr.log"
start_position => "beginning"
}
}

4

I would recommend that you install X-Pack in order to enable Logstash monitoring. This will clearly show you if any data is flowing through Logstash or not.

5

@Christian_Dahlqvist, this is the output of my monitoring section after I ran the logstash pipeline for a minute. Clearly, events were sent:

[Album] Imgur: The magic of the Internet

But, when I navigate to the 'Dashboard' to view the logs, it redirects me to the 'Management' section where it says - Unable to fetch mapping of the format logstash-*

[Album] Imgur: The magic of the Internet

But, if you look at my logstash output snippet, I've clearly mentioned index => "logstash-postfix-%{+YYYY.MM.dd}". What's is the issue here?

Thanks.

6

What is the output of the cat indices API?

7

@Christian_Dahlqvist, here it is:

curl -k -u elastic -XGET 'https://elk03-pco.easydns.vpn:9200/_cat/indices/logstash*?v&s=index'

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-postfix-2017.12.26 uL50rgHJQLqVTHTBKSMGyw 5 1 168404 0 48.6mb 24.1mb

So, indices were created, it's just kibana not showing me the logs?

Thanks again.

8

Is Kibana able to access Elasticsearch? Can you try opening Console (under Dev Tools) and run the 'cat indices' command from there?

9

Yes, that's also a success:

[Album] Imgur: The magic of the Internet

(I'm logged in as 'kibana' user into Kibana dashboard)

10

The 'kibana' user is an internal role that does not have access to any data in the cluster. You need to create a new user that gets the 'kibana_user' role (can access the .kibana index) together with one or more roles that give access to the 'logstash-*' indices.

1 Like
11

Solved!

Yes, it was the user privileges that was causing the issue. 'kibana' (the reserved user) didn't have any access to logstash-* indices.

Thank you very much :slight_smile:

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.