Kibana not able to filter beat.hostname after certain day

[kko]

Hi,

I encountered a problem with Kibana. I have a series of indices in Elasticsearch created from metricbeat. In Kibana I created an index pattern for the series of indices. I am able to create a Timelion with .es(metric=avg:system.cpu.user.pct, index=Metricbeat-6.2.3*) up to the current time. However, when I try putting in a split query, it only show graph up to a certain date.

Thanks,
Kayson

[Marius_Dragomir]

Hello,

Can you be more specific on the split query? How does it look?

[kko]

Hi Marius,

Thank you for your reply.

I created an index from metricbeat feed from 3 hosts. When I do the following timelion. It give me graph up to now

But if I split on beat.hostname.keyword, it give me this

It’s missing the graph from April 22.

Thanks,

Kayson

[Marius_Dragomir]

The split in timelion always sorts by the absolut count of documents so this split will cause the top 4 hostnames by count to be shown.
It can be that these hostnames have just been shutdown in the end of the graph and the documents from the first chart are from other hostnames and thus not in the second one.
In order to check this, can you draw? .es(q=beat.name.keyword:SE123376), .es(q=beat.name.keyword:SE123376, metric=avg:system.cpu.user.pct)

[kko]

Hi Marius,

Yes, one host stopped at some point in time and that’s where the split graph stopped.

I am able to draw all host in separate .es series, e.g. .es(q=beat.name.keyword:SE123376, metric=avg:system.cpu.user.pct), .es(q=beat.name.keyword:SE123377, metric=avg:system.cpu.user.pct), etc.

``

So, is this the only way to visualize all hosts? I can’t use the split keyword, right?

``

Thanks,

kayson

[Marius_Dragomir]

Yep, this is pretty much the best way to get the info that you need.

[kko]

Hi Marius,

We are setting up our Logstash server to send log to our secure Enterprise Elasticsearch server.

However, I encountered some issue and seems like we need to setup the certificate for Logstash to connect ssl with our Enterprise ES server.

Is there anyway to configure logstash to ignore the ssl certificate configuration and run as insecure mode?

Below is the error log for logstash:

[2018-05-07T10:25:59,467][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5045"}

[2018-05-07T10:25:59,515][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5043"}

[2018-05-07T10:25:59,685][INFO ][logstash.pipeline ] Pipeline started {"pipeline.id"=>"main"}

[2018-05-07T10:25:59,700][INFO ][org.logstash.beats.Server] Starting server on port: 5043

[2018-05-07T10:25:59,700][INFO ][org.logstash.beats.Server] Starting server on port: 5045

[2018-05-07T10:25:59,704][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}

[2018-05-07T10:26:03,761][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://SRBL0SRVIMSSKibana:xxxxxx@d154888bca74484e9097b3893b2018ed.ece.saifg.rbc.com:9243/, :path=>"/"}

[2018-05-07T10:26:03,795][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://SRBL0SRVIMSSKibana:xxxxxx@d154888bca74484e9097b3893b2018ed.ece.saifg.rbc.com:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://SRBL0SRVIMSSKibana:xxxxxx@d154888bca74484e9097b3893b2018ed.ece.saifg.rbc.com:9243/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

[2018-05-07T10:26:08,804][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://SRBL0SRVIMSSKibana:xxxxxx@d154888bca74484e9097b3893b2018ed.ece.saifg.rbc.com:9243/, :path=>"/"}

[2018-05-07T10:26:08,821][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://SRBL0SRVIMSSKibana:xxxxxx@d154888bca74484e9097b3893b2018ed.ece.saifg.rbc.com:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://SRBL0SRVIMSSKibana:xxxxxx@d154888bca74484e9097b3893b2018ed.ece.saifg.rbc.com:9243/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

[2018-05-07T10:26:13,827][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://SRBL0SRVIMSSKibana:xxxxxx@d154888bca74484e9097b3893b2018ed.ece.saifg.rbc.com:9243/, :path=>"/"}

[2018-05-07T10:26:13,851][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://SRBL0SRVIMSSKibana:xxxxxx@d154888bca74484e9097b3893b2018ed.ece.saifg.rbc.com:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://SRBL0SRVIMSSKibana:xxxxxx@d154888bca74484e9097b3893b2018ed.ece.saifg.rbc.com:9243/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

[2018-05-07T10:26:18,859][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://SRBL0SRVIMSSKibana:xxxxxx@d154888bca74484e9097b3893b2018ed.ece.saifg.rbc.com:9243/, :path=>"/"}

[2018-05-07T10:26:18,871][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://SRBL0SRVIMSSKibana:xxxxxx@d154888bca74484e9097b3893b2018ed.ece.saifg.rbc.com:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://SRBL0SRVIMSSKibana:xxxxxx@d154888bca74484e9097b3893b2018ed.ece.saifg.rbc.com:9243/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

Thanks,

Kayson

[Marius_Dragomir]

Hello,
you ask this in the Logstash forums here:
https://discuss.elastic.co/c/logstash