Possbile Kibana bug for Logstash monitoring

[jclose (Jason)]

I noticed that the Monitoring page for Logstash seems to have datas that are not a proper sum of the values of the individual nodes.

I have attached screenshots. We are running 4 Logstash nodes, and they should be seeing 5k-7k eps. When I look at the individual nodes in Kibana, that is pretty consistent with what I'm seeing. But on the overview page, the stats are not right.

I'd like to know if I am, in fact, looking at a bug, or if there is something we are not doing correctly.

The aggregate screen. This is what I think is off. To me, the events received/emitted should be much larger.

Indexing total, showing 6-7k eps

Node 1 = ~1500eps

Node 2 = ~1500eps

Node 3 = ~1500eps

Node 4 = ~1200 eps

[pickypg (Chris Earle)]

Hi @jclose

Are you looking at Kibana 5.2.0 by any chance? There were numerous bug fixes to this screen in 5.2.1.

Let us know,
Chris

[_Ian]

@pickypg This is Kibana version 5.3.0

[jclose (Jason)]

Chris,

Is there anything that I can do to confirm if this is, in fact, a bug?

It really looks like the 'Overview' graph is a fair mirror of what is seen on the first node.

[pickypg (Chris Earle)]

Hi @jclose,

I'm not able to check today, but I should be able to check on this tomorrow. I was going to run some local Logstash nodes to reproduce this locally.

I know that we focused effort on calculating this properly for the overview (and I even checked to ensure that Kibana 5.3.0 had the fix) because it was a recognized bug, so I am shocked to see it's still occuring in 5.3.

I guess the one thing you could do as a result is to confirm that this is indeed Kibana 5.3? Kibana 5.2 is compatible with data coming from 5.3, so it's no guarantee that seeing 5.3 Logstash nodes means that you're running Kibana 5.2. To check, you can either go to Kibana's root directory and check its package.json for the version field or go to kibana_server:5601/status in the browser and check the @5.x.y version for plugins.

Thanks,
Chris

[jclose (Jason)]

We are using Kibana 5.3.0 for this.

Let me know what other information we can give you to help troubleshoot. I can give you more screenshots if you'd like.

[_Ian]

@ES05:/opt/kibana# cat package.json 
{
  "name": "kibana",
  "description": "Kibana is an open source (Apache Licensed), browser based analytics and search dashboard for Elasticsearch. Kibana is a snap to setup and start using. Kibana strives to be easy to get started with, while also being flexible and powerful, just like Elasticsearch.",
  "keywords": [
    "kibana",
    "elasticsearch",
    "logstash",
    "analytics",
    "visualizations",
    "dashboards",
    "dashboarding"
  ],
  "version": "5.3.0",
  "build": {
    "number": 14823,
    "sha": "ce7908cdac87af1e3b02ac4038fc3985602cf95a"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/elastic/kibana.git"
  },
  "engines": {
    "node": "6.9.5"
  }
}

[pickypg (Chris Earle)]

Hi @jclose and @_Ian,

This looks like the case of a bad merge. It somehow skipped the 5.3 branch, while making it into 5.2 (for 5.2.1) and 5.4 branches.

Thanks for pointing it out! I'm going to get this fixed right now, which should hopefully be in time for 5.3.1.

Thanks,
Chris

[_Ian]

Thanks! Do you have an idea on the release timeframe for 5.3.1? Just a quality-of-life issue for us as we troubleshoot pipeline speeds.

[pickypg (Chris Earle)]

I don't think that we have any public release date, but "soon" should be true. There have been a few bugs (like this) spotted that will make it into 5.3.1, which means we want it to get out sooner rather than later.

Thanks again for pointing this out; it was quite confusing since it simply missed the branch.

Chris

[Ryan_Groten]

Hey Chris, will 5.3.1 also contain a fix for this bug: Kibana Monitoring shows only 10 logstash nodes.
I'm on 5.3.0 and appear to be affected by the 10 node display limit bug as well (supposedly it's fixed in 5.2.1). Sorry for highjacking, the original thread is closed now.

Thanks,
Ryan

[pickypg (Chris Earle)]

Hi @Ryan_Groten,

I just confirmed that the fix is in 5.3.1 alongside the earlier issue!

Given this bug in ES 5.3.0, I expect 5.3.1 to be around the corner.

Hope that helps,
Chris

[Ryan_Groten]

Perfect, thanks @pickypg!