No results in Kibana with metricbeat

saiterio · December 20, 2016, 3:48pm

Hi,

I'm using Metricbeat 5.1.1 ==> Elastic Search 5.1.1 ==> Kibana 5.1.1, I used import_dashboards script to import metricbeat dashboards in Kibana.
All components are installed on the same machine (localhost).

When I request Elastic Search with http://localhost:9200/metricbeat-*/_search?pretty , I get the following response:

{"took":9,"timed_out":false,"_shards":{"total":10,"successful":10,"failed":0},"hits":{"total":4808372,"max_score":1.0,"hits":[{"_index":"metricbeat-2016.12.19","_type":"metricsets","_id":"AVkXv8Cafy-kld0Gs4-S","_score":1.0,"_source":{"@timestamp":"2016-12-19T15:42:29.606Z","system":{"process":{"memory":{"rss":{"pct":0.0,"bytes":0},"size":0,"share":0},"pgid":0,"name":"kfd_process_wq","cpu":{"start_time":"2016-12-19T10:12:36.000Z","total":{"pct":0.0}},"pid":584,"state":"sleeping","fd":{"limit":{"hard":4096,"soft":1024},"open":0},"username":"root","ppid":2}},"beat":{"hostname":"iota","name":"iota","version":"5.1.1"},"@version":"1","host":"iota","metricset":{"rtt":146217,"module":"system","name":"process"},"type":"metricsets","tags":["beats_input_raw_event"]}}....

It sounds OK for me.
Although the default index used in kibana is metricbeat-, in discover tab, I can read "No results found". When I check the request done by kibana (http://localhost:9200/metricbeat-/mapping/field/*?=1482248436057&ignore_unavailable=false&allow_no_indices=false&include_defaults=true), I get an error "index not found":

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","index":"metricbeat-*"}],"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","index":"metricbeat-*"},"status":404}

I checked that I have only one instance of elastic search.

What could I miss?

Thank you,
Damien

Stacey_Gammon · December 20, 2016, 4:16pm

That url doesn't work because it's missing the * after metricbeat-.

Is your index in kibana metricbeat- or metricbeat-*? The import_dashboards script should have used the latter, and I verified that it is working correctly for me. If you are missing the *, try adding a new index manually and see if it works.

saiterio · December 20, 2016, 4:39pm

Hi Stacey,

Thank you for your help and your time.

I checked the URL, and it's http://localhost:5601/elasticsearch/metricbeat-/_mapping/field/?_=1482251759844&ignore_unavailable=false&allow_no_indices=false&include_defaults=true, so the * was not missing.

However, I try to reinstall all the components without x-pack, and it works, I succeed to discover data from kibana, but when I try to re-install x-pack, it doesn't work. (the user/password are rightly defined in kibana and metricbeat configuration), and the error is still index not found...

Regards,
Damien

Stacey_Gammon · December 20, 2016, 4:57pm

Strange.

Did you delete the index before re-installing x-pack? Is the issue that the index is not being created from the import dashboards script when x-pack is installed? Or did the index disappear after installing x-pack?

I ran through those steps but everything is continuing to work for me.

saiterio · December 20, 2016, 5:42pm

No I didn't remove the index before re-installing x-pack, but I just tried to do it, and I get still the same issue...
When I remove or install x-pack, the index is not deleted. Moreover if I delete the index, and re-create it, I keep the same issue with x-pack.
I tried to add manually the index metricbeat, It works without x-pack, but It doesn't work with... I have the issue if I create manually logstash index... I think I miss something....

Are you using 5.1.1?

My configuration:
Ubuntu 14.04
ELK 5.1.1
Java 8

Stacey_Gammon · December 20, 2016, 6:38pm

Is metricbeat output showing any errors?

If you uninstall x-pack and re-create your metricbeat index so everything is working, can you show me the output of http://localhost:9200/_cat/indices? Then can you install x-pack and visit that url again and tell me the output? It should be the same, the only difference should be that after you install x-pack it will ask for a username and password. I want to see if your metricbeat index shows up both times.

saiterio · December 21, 2016, 7:18am

I see no error log for metricbeat, Elasticsearch and kibana.

http://localhost:9200/_cat/indices with x-pack returns:

yellow open metricbeat-2016.12.19 wx-am55QSLKXJcSGzpRY2Q 5 1 270211 0 100.8mb 100.8mb
yellow open .monitoring-es-2-2016.12.19 HsdoID4oSCyKh4oXqrgyDQ 1 1 82533 417 37mb 37mb
yellow open metricbeat-2016.12.20 Lz8JROgbQ4Sh7fJln17bQQ 5 1 4556531 0 823.7mb 823.7mb
yellow open .monitoring-kibana-2-2016.12.19 kze_nHhCTfurbgjLxQthrA 1 1 11839 0 2.7mb 2.7mb
yellow open .monitoring-data-2 KZjD8qtFQ4WOg6lpEXZnNA 1 1 3 0 7.1kb 7.1kb
yellow open .monitoring-es-2-2016.12.20 CdNLFcJvSlOCdTOLxc11vA 1 1 102543 224 50mb 50mb
yellow open .monitoring-es-2-2016.12.16 aJ8VBIEzR5un2nu0UCfpfQ 1 1 14180 48 5.4mb 5.4mb
yellow open filebeat-2016.12.19 cJX2xBoQQB6tJQ5fa-ZVPg 5 1 311 0 199.4kb 199.4kb
yellow open .monitoring-kibana-2-2016.12.16 Xat608dwREWMvyKe2arFww 1 1 150 0 87.5kb 87.5kb
yellow open logstash-2016.12.19 OBvP3jEpTJ2MfGggPmAyYA 5 1 33248 0 12.1mb 12.1mb
yellow open .monitoring-kibana-2-2016.12.20 kFdahubUTuKAqw_l9RVBNg 1 1 11820 0 2.7mb 2.7mb
yellow open .kibana MHaGnUimQfOxYB0A_iaR-A 1 1 95 76 291.1kb 291.1kb

http://localhost:9200/_cat/indices with x-pack removed:

yellow open metricbeat-2016.12.19 wx-am55QSLKXJcSGzpRY2Q 5 1 270211 0 100.8mb 100.8mb
yellow open .monitoring-es-2-2016.12.19 HsdoID4oSCyKh4oXqrgyDQ 1 1 82533 417 37mb 37mb
yellow open .monitoring-kibana-2-2016.12.21 3wl2HKnwS0yBxoWC_Npdqg 1 1 108 0 54.8kb 54.8kb
yellow open metricbeat-2016.12.20 Lz8JROgbQ4Sh7fJln17bQQ 5 1 4556531 0 823.7mb 823.7mb
yellow open .monitoring-es-2-2016.12.20 CdNLFcJvSlOCdTOLxc11vA 1 1 102543 224 50mb 50mb
yellow open .monitoring-data-2 KZjD8qtFQ4WOg6lpEXZnNA 1 1 3 0 7.1kb 7.1kb
yellow open .monitoring-es-2-2016.12.16 aJ8VBIEzR5un2nu0UCfpfQ 1 1 14180 48 5.4mb 5.4mb
yellow open logstash-2016.12.19 OBvP3jEpTJ2MfGggPmAyYA 5 1 33248 0 12.1mb 12.1mb
yellow open .monitoring-es-2-2016.12.21 QnZ9mBK8SFGO8Pi7ylm6MQ 1 1 1484 140 800.5kb 800.5kb
yellow open .monitoring-kibana-2-2016.12.19 kze_nHhCTfurbgjLxQthrA 1 1 11839 0 2.7mb 2.7mb
yellow open filebeat-2016.12.19 cJX2xBoQQB6tJQ5fa-ZVPg 5 1 311 0 199.4kb 199.4kb
yellow open metricbeat-2016.12.21 j3XGjhigR-2P0Opmn3xGtg 5 1 2183 0 1.1mb 1.1mb
yellow open .monitoring-kibana-2-2016.12.16 Xat608dwREWMvyKe2arFww 1 1 150 0 87.5kb 87.5kb
yellow open .kibana MHaGnUimQfOxYB0A_iaR-A 1 1 95 50 248.8kb 248.8kb
yellow open .monitoring-kibana-2-2016.12.20 kFdahubUTuKAqw_l9RVBNg 1 1 11820 0 2.7mb 2.7mb

I also tried http://localhost:5601/elasticsearch/_cat/indices/ with x-pack installed, and I get this response:

yellow open metricbeat-2016.12.19 wx-am55QSLKXJcSGzpRY2Q 5 1 270211 0 100.8mb 100.8mb
yellow open metricbeat-2016.12.20 Lz8JROgbQ4Sh7fJln17bQQ 5 1 4556531 0 823.7mb 823.7mb
yellow open .monitoring-kibana-2-2016.12.21 3wl2HKnwS0yBxoWC_Npdqg 1 1 452 0 195.9kb 195.9kb
yellow open .monitoring-es-2-2016.12.19 HsdoID4oSCyKh4oXqrgyDQ 1 1 82533 417 37mb 37mb
yellow open .monitoring-es-2-2016.12.20 CdNLFcJvSlOCdTOLxc11vA 1 1 102543 224 50mb 50mb
yellow open .monitoring-data-2 KZjD8qtFQ4WOg6lpEXZnNA 1 1 3 0 14kb 14kb
yellow open .monitoring-es-2-2016.12.16 aJ8VBIEzR5un2nu0UCfpfQ 1 1 14180 48 5.4mb 5.4mb
yellow open logstash-2016.12.19 OBvP3jEpTJ2MfGggPmAyYA 5 1 33248 0 12.1mb 12.1mb
yellow open .monitoring-es-2-2016.12.21 QnZ9mBK8SFGO8Pi7ylm6MQ 1 1 5297 490 3.4mb 3.4mb
yellow open .monitoring-kibana-2-2016.12.19 kze_nHhCTfurbgjLxQthrA 1 1 11839 0 2.7mb 2.7mb
yellow open filebeat-2016.12.19 cJX2xBoQQB6tJQ5fa-ZVPg 5 1 311 0 199.4kb 199.4kb
yellow open metricbeat-2016.12.21 j3XGjhigR-2P0Opmn3xGtg 5 1 5623 0 4.1mb 4.1mb
yellow open .monitoring-kibana-2-2016.12.16 Xat608dwREWMvyKe2arFww 1 1 150 0 87.5kb 87.5kb
yellow open .kibana MHaGnUimQfOxYB0A_iaR-A 1 1 95 51 273.6kb 273.6kb
yellow open .monitoring-kibana-2-2016.12.20 kFdahubUTuKAqw_l9RVBNg 1 1 11820 0 2.7mb 2.7mb

Thank you,
Damien

saiterio · December 21, 2016, 9:22am

Hi,

When I create a new user with superuser role, I can add manually a new index, the index metricbeat-* is found.

However, I have still an issue on discover tab, where no result is displayed. I saw on kibana that the parameters in http://localhost:5601/elasticsearch/_msearch request are not the same with or without x-pack:

with x-pack:

{"index":[".kibana"],"ignore_unavailable":true,"preference":1482308582153}
{"query":{"bool":{"must_not":[{"match_all":{}}]}}}

-without x-pack:

{"index":["metricbeat-2016.12.21"],"ignore_unavailable":true,"preference":1482309012376}
{"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"query":{"bool":{"must":[{"query_string":{"analyze_wildcard":true,"query":""}},{"range":{"@timestamp":{"gte":1482308116549,"lte":1482309016549,"format":"epoch_millis"}}}],"must_not":[]}},"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"":{}},"require_field_match":false,"fragment_size":2147483647},"_source":{"excludes":},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"30s","time_zone":"Europe/Berlin","min_doc_count":1}}},"stored_fields":["*"],"script_fields":{},"docvalue_fields":["@timestamp","docker.container.created","mongodb.status.local_time","mongodb.status.background_flushing.last_finished","postgresql.activity.backend_start","postgresql.activity.transaction_start","postgresql.activity.query_start","postgresql.activity.state_change","postgresql.bgwriter.stats_reset","postgresql.database.stats_reset","system.process.cpu.start_time"]}

In both case, the url in the browser navigation bar is http://localhost:5601/app/kibana#/discover?_g=()&_a=(columns:!(_source),index:'metricbeat-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'*')),sort:!(%27@timestamp%27,desc))

Here is the screen "discover" on kibana with x-pack:

Do you have an idea of what is misconfigured on my platform?

Thank you,
Damien

saiterio · January 3, 2017, 2:23pm

Hello,

I still have the issue, is there someone have an idea?

Thank you,
Damien

Stacey_Gammon · January 3, 2017, 3:30pm

Can you change the time picker to Last 30 days instead of Last 15 minutes? I see your indexes are for the date range of 2016-12-19 to 2016-12-21 so I want to make sure the time picker is including those date ranges.

If you run the following query in dev tools, both with and without x-pack installed, can you paste the results?

POST metric*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "analyze_wildcard": true,
            "query": "*"
          }
        }
      ]
    }
  }
}

I see you have the index in both situations, and it is the same size, so data should exist in your elasticsearch instance.

Make sure you also restart metricbeat after you installed x-pack so it picks up the credentials.

saiterio · January 3, 2017, 4:44pm

Hi Stacey,

Without x-pack:

{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 15,
    "successful": 15,
    "failed": 0
  },
  "hits": {
    "total": 6543,
    "max_score": 1,
    "hits": [
      {
        "_index": "metricbeat",
        "_type": "metricsets",
        "_id": "AVllBkWgmNy-OguxyKZv",
        "_score": 1,
        "_source": {
          "@timestamp": "2017-01-03T15:17:16.268Z",
          "beat": {
            "hostname": "iota",
            "name": "iota",
            "version": "5.1.1"
          },
          "metricset": {
            "module": "system",
            "name": "process",
            "rtt": 105952
          },
          "system": {
            "process": {
              "cmdline": "/usr/lib/gvfs/gvfs-gphoto2-volume-monitor",
              "cpu": {
                "start_time": "2017-01-03T08:33:34.000Z",
                "total": {
                  "pct": 0
                }
              },
              "fd": {
                "limit": {
                  "hard": 4096,
                  "soft": 1024
                },
                "open": 7
              },
              "memory": {
                "rss": {
                  "bytes": 3543040,
                  "pct": 0.001
                },
                "share": 3543040,
                "size": 217858048
              },
              "name": "gvfs-gphoto2-vo",
              "pgid": 2094,
              "pid": 2491,
              "ppid": 2028,
              "state": "sleeping",
              "username": "damien"
            }
          },
          "type": "metricsets"
        }
      },
     ....
    ]
  }
}

with x-pack:

{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 15,
    "successful" : 15,
    "failed" : 0
  },
  "hits" : {
    "total" : 7065,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "metricbeat",
        "_type" : "metricsets",
        "_id" : "AVllBkWgmNy-OguxyKZv",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2017-01-03T15:17:16.268Z",
          "beat" : {
            "hostname" : "iota",
            "name" : "iota",
            "version" : "5.1.1"
          },
          "metricset" : {
            "module" : "system",
            "name" : "process",
            "rtt" : 105952
          },
          "system" : {
            "process" : {
              "cmdline" : "/usr/lib/gvfs/gvfs-gphoto2-volume-monitor",
              "cpu" : {
                "start_time" : "2017-01-03T08:33:34.000Z",
                "total" : {
                  "pct" : 0.0
                }
              },
              "fd" : {
                "limit" : {
                  "hard" : 4096,
                  "soft" : 1024
                },
                "open" : 7
              },
              "memory" : {
                "rss" : {
                  "bytes" : 3543040,
                  "pct" : 0.001
                },
                "share" : 3543040,
                "size" : 217858048
              },
              "name" : "gvfs-gphoto2-vo",
              "pgid" : 2094,
              "pid" : 2491,
              "ppid" : 2028,
              "state" : "sleeping",
              "username" : "damien"
            }
          },
          "type" : "metricsets"
        }
      },
      ...
    ]
  }
}

Thank you,
Damien

Stacey_Gammon · January 3, 2017, 5:12pm

Well your data definitely exists in both situations. Changing the time range didn't help?

What if you go to Visualize and create a Metric visualization so it simply shows you the count of hits. Maybe sure you time range is large enough. See if the count is greater than 0. If it's not, inspect the Request in the spy panel (bottom left ^ icon). Then we can see if there is something strange about the query Kibana is sending to Elasticsearch and why it is not grabbing results.

The only other thing I can think of is user permissions. Are you logged in as a super user when trying to view the data?

saiterio · January 3, 2017, 5:27pm

No, the time range do not solve the issue.

For the account, I'm using the 'kibana' built-in account. For metricbeat, I already restarted it several times. I tried also to customise index (metrics-%{+yyyy.MM.dd}), and the issue is the same on kibana (it works on elastic, I can see the index newly created). I tried to change kibana configuration with wrong user/password, in this case, I cannot access to the kibana page.

The request body in visualization:

{
"size": 0,
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "*",
"analyze_wildcard": true
}
},
{
"range": {
"@timestamp": {
"gte": 1325610932892,
"lte": 1483463732894,
"format": "epoch_millis"
}
}
}
],
"must_not":
}
},
"_source": {
"excludes":
},
"aggs": {}
}

The result:

{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits":
},
"status": 200
}

Stacey_Gammon · January 3, 2017, 6:46pm

Okay, so it looks like the first query you sent to elasticsearch is searching more indexes than the visualization. Notice that the query you ran in dev tools had returned 15 shards

 "_shards" : {
    "total" : 15,
    "successful" : 15,
    "failed" : 0
  }

While the visualization returned only 1:

"_shards": {
"total": 1,
"successful": 1,
"failed": 0
}

When I had you run the query in devtools, I used an index pattern of metric*. What is the index pattern you are using when creating the visualization? Can you create an index in Kibana with the pattern of metric* and see if it returns your data?

Based on the returned hits in the first query you ran, it looks like the data is at an index named metricbeat so a pattern of metricbeat-* wouldn't match because of the added -.

Can you once more post the outcome of GET _cat/indices?

and can you post the outcome of running the following in devtools?

POST metricbeat-*/_search
{
  "size": 0
}

and

POST metricbeat/_search
{
  "size": 0
}

saiterio · January 4, 2017, 7:57am

Hi Stacey,

Thank you again for your answer and your time. I can only create index metric* without x-pack (via kibana), because with x-pack I get "index not found"...

Indices with x-pack

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   metricbeat-2017.01.03           VM5MPKGpS9uw6aEqG4XDxA   5   1        783            0    680.6kb        680.6kb
yellow open   metricbeat                      zj5RSadDQNKASB4DDoJV7w   5   1       2554            0      1.4mb          1.4mb
yellow open   metricbeat-2016.12.21           BHAEMqNXTK6XtrMCE5Fnwg   5   1       3728            0      2.1mb          2.1mb
yellow open   .monitoring-data-2              7ty7deKMTq2AiKaOhIUKeg   1   1          3            0       14kb           14kb
yellow open   metrics-2017.01.03              IjUCZdm3QKiUDgAlSbQ67Q   5   1       4022            0        2mb            2mb
yellow open   .monitoring-es-2-2016.12.21     jHOmvPD0SEOWcIyvvvrbKg   1   1      31437           38     13.8mb         13.8mb
yellow open   .monitoring-kibana-2-2016.12.21 L8buvVGnReCQJZS8jEqc-Q   1   1       5275            0      1.1mb          1.1mb
yellow open   .monitoring-kibana-2-2017.01.04 IivEQSjfTvWuBX2q1mhMbw   1   1        208            0    255.7kb        255.7kb
yellow open   .monitoring-es-2-2017.01.03     yiOpwRWrQlmUHrLSisZ5sQ   1   1      44820          477     19.9mb         19.9mb
yellow open   .monitoring-es-2-2017.01.04     ELzU34_ZRYysw3stDy4R8Q   1   1       4223          171      2.9mb          2.9mb
yellow open   .monitoring-kibana-2-2017.01.03 PH5ltRrbQVG_Jb24MftqNA   1   1       5897            0      1.3mb          1.3mb
yellow open   .kibana                         MHaGnUimQfOxYB0A_iaR-A   1   1         96           50      279kb          279kb
green  open   .security                       y3gwQIzxQ2q5dS7JXpXq9w   1   0          5            0     16.7kb         16.7kb

Witout x-pack:

metricbeat-*

{
"took" : 28,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 10,
"failed" : 0
},
"hits" : {
"total" : 4511,
"max_score" : 1.0,
"hits" : [
{
"_index" : "metricbeat-2016.12.21",
"_type" : "metricsets",
"id" : "AVkgaOv8chSzX1pS5Y-",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2016-12-21T08:04:22.720Z",
...
},
...
]
}
}

metric*

{
"took" : 7,
"timed_out" : false,
"_shards" : {
"total" : 20,
"successful" : 20,
"failed" : 0
},
"hits" : {
"total" : 11087,
"max_score" : 1.0,
"hits" : [
{
"_index" : "metricbeat",
"_type" : "metricsets",
"_id" : "AVllBkWgmNy-OguxyKZv",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2017-01-03T15:17:16.268Z",
...
},
"type" : "metricsets"
}
},
...
]
}
}

with x-pack:

metribeat-*

{
"took" : 39,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 10,
"failed" : 0
},
"hits" : {
"total" : 4511,
"max_score" : 1.0,
"hits" : [
{
"_index" : "metricbeat-2016.12.21",
"_type" : "metricsets",
"id" : "AVkgaOv8chSzX1pS5Y-",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2016-12-21T08:04:22.720Z",
...
},
...
]
}
}

metric*

{
"took" : 23,
"timed_out" : false,
"_shards" : {
"total" : 20,
"successful" : 20,
"failed" : 0
},
"hits" : {
"total" : 11087,
"max_score" : 1.0,
"hits" : [
{
"_index" : "metricbeat",
"_type" : "metricsets",
"_id" : "AVllBkWgmNy-OguxyKZv",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2017-01-03T15:17:16.268Z",
...
},
...
]
}
}

Regards,
Damien

Stacey_Gammon · January 4, 2017, 3:17pm

Okay so I think I just successfully repro'ed the issue and I believe this is working as expected. The default kibana user does not have permissions to create an index template.

I usually log in with the default elastic user (it's a built in super user and has a default password of changeme) so I didn't notice the issue. As soon as I logged in as the kibana user, I noticed I was not able to create an index pattern.

Still, you should be able to view and use the indexes your super user created as well, as those you created prior to x-pack being installed.

Can you log in as a super user with x-pack installed and create the metric* index pattern?

saiterio · January 19, 2017, 11:26am

Hi Stacey,

Sorry for the delay in getting back to you, but I had to work on other subjects.
Thank you for your answer, it solves my issue, I didn't understand that default kibana user was not consider as super user...

Have a good day,
Damien

system · February 16, 2017, 11:26am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.