Dashboard error: Could not locate that index-pattern although it exists


(andig) #1

Running ES 6.0 with Kibana and Metricbeat of the same version I'm getting

Could not locate that index-pattern (id: metricbeat-*), click here to re-create it

in the Metricbeat Host dashboard. This only happens for the Number of processes visualization, the rest shows up fine. Editing this vis I can see this message and search:

There is a problem with this saved object
The index pattern associated with this object no longer exists.
If you know what this error means, go ahead and fix it — otherwise click the delete button above.


{
"index": "metricbeat-*",
"query": {
    "query_string": {
    "query": "*",
    "analyze_wildcard": true
    }
},
"filter": []
}

Running the query through Dev Tools I get:

{
"error": {
    "root_cause": [
    {
        "type": "parsing_exception",
        "reason": "Unknown key for a VALUE_STRING in [index].",
        "line": 2,
        "col": 12
    }
    ],
    "type": "parsing_exception",
    "reason": "Unknown key for a VALUE_STRING in [index].",
    "line": 2,
    "col": 12
},
"status": 400
}

Which is odd since the metricbeat-* indices exist.

What could be wrong here?


Cannot get dashboard/visualizations to load
Could not locate that index-pattern (id: metricbeat-*),
(Chris Roberson) #2

Hi @andig,

That is strange. I'm guessing it happens every time you try and load the dashboard? And you see the same error if you try and open the individual visualization?

Let's verify a couple things.

Can you give me the output of these calls:

POST .kibana/_search
{
  "size": 1, 
  "query": {
    "bool": {
      "filter": [{
        "term": {
          "type": "visualization"
        }
      }],
      "must": [
        {
          "match": {
            "visualization.title": "Number of processes [Metricbeat System]"
          }
        }
      ]
    }
  }
}

and

POST .kibana/_search
{
  "_source": {
    "excludes": ["index-pattern.fields","index-pattern.fieldFormatMap"]
  },
  "query": {
    "bool": {
      "filter": [{
        "term": {
          "type": "index-pattern"
        }
      }],
      "must": [
        {
          "match": {
            "index-pattern.title": "metricbeat-*"
          }
        }
      ]
    }
  }
}

(andig) #3

Great, here are the responses- lgtm?

{
"took": 3,
"timed_out": false,
"_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
},
"hits": {
    "total": 104,
    "max_score": 12.165884,
    "hits": [
    {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "visualization:590a60f0-5d87-11e7-8884-1bb4c3b890e4",
        "_score": 12.165884,
        "_source": {
        "type": "visualization",
        "visualization": {
            "description": "",
            "kibanaSavedObjectMeta": {
            "searchSourceJSON": """
{
"index": "metricbeat-*",
"query": {
    "query_string": {
    "query": "*",
    "analyze_wildcard": true
    }
},
"filter": []
}
"""
            },
            "title": "Number of processes [Metricbeat System]",
            "uiStateJSON": """
{
"vis": {
    "defaultColors": {
    "0 - 100": "rgb(0,104,55)"
    }
}
}
""",
            "version": 1,
            "visState": """
{
"title": "Number of processes",
"type": "metric",
"params": {
    "addLegend": false,
    "addTooltip": true,
    "gauge": {
    "autoExtend": false,
    "backStyle": "Full",
    "colorSchema": "Green to Red",
    "colorsRange": [
        {
        "from": 0,
        "to": 100
        }
    ],
    "gaugeColorMode": "None",
    "gaugeStyle": "Full",
    "gaugeType": "Metric",
    "invertColors": false,
    "labels": {
        "color": "black",
        "show": true
    },
    "orientation": "vertical",
    "percentageMode": false,
    "scale": {
        "color": "#333",
        "labels": false,
        "show": false,
        "width": 2
    },
    "style": {
        "bgColor": false,
        "bgFill": "#000",
        "fontSize": 60,
        "labelColor": false,
        "subText": ""
    },
    "type": "simple",
    "useRange": false,
    "verticalSplit": false
    },
    "type": "gauge"
},
"aggs": [
    {
    "id": "1",
    "enabled": true,
    "type": "cardinality",
    "schema": "metric",
    "params": {
        "field": "system.process.pid",
        "customLabel": "Processes"
    }
    }
],
"listeners": {}
}
"""
        }
        }
    }
    ]
}
}

and

{
"took": 29,
"timed_out": false,
"_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
},
"hits": {
    "total": 1,
    "max_score": 0.6931472,
    "hits": [
    {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "index-pattern:31052260-e5b9-11e7-ad24-8185b16808d1",
        "_score": 0.6931472,
        "_source": {
        "index-pattern": {
            "timeFieldName": "@timestamp",
            "title": "metricbeat-*"
        },
        "type": "index-pattern"
        }
    }
    ]
}
}

(Chris Roberson) #4

Interesting.

Try loading the visualization directly. I'm assuming one of the network calls will error out. Can you find that one and post the request and response?

Url: /app/kibana#/visualize/edit/590a60f0-5d87-11e7-8884-1bb4c3b890e4


(andig) #5

Try loading the visualization directly.

Done. Same error message about the index-pattern.

I'm assuming one of the network calls will error out. Can you find that one and post the request and response?

Note quite sure how to do this. No timeouts. Apart from fonts, images and js files, I'm getting 3 XHRs that all return 200 and a bit of binary data:

curl 'http://nas.fritz.box:5601/api/saved_objects/bulk_get' \
-XPOST \
-H 'Referer: http://nas.fritz.box:5601/app/kibana' \
-H 'Content-Type: application/json;charset=UTF-8' \
-H 'Origin: http://nas.fritz.box:5601' \
-H 'Host: nas.fritz.box:5601' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Connection: keep-alive' \
-H 'Content-Length: 70' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7' \
-H 'Cookie: id=uewzmfxeNGsQk1790PCN029305; stay_login=1; splunkweb_csrf_token_8000=8370888078406587394' \
-H 'Accept-Language: en-us' \
-H 'kbn-version: 6.0.0' \
--data-binary '[{"id":"590a60f0-5d87-11e7-8884-1bb4c3b890e4","type":"visualization"}]'

curl 'http://nas.fritz.box:5601/api/saved_objects/bulk_get' \
-XPOST \
-H 'Referer: http://nas.fritz.box:5601/app/kibana' \
-H 'Content-Type: application/json;charset=UTF-8' \
-H 'Origin: http://nas.fritz.box:5601' \
-H 'Host: nas.fritz.box:5601' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Connection: keep-alive' \
-H 'Content-Length: 46' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7' \
-H 'Cookie: id=uewzmfxeNGsQk1790PCN029305; stay_login=1; splunkweb_csrf_token_8000=8370888078406587394' \
-H 'Accept-Language: en-us' \
-H 'kbn-version: 6.0.0' \
--data-binary '[{"id":"metricbeat-*","type":"index-pattern"}]'

curl 'http://nas.fritz.box:5601/api/saved_objects/bulk_get' \
-XPOST \
-H 'Referer: http://nas.fritz.box:5601/app/kibana' \
-H 'Content-Type: application/json;charset=UTF-8' \
-H 'Origin: http://nas.fritz.box:5601' \
-H 'Host: nas.fritz.box:5601' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Connection: keep-alive' \
-H 'Content-Length: 70' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7' \
-H 'Cookie: id=uewzmfxeNGsQk1790PCN029305; stay_login=1; splunkweb_csrf_token_8000=8370888078406587394' \
-H 'Accept-Language: en-us' \
-H 'kbn-version: 6.0.0' \
--data-binary '[{"id":"590a60f0-5d87-11e7-8884-1bb4c3b890e4","type":"visualization"}]'

(andig) #6

If it's of any help I could PM you access to Kibana. It's not a production system running inside Docker, so I'm pretty relaxed about access.


(andig) #7

Another attempt, without accepting gzip:

❯       curl 'http://nas.fritz.box:5601/api/saved_objects/bulk_get' \
>       -XPOST \
>       -H 'Referer: http://nas.fritz.box:5601/app/kibana' \
>       -H 'Content-Type: application/json;charset=UTF-8' \
>       -H 'Origin: http://nas.fritz.box:5601' \
>       -H 'Host: nas.fritz.box:5601' \
>       -H 'Accept: application/json, text/plain, */*' \
>       -H 'Connection: keep-alive' \
>       -H 'Content-Length: 70' \
>       -H 'kbn-version: 6.0.0' \
>       --data-binary '[{"id":"590a60f0-5d87-11e7-8884-1bb4c3b890e4","type":"visualization"}]'

{"saved_objects":[{"id":"590a60f0-5d87-11e7-8884-1bb4c3b890e4","type":"visualization","version":3,"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\n  \"index\": \"metricbeat-*\",\n  \"query\": {\n   \"query_string\": {\n      \"query\": \"*\",\n      \"analyze_wildcard\": true\n    }\n  },\n  \"filter\": []\n}"},"title":"Number of processes [Metricbeat System]","uiStateJSON":"{\n  \"vis\": {\n    \"defaultColors\": {\n      \"0 - 100\": \"rgb(0,104,55)\"\n    }\n  }\n}","version":1,"visState":"{\n  \"title\": \"Number of processes\",\n  \"type\": \"metric\",\n  \"params\": {\n    \"addLegend\": false,\n    \"addTooltip\": true,\n    \"gauge\": {\n      \"autoExtend\": false,\n      \"backStyle\": \"Full\",\n      \"colorSchema\": \"Green to Red\",\n      \"colorsRange\": [\n        {\n          \"from\": 0,\n          \"to\": 100\n        }\n      ],\n      \"gaugeColorMode\": \"None\",\n      \"gaugeStyle\": \"Full\",\n      \"gaugeType\": \"Metric\",\n      \"invertColors\": false,\n      \"labels\": {\n        \"color\": \"black\",\n        \"show\": true\n      },\n      \"orientation\": \"vertical\",\n      \"percentageMode\": false,\n      \"scale\": {\n        \"color\": \"#333\",\n        \"labels\": false,\n        \"show\": false,\n        \"width\": 2\n      },\n      \"style\": {\n        \"bgColor\": false,\n        \"bgFill\": \"#000\",\n        \"fontSize\": 60,\n        \"labelColor\": false,\n        \"subText\": \"\"\n      },\n      \"type\": \"simple\",\n      \"useRange\": false,\n      \"verticalSplit\":false\n    },\n    \"type\": \"gauge\"\n  },\n  \"aggs\": [\n    {\n      \"id\": \"1\",\n      \"enabled\": true,\n      \"type\": \"cardinality\",\n      \"schema\": \"metric\",\n      \"params\": {\n        \"field\": \"system.process.pid\",\n        \"customLabel\": \"Processes\"\n      }\n    }\n  ],\n  \"listeners\": {}\n}"}}]}

❯       curl 'http://nas.fritz.box:5601/api/saved_objects/bulk_get' \
>       -XPOST \
>       -H 'Referer: http://nas.fritz.box:5601/app/kibana' \
>       -H 'Content-Type: application/json;charset=UTF-8' \
>       -H 'Origin: http://nas.fritz.box:5601' \
>       -H 'Host: nas.fritz.box:5601' \
>       -H 'Accept: application/json, text/plain, */*' \
>       -H 'Connection: keep-alive' \
>       -H 'Content-Length: 46' \
>       -H 'kbn-version: 6.0.0' \
>       --data-binary '[{"id":"metricbeat-*","type":"index-pattern"}]'

{"saved_objects":[{"id":"metricbeat-*","type":"index-pattern","error":{"statusCode":404,"message":"Not found"}}]}

I don't get why it doesn't see the index-pattern that the other visualizations are happy with???

And last but not least there is also:

GET /api/_search?q=*
{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "api",
        "_type": "saved_objects",
        "_id": "bulk_get",
        "_score": 1,
        "_source": {
          "id": "metricbeat-*",
          "type": "index-pattern"
        }
      }
    ]
  }
}

(andig) #8

And there is:

POST .kibana/_search
{
  "_source": {
    "excludes": ["index-pattern.fields","index-pattern.fieldFormatMap"]
  },
  "query": {
    "bool": {
      "filter": [{
        "term": {
          "type": "index-pattern"
        }
      }]
    }
  }
}

{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 0,
    "hits": [
      {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "index-pattern:31052260-e5b9-11e7-ad24-8185b16808d1",
        "_score": 0,
        "_source": {
          "index-pattern": {
            "timeFieldName": "@timestamp",
            "title": "metricbeat-*"
          },
          "type": "index-pattern"
        }
      },
      {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "index-pattern:filebeat-*",
        "_score": 0,
        "_source": {
          "index-pattern": {
            "timeFieldName": "@timestamp",
            "title": "filebeat-*"
          },
          "type": "index-pattern"
        }
      }
    ]
  }
}

Maybe something with index-pattern id vs index-pattern?


(andig) #9

@chrisronline any more ideas what wrong here? The server side is the stock docker image...


(Chris Roberson) #10

Hi @andig,

If you can PM me access to the Kibana instance, I can check it out more


#11

I have had the same issue. Apparently it was looking for the index pattern ID, not the name. I fixed mine by deleting the old index pattern and adding a new one with the same name and the ID (the ID field is available under advanced options) set to the ID the error returns. In your case that would be metricbeat-*. It is still strange though and I don't believe this is the best solution.


(andig) #12

Hi Chris, nixhoofd,

I'll post Chris' pm here fore reference:

I'll also add the output of the query Chris mentions:

POST .kibana/_search
{
  "_source": {
    "excludes": ["index-pattern.fields","index-pattern.fieldFormatMap"]
  },
  "query": {
    "bool": {
      "filter": {
        "term": {
          "type": "index-pattern"
        }
      }
    }
  }
}

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 0,
    "hits": [
      {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "index-pattern:31052260-e5b9-11e7-ad24-8185b16808d1",
        "_score": 0,
        "_source": {
          "index-pattern": {
            "timeFieldName": "@timestamp",
            "title": "metricbeat-*"
          },
          "type": "index-pattern"
        }
      },
      {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "index-pattern:filebeat-*",
        "_score": 0,
        "_source": {
          "index-pattern": {
            "timeFieldName": "@timestamp",
            "title": "filebeat-*"
          },
          "type": "index-pattern"
        }
      }
    ]
  }
}

Now for the history- this instance is a vanilla docker install, no upgrades or similar. Playing with filebeat and metricbeat it took me a while to understand that I needed to tell kibana about the related indexes. I've created and dropped index patterns a number of times (initially indexes not created, templates not imported etc- beginners problems).

The kibana index pattern creation is still a mystery to me: I can create a pattern plus choose the id but I cannot change the id later.

As for the dashboard: maybe it is a question of how those dashboards are actually created. I assume this is one from *beat side? How do the dashboards reference the indexes then- my pattern title or by id (assuming the latter). If thats the case I still don't get why only a single visualization has the problem...

So, what would be the fix- update the broken index pattern's id? If I do that- won't it break all the other visualizations that are currently working?

Still struggling with understanding how and when visualizations and index patterns are tied together...

Cheers,
Andreas


(Chris Roberson) #13

Can you provide step-by-step instructions for how you got the beats setup? Are you running the setup script for each beat? Is that before or after you create an index pattern within Kibana for the beat? Are you creating any visualizations manually?

Thanks


(system) #14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.