Dashboard error: Could not locate that index-pattern although it exists

[andig]

Running ES 6.0 with Kibana and Metricbeat of the same version I'm getting

Could not locate that index-pattern (id: metricbeat-*), click here to re-create it

in the Metricbeat Host dashboard. This only happens for the Number of processes visualization, the rest shows up fine. Editing this vis I can see this message and search:

There is a problem with this saved object
The index pattern associated with this object no longer exists.
If you know what this error means, go ahead and fix it — otherwise click the delete button above.


{
"index": "metricbeat-*",
"query": {
    "query_string": {
    "query": "*",
    "analyze_wildcard": true
    }
},
"filter": []
}

Running the query through Dev Tools I get:

{
"error": {
    "root_cause": [
    {
        "type": "parsing_exception",
        "reason": "Unknown key for a VALUE_STRING in [index].",
        "line": 2,
        "col": 12
    }
    ],
    "type": "parsing_exception",
    "reason": "Unknown key for a VALUE_STRING in [index].",
    "line": 2,
    "col": 12
},
"status": 400
}

Which is odd since the metricbeat-* indices exist.

What could be wrong here?

[chrisronline (Chris Roberson)]

Hi @andig,

That is strange. I'm guessing it happens every time you try and load the dashboard? And you see the same error if you try and open the individual visualization?

Let's verify a couple things.

Can you give me the output of these calls:

POST .kibana/_search
{
  "size": 1, 
  "query": {
    "bool": {
      "filter": [{
        "term": {
          "type": "visualization"
        }
      }],
      "must": [
        {
          "match": {
            "visualization.title": "Number of processes [Metricbeat System]"
          }
        }
      ]
    }
  }
}

and

POST .kibana/_search
{
  "_source": {
    "excludes": ["index-pattern.fields","index-pattern.fieldFormatMap"]
  },
  "query": {
    "bool": {
      "filter": [{
        "term": {
          "type": "index-pattern"
        }
      }],
      "must": [
        {
          "match": {
            "index-pattern.title": "metricbeat-*"
          }
        }
      ]
    }
  }
}

[andig]

Great, here are the responses- lgtm?

{
"took": 3,
"timed_out": false,
"_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
},
"hits": {
    "total": 104,
    "max_score": 12.165884,
    "hits": [
    {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "visualization:590a60f0-5d87-11e7-8884-1bb4c3b890e4",
        "_score": 12.165884,
        "_source": {
        "type": "visualization",
        "visualization": {
            "description": "",
            "kibanaSavedObjectMeta": {
            "searchSourceJSON": """
{
"index": "metricbeat-*",
"query": {
    "query_string": {
    "query": "*",
    "analyze_wildcard": true
    }
},
"filter": []
}
"""
            },
            "title": "Number of processes [Metricbeat System]",
            "uiStateJSON": """
{
"vis": {
    "defaultColors": {
    "0 - 100": "rgb(0,104,55)"
    }
}
}
""",
            "version": 1,
            "visState": """
{
"title": "Number of processes",
"type": "metric",
"params": {
    "addLegend": false,
    "addTooltip": true,
    "gauge": {
    "autoExtend": false,
    "backStyle": "Full",
    "colorSchema": "Green to Red",
    "colorsRange": [
        {
        "from": 0,
        "to": 100
        }
    ],
    "gaugeColorMode": "None",
    "gaugeStyle": "Full",
    "gaugeType": "Metric",
    "invertColors": false,
    "labels": {
        "color": "black",
        "show": true
    },
    "orientation": "vertical",
    "percentageMode": false,
    "scale": {
        "color": "#333",
        "labels": false,
        "show": false,
        "width": 2
    },
    "style": {
        "bgColor": false,
        "bgFill": "#000",
        "fontSize": 60,
        "labelColor": false,
        "subText": ""
    },
    "type": "simple",
    "useRange": false,
    "verticalSplit": false
    },
    "type": "gauge"
},
"aggs": [
    {
    "id": "1",
    "enabled": true,
    "type": "cardinality",
    "schema": "metric",
    "params": {
        "field": "system.process.pid",
        "customLabel": "Processes"
    }
    }
],
"listeners": {}
}
"""
        }
        }
    }
    ]
}
}

and

{
"took": 29,
"timed_out": false,
"_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
},
"hits": {
    "total": 1,
    "max_score": 0.6931472,
    "hits": [
    {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "index-pattern:31052260-e5b9-11e7-ad24-8185b16808d1",
        "_score": 0.6931472,
        "_source": {
        "index-pattern": {
            "timeFieldName": "@timestamp",
            "title": "metricbeat-*"
        },
        "type": "index-pattern"
        }
    }
    ]
}
}

[chrisronline (Chris Roberson)]

Interesting.

Try loading the visualization directly. I'm assuming one of the network calls will error out. Can you find that one and post the request and response?

Url: /app/kibana#/visualize/edit/590a60f0-5d87-11e7-8884-1bb4c3b890e4

[andig]

Try loading the visualization directly.

Done. Same error message about the index-pattern.

I'm assuming one of the network calls will error out. Can you find that one and post the request and response?

Note quite sure how to do this. No timeouts. Apart from fonts, images and js files, I'm getting 3 XHRs that all return 200 and a bit of binary data:

curl 'http://nas.fritz.box:5601/api/saved_objects/bulk_get' \
-XPOST \
-H 'Referer: http://nas.fritz.box:5601/app/kibana' \
-H 'Content-Type: application/json;charset=UTF-8' \
-H 'Origin: http://nas.fritz.box:5601' \
-H 'Host: nas.fritz.box:5601' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Connection: keep-alive' \
-H 'Content-Length: 70' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7' \
-H 'Cookie: id=uewzmfxeNGsQk1790PCN029305; stay_login=1; splunkweb_csrf_token_8000=8370888078406587394' \
-H 'Accept-Language: en-us' \
-H 'kbn-version: 6.0.0' \
--data-binary '[{"id":"590a60f0-5d87-11e7-8884-1bb4c3b890e4","type":"visualization"}]'

curl 'http://nas.fritz.box:5601/api/saved_objects/bulk_get' \
-XPOST \
-H 'Referer: http://nas.fritz.box:5601/app/kibana' \
-H 'Content-Type: application/json;charset=UTF-8' \
-H 'Origin: http://nas.fritz.box:5601' \
-H 'Host: nas.fritz.box:5601' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Connection: keep-alive' \
-H 'Content-Length: 46' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7' \
-H 'Cookie: id=uewzmfxeNGsQk1790PCN029305; stay_login=1; splunkweb_csrf_token_8000=8370888078406587394' \
-H 'Accept-Language: en-us' \
-H 'kbn-version: 6.0.0' \
--data-binary '[{"id":"metricbeat-*","type":"index-pattern"}]'

curl 'http://nas.fritz.box:5601/api/saved_objects/bulk_get' \
-XPOST \
-H 'Referer: http://nas.fritz.box:5601/app/kibana' \
-H 'Content-Type: application/json;charset=UTF-8' \
-H 'Origin: http://nas.fritz.box:5601' \
-H 'Host: nas.fritz.box:5601' \
-H 'Accept: application/json, text/plain, */*' \
-H 'Connection: keep-alive' \
-H 'Content-Length: 70' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7' \
-H 'Cookie: id=uewzmfxeNGsQk1790PCN029305; stay_login=1; splunkweb_csrf_token_8000=8370888078406587394' \
-H 'Accept-Language: en-us' \
-H 'kbn-version: 6.0.0' \
--data-binary '[{"id":"590a60f0-5d87-11e7-8884-1bb4c3b890e4","type":"visualization"}]'

[andig]

If it's of any help I could PM you access to Kibana. It's not a production system running inside Docker, so I'm pretty relaxed about access.

[andig]

Another attempt, without accepting gzip:

❯       curl 'http://nas.fritz.box:5601/api/saved_objects/bulk_get' \
>       -XPOST \
>       -H 'Referer: http://nas.fritz.box:5601/app/kibana' \
>       -H 'Content-Type: application/json;charset=UTF-8' \
>       -H 'Origin: http://nas.fritz.box:5601' \
>       -H 'Host: nas.fritz.box:5601' \
>       -H 'Accept: application/json, text/plain, */*' \
>       -H 'Connection: keep-alive' \
>       -H 'Content-Length: 70' \
>       -H 'kbn-version: 6.0.0' \
>       --data-binary '[{"id":"590a60f0-5d87-11e7-8884-1bb4c3b890e4","type":"visualization"}]'

{"saved_objects":[{"id":"590a60f0-5d87-11e7-8884-1bb4c3b890e4","type":"visualization","version":3,"attributes":{"description":"","kibanaSavedObjectMeta":{"searchSourceJSON":"{\n  \"index\": \"metricbeat-*\",\n  \"query\": {\n   \"query_string\": {\n      \"query\": \"*\",\n      \"analyze_wildcard\": true\n    }\n  },\n  \"filter\": []\n}"},"title":"Number of processes [Metricbeat System]","uiStateJSON":"{\n  \"vis\": {\n    \"defaultColors\": {\n      \"0 - 100\": \"rgb(0,104,55)\"\n    }\n  }\n}","version":1,"visState":"{\n  \"title\": \"Number of processes\",\n  \"type\": \"metric\",\n  \"params\": {\n    \"addLegend\": false,\n    \"addTooltip\": true,\n    \"gauge\": {\n      \"autoExtend\": false,\n      \"backStyle\": \"Full\",\n      \"colorSchema\": \"Green to Red\",\n      \"colorsRange\": [\n        {\n          \"from\": 0,\n          \"to\": 100\n        }\n      ],\n      \"gaugeColorMode\": \"None\",\n      \"gaugeStyle\": \"Full\",\n      \"gaugeType\": \"Metric\",\n      \"invertColors\": false,\n      \"labels\": {\n        \"color\": \"black\",\n        \"show\": true\n      },\n      \"orientation\": \"vertical\",\n      \"percentageMode\": false,\n      \"scale\": {\n        \"color\": \"#333\",\n        \"labels\": false,\n        \"show\": false,\n        \"width\": 2\n      },\n      \"style\": {\n        \"bgColor\": false,\n        \"bgFill\": \"#000\",\n        \"fontSize\": 60,\n        \"labelColor\": false,\n        \"subText\": \"\"\n      },\n      \"type\": \"simple\",\n      \"useRange\": false,\n      \"verticalSplit\":false\n    },\n    \"type\": \"gauge\"\n  },\n  \"aggs\": [\n    {\n      \"id\": \"1\",\n      \"enabled\": true,\n      \"type\": \"cardinality\",\n      \"schema\": \"metric\",\n      \"params\": {\n        \"field\": \"system.process.pid\",\n        \"customLabel\": \"Processes\"\n      }\n    }\n  ],\n  \"listeners\": {}\n}"}}]}

❯       curl 'http://nas.fritz.box:5601/api/saved_objects/bulk_get' \
>       -XPOST \
>       -H 'Referer: http://nas.fritz.box:5601/app/kibana' \
>       -H 'Content-Type: application/json;charset=UTF-8' \
>       -H 'Origin: http://nas.fritz.box:5601' \
>       -H 'Host: nas.fritz.box:5601' \
>       -H 'Accept: application/json, text/plain, */*' \
>       -H 'Connection: keep-alive' \
>       -H 'Content-Length: 46' \
>       -H 'kbn-version: 6.0.0' \
>       --data-binary '[{"id":"metricbeat-*","type":"index-pattern"}]'

{"saved_objects":[{"id":"metricbeat-*","type":"index-pattern","error":{"statusCode":404,"message":"Not found"}}]}

I don't get why it doesn't see the index-pattern that the other visualizations are happy with???

And last but not least there is also:

GET /api/_search?q=*
{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "api",
        "_type": "saved_objects",
        "_id": "bulk_get",
        "_score": 1,
        "_source": {
          "id": "metricbeat-*",
          "type": "index-pattern"
        }
      }
    ]
  }
}

[andig]

And there is:

POST .kibana/_search
{
  "_source": {
    "excludes": ["index-pattern.fields","index-pattern.fieldFormatMap"]
  },
  "query": {
    "bool": {
      "filter": [{
        "term": {
          "type": "index-pattern"
        }
      }]
    }
  }
}

{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 0,
    "hits": [
      {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "index-pattern:31052260-e5b9-11e7-ad24-8185b16808d1",
        "_score": 0,
        "_source": {
          "index-pattern": {
            "timeFieldName": "@timestamp",
            "title": "metricbeat-*"
          },
          "type": "index-pattern"
        }
      },
      {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "index-pattern:filebeat-*",
        "_score": 0,
        "_source": {
          "index-pattern": {
            "timeFieldName": "@timestamp",
            "title": "filebeat-*"
          },
          "type": "index-pattern"
        }
      }
    ]
  }
}

Maybe something with index-pattern id vs index-pattern?

[andig]

@chrisronline any more ideas what wrong here? The server side is the stock docker image...

[chrisronline (Chris Roberson)]

Hi @andig,

If you can PM me access to the Kibana instance, I can check it out more

[nixhoofd]

I have had the same issue. Apparently it was looking for the index pattern ID, not the name. I fixed mine by deleting the old index pattern and adding a new one with the same name and the ID (the ID field is available under advanced options) set to the ID the error returns. In your case that would be metricbeat-*. It is still strange though and I don't believe this is the best solution.

[andig]

Hi Chris, nixhoofd,

I'll post Chris' pm here fore reference:

@chrisronline

I think I know what's wrong, but I don't really know how it ended up like that.

If you run this query, you'll notice that the _id field is a bit different for metricbeat-* and filebeat-*:

POST .kibana/_search
{
  "query": {
    "bool": {
      "filter": {
        "term": {
          "type": "index-pattern"
        }
      }
    }
  }
}

The metricbeat-* id (which contains a random guid) is the new way we're creating these so that's correct. I don't know why the packetbeat-* doesn't contain a guid.

Now, some of your saved searches (I'm guessing mostly for metricbeat) contain references to metricbeat-* as the index pattern id, instead of the random guid.

See: GET .kibana/doc/search:Metricbeat-Docker

But the kicker is that some of your saved searches also have random GUIDs:

See: GET .kibana/doc/search:67e88e60-0005-11e7-aaf1-b342e4b94bb0

Both of those point to just metricbeat-* and if you change that to use the actual guid for the metricbeat-* index pattern, it will start working properly.

With all that said and done, I don't really know what's going on. It seems the saved objects contains a mismatch of two different naming schemes. Maybe you mentioned before, but was there an upgrade involved with this issue? Or maybe provide me with a little more information about the history of this instance?

I'll also add the output of the query Chris mentions:

POST .kibana/_search
{
  "_source": {
    "excludes": ["index-pattern.fields","index-pattern.fieldFormatMap"]
  },
  "query": {
    "bool": {
      "filter": {
        "term": {
          "type": "index-pattern"
        }
      }
    }
  }
}

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 0,
    "hits": [
      {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "index-pattern:31052260-e5b9-11e7-ad24-8185b16808d1",
        "_score": 0,
        "_source": {
          "index-pattern": {
            "timeFieldName": "@timestamp",
            "title": "metricbeat-*"
          },
          "type": "index-pattern"
        }
      },
      {
        "_index": ".kibana",
        "_type": "doc",
        "_id": "index-pattern:filebeat-*",
        "_score": 0,
        "_source": {
          "index-pattern": {
            "timeFieldName": "@timestamp",
            "title": "filebeat-*"
          },
          "type": "index-pattern"
        }
      }
    ]
  }
}

Now for the history- this instance is a vanilla docker install, no upgrades or similar. Playing with filebeat and metricbeat it took me a while to understand that I needed to tell kibana about the related indexes. I've created and dropped index patterns a number of times (initially indexes not created, templates not imported etc- beginners problems).

The kibana index pattern creation is still a mystery to me: I can create a pattern plus choose the id but I cannot change the id later.

As for the dashboard: maybe it is a question of how those dashboards are actually created. I assume this is one from *beat side? How do the dashboards reference the indexes then- my pattern title or by id (assuming the latter). If thats the case I still don't get why only a single visualization has the problem...

So, what would be the fix- update the broken index pattern's id? If I do that- won't it break all the other visualizations that are currently working?

Still struggling with understanding how and when visualizations and index patterns are tied together...

Cheers,
Andreas

[chrisronline (Chris Roberson)]

Can you provide step-by-step instructions for how you got the beats setup? Are you running the setup script for each beat? Is that before or after you create an index pattern within Kibana for the beat? Are you creating any visualizations manually?

Thanks

[ChrsMark (Chris Mark)]

Just to add a note to this, you need to make sure that during setup the Beat you are running can access Kibana. Beats, let's use metricbeat for example, will have to POST to Kibana API the index-pattern with _id= metricbeat-* so as the pre-built visualizations and dashboards to be valid.