Fixed -- "Elasticsearch plugin is red" after 6.0.0 upgrade

Elastic Stack Kibana
1

Hi,

I upgraded succesfully my cluster from 5.6.4 to 6.0.0.
I upgraded logstash & kibana to 6.0.0.
Platform debian 8.9, installation from packages (https://artifacts.elastic.co/downloads/elasticsearch/*)

ES is ok (local curl requests, cerebro)
When I launch kibana, I have the famous ""Elasticsearch plugin is red" problem, but unable to solve it...

x-pack security is disabled (kibana, ES)
kibana index was deleted, kibana restarted -> new .kibana index exists and is correctly templated

I tried without x-pack plugin, no change.

22
screenshot 2017-11-29 à 14.53.22.png2310×1352 330 KB

Nothing to do for hours, unable to get kibana functional again :o/

Someone for an idea please?

Thx in advance for your help
Antony

2

kibana config ->

server.host: "0.0.0.0"
server.name: "KIBANA MT"
elasticsearch.url: "http://localhost:9200"
xpack.security.enabled: false

3

systemctl restart kibana.service ->

Nov 29 14:47:00 elasticsearch-cluster1-vm1 systemd[1]: Stopping Kibana...
Nov 29 14:47:00 elasticsearch-cluster1-vm1 systemd[1]: Starting Kibana...
Nov 29 14:47:00 elasticsearch-cluster1-vm1 systemd[1]: Started Kibana.
Nov 29 14:47:05 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:05Z","tags":["status","plugin:kibana@6.0.0","info"],"pid":21346,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Nov 29 14:47:05 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:05Z","tags":["status","plugin:elasticsearch@6.0.0","info"],"pid":21346,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
Nov 29 14:47:05 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:05Z","tags":["status","plugin:xpack_main@6.0.0","info"],"pid":21346,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
Nov 29 14:47:05 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:05Z","tags":["status","plugin:graph@6.0.0","info"],"pid":21346,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
Nov 29 14:47:05 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:05Z","tags":["status","plugin:monitoring@6.0.0","info"],"pid":21346,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Nov 29 14:47:06 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:06Z","tags":["reporting","warning"],"pid":21346,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
Nov 29 14:47:06 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:06Z","tags":["status","plugin:reporting@6.0.0","info"],"pid":21346,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
Nov 29 14:47:06 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:06Z","tags":["status","plugin:searchprofiler@6.0.0","info"],"pid":21346,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
Nov 29 14:47:06 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:06Z","tags":["status","plugin:xpack_main@6.0.0","error"],"pid":21346,"state":"red","message":"Status changed from yellow to red - Your Kibana index is out of date, reset it or use the X-Pack upgrade assistant.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
Nov 29 14:47:06 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:06Z","tags":["status","plugin:graph@6.0.0","error"],"pid":21346,"state":"red","message":"Status changed from yellow to red - Your Kibana index is out of date, reset it or use the X-Pack upgrade assistant.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
Nov 29 14:47:06 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:06Z","tags":["status","plugin:reporting@6.0.0","error"],"pid":21346,"state":"red","message":"Status changed from yellow to red - Your Kibana index is out of date, reset it or use the X-Pack upgrade assistant.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
Nov 29 14:47:06 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:06Z","tags":["status","plugin:searchprofiler@6.0.0","error"],"pid":21346,"state":"red","message":"Status changed from yellow to red - Your Kibana index is out of date, reset it or use the X-Pack upgrade assistant.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
Nov 29 14:47:06 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:06Z","tags":["status","plugin:elasticsearch@6.0.0","error"],"pid":21346,"state":"red","message":"Status changed from yellow to red - Your Kibana index is out of date, reset it or use the X-Pack upgrade assistant.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
Nov 29 14:47:06 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:06Z","tags":["status","plugin:ml@6.0.0","error"],"pid":21346,"state":"red","message":"Status changed from uninitialized to red - Your Kibana index is out of date, reset it or use the X-Pack upgrade assistant.","prevState":"uninitialized","prevMsg":"uninitialized"}
...
Nov 29 14:47:07 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:07Z","tags":["status","plugin:console@6.0.0","info"],"pid":21346,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Nov 29 14:47:07 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:07Z","tags":["status","plugin:metrics@6.0.0","info"],"pid":21346,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Nov 29 14:47:07 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:07Z","tags":["status","plugin:timelion@6.0.0","info"],"pid":21346,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
Nov 29 14:47:07 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:07Z","tags":["listening","info"],"pid":21346,"message":"Server running at http://0.0.0.0:5601"}
Nov 29 14:47:07 elasticsearch-cluster1-vm1 kibana[21346]: {"type":"log","@timestamp":"2017-11-29T13:47:07Z","tags":["status","ui settings","error"],"pid":21346,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}

4

Upgrading from 5 to 6 requires a reindex, either through the upgrade assistant or manually. https://www.elastic.co/guide/en/kibana/current/migrating-6.0-index.html has instructions.

With that said, a new .kibana index should be functional so that sounds like a problem. Can you share your index's mappings?

5

Thanks for your feedback.

I used these commands as a first try, without success.
I tried with or without pre-templating, same results.
Then I scratched .kibana and .kibana-6 index.

I'm limited to 7000 chars in my answer -> I'll try to post it in 2 parts ;o) :

6 
{
  "settings" : {
    "number_of_shards" : 1,
    "index.mapper.dynamic": false
  },
  "mappings" : {
    "doc": {
      "properties": {
        "type": {
          "type": "keyword"
        },
        "updated_at": {
          "type": "date"
        },
        "config": {
          "properties": {
            "buildNum": {
              "type": "keyword"
            }
          }
        },
        "index-pattern": {
          "properties": {
            "fieldFormatMap": {
              "type": "text"
            },
            "fields": {
              "type": "text"
            },
            "intervalName": {
              "type": "keyword"
            },
            "notExpandable": {
              "type": "boolean"
            },
            "sourceFilters": {
              "type": "text"
            },
            "timeFieldName": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            }
          }
        },
        "visualization": {
          "properties": {
            "description": {
              "type": "text"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "savedSearchId": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "uiStateJSON": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            },
            "visState": {
              "type": "text"
            }
          }
        },
        "search": {
          "properties": {
            "columns": {
              "type": "keyword"
            },
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "sort": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        },
        "dashboard": {
          "properties": {
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "optionsJSON": {
              "type": "text"
            },
            "panelsJSON": {
              "type": "text"
            },
            "refreshInterval": {
              "properties": {
                "display": {
                  "type": "keyword"
                },
                "pause": {
                  "type": "boolean"
                },
                "section": {
                  "type": "integer"
                },
                "value": {
                  "type": "integer"
                }
              }
            },
            "timeFrom": {
              "type": "keyword"
            },
            "timeRestore": {
              "type": "boolean"
            },
            "timeTo": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "uiStateJSON": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        },
        "url": {
          "properties": {
            "accessCount": {
              "type": "long"
            },
            "accessDate": {
              "type": "date"
            },
            "createDate": {
              "type": "date"
            },
            "url": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 2048
                }
              }
            }
          }
        },
7 
"server": {
  "properties": {
    "uuid": {
      "type": "keyword"
    }
  }
},
"timelion-sheet": {
  "properties": {
    "description": {
      "type": "text"
    },
    "hits": {
      "type": "integer"
    },
    "kibanaSavedObjectMeta": {
      "properties": {
        "searchSourceJSON": {
          "type": "text"
        }
      }
    },
    "timelion_chart_height": {
      "type": "integer"
    },
    "timelion_columns": {
      "type": "integer"
    },
    "timelion_interval": {
      "type": "keyword"
    },
    "timelion_other_interval": {
      "type": "keyword"
    },
    "timelion_rows": {
      "type": "integer"
    },
    "timelion_sheet": {
      "type": "text"
    },
    "title": {
      "type": "text"
    },
    "version": {
      "type": "integer"
    }
  }
},
"graph-workspace": {
  "properties": {
    "description": {
      "type": "text"
    },
    "kibanaSavedObjectMeta": {
      "properties": {
        "searchSourceJSON": {
          "type": "text"
        }
      }
    },
    "numLinks": {
      "type": "integer"
    },
    "numVertices": {
      "type": "integer"
    },
    "title": {
      "type": "text"
    },
    "version": {
      "type": "integer"
    },
    "wsState": {
      "type": "text"
    }
  }
}

}
}
}
}

8

Fixed.

We had an oooooold index template pattern with the following definition:
"index_patterns": [
"*"
],
...

This pattern redefined "type" field with a different type than kibana, which caused conflict and error.

-> we updated our template
->we deleted concerned ES index template pattern(s)
That fixed the problem.

Thx Jon ;o)

Antony

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.