Version conflict, document already exists: index=".kibana"

[jwr_cr (jwr)]

I have a freshly installed elasticsearch cluster that hasn't yet ingested any data. I then set up a freshly installed kibana server and pointed my elasticsearch.url to the load balancer in front of elasticsearch. Upon logging in to the UI of kibana I'm seeing:

Anybody know what I did wrong here? Thanks.

Relevant versions:
kibana-5.2.2-1.x86_64
elasticsearch-5.2.2-1.noarch

Here's the full entry from kibana.stdout:

{
  "type": "log",
  "@timestamp": "2017-11-01T18:33:08Z",
  "tags": [
    "status",
    "plugin:elasticsearch@5.2.2",
    "error"
  ],
  "pid": 3237,
  "state": "red",
  "message": "Status changed from green to red - [version_conflict_engine_exception] [config][5.2.2]: version conflict, document already exists (current version [1]), with { index_uuid=\"_E5U0bWxSHSmLLKOYVjOCg\" & shard=\"2\" & index=\".kibana\" }",
  "prevState": "green",
  "prevMsg": "Kibana index ready"
}
{
  "type": "log",
  "@timestamp": "2017-11-01T18:33:08Z",
  "tags": [
    "status",
    "ui settings",
    "error"
  ],
  "pid": 3237,
  "state": "red",
  "message": "Status changed from green to red - Elasticsearch plugin is red",
  "prevState": "green",
  "prevMsg": "Ready"
}

[jwr_cr (jwr)]

Tried 5.6.3 too, same error.

[rashmi (kulkarni)]

@jwr_cr

oad balancer in front of elasticsearch.

Hello

Am not too sure what kind of load balancer you are using. I would suggest you to configure kibana to connect directly to elasticsearch to see if that works OK. Then we can debug more if that's successful

Thanks
Rashmi

[jwr_cr (jwr)]

Thanks for your reply. I tried as you suggested and here is what I found.

Starting by tearing down all of Elasticsearch and Kibana and building new stacks:

Using a 1 node ES cluster, I set Kibana's elasticsearch.url to the private IP of that one ES node, bypassing any load balancer. This worked fine.

Tear down and rebuild both stacks.

Using a 3 node ES cluster, I set Kibana's elasticsearch.url to the private IP of one of my three ES nodes, chosen at random, bypassing any load balancer. This also worked fine.

Tear down and rebuild both stacks.

Using a 3 node ES cluster, I set Kibana's elasticsearch.url to the Amazon ELB which sits in front of the 3 node ES cluster. This resulted in an error creating the '.kibana' index.

{
  "type": "log",
  "@timestamp": "2017-11-01T21:40:18Z",
  "tags": [
    "status",
    "plugin:elasticsearch@5.6.3",
    "info"
  ],
  "pid": 2760,
  "state": "yellow",
  "message": "Status changed from yellow to yellow - No existing Kibana index found",
  "prevState": "yellow",
  "prevMsg": "Waiting for Elasticsearch"
}
{
  "type": "log",
  "@timestamp": "2017-11-01T21:40:49Z",
  "tags": [
    "status",
    "plugin:elasticsearch@5.6.3",
    "error"
  ],
  "pid": 2760,
  "state": "red",
  "message": "Status changed from yellow to red - Waiting for Kibana index \".kibana\" to come online failed.",
  "prevState": "yellow",
  "prevMsg": "No existing Kibana index found"
}

However, the '.kibana' index did actually get created. My Elasticsearch logs from a few seconds following the above entry:

[2017-11-01T21:43:36,195][INFO ][o.e.c.m.MetaDataCreateIndexService] [i-030fd4186fc5c5b96] [.kibana] creating index, cause [api], templates [], shards [1]/[1], mappings [_default_, index-pattern, server, visualization, search, timelion-sheet, config, dashboard, url]

curl http://internal-elasticsearch-1317456081.us-east-1.elb.amazonaws.com:9200/_cat/indices?v
health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .kibana WMs0MrFfSFqJ6VJUac7ykg   1   1          0            0       162b           162b

Since the error seems to happen when I have a load balancer in front of Elasticsearch, do you have any advice for configuring an Amazon ELB to sit in front of ES and serve as my elasticsearch.url?

Here's the ELB config I have currently:

Since I'm running a 3 node Elasticsearch cluster, my preference is to avoid using any of those individual IPs as my elasticsearch.url, as any of those 3 could die and be replaced by autoscaling. So my preference is to use an ELB (or any kind of load balancer) as the elasticsearch.url.

Thanks.

[jwr_cr (jwr)]

I ended up tearing out the ELB and doing this instead: https://www.elastic.co/guide/en/kibana/current/production.html#load-balancing