Elasticsearch fails to start 'unexpected character'

[fribse (Kenneth Fribert)]

Hi All

We have an elderly Elasticsearch 6.5.1 that we've never had the time to update.
It's running as docker container.

I was about to look into updating this, and shut it down to fix a few things to get it ready, but now it will not start, and I don't think it's a good idea to try an update on something that wont start

I see this in the logfile:

elasticsearch     | [2021-10-27T11:22:59,541][WARN ][r.suppressed             ] [es01] path: /winlog*/_search, params: {index=winlog*}
elasticsearch     | com.fasterxml.jackson.core.JsonParseException: Unexpected character (']' (code 93)): expected a value
elasticsearch     |  at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@62ea5647; line: 37, column: 12]

So it's supposedly something in our winlog.
It was running before I shut it down, and now everything is on the fritz.
How on earth do I fix this?

Our docker-compose looks like this

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.5.1
    container_name: elasticsearch
    environment:
      - node.name=es01
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms28g -Xmx28g"
      - 'path.repo=/opt/elasticsearch/backup'
# DO NOT GO ABOVE 31 GB, as that will move it to 64 bits, which is not recommended"
    ulimits:
      nproc: 65535
      memlock:
        soft: -1
        hard: -1
    cap_add:
      - ALL
    privileged: true
    volumes:
      - esdata1:/usr/share/elasticsearch/data
      - elastic-backup:/opt/elasticsearch/backup
    ports:
      - 9300:9300
    restart: always
  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.5.1
    container_name: elasticsearch2
    environment:
      - node.name=es02
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms28g -Xmx28g"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
      - 'path.repo=/opt/elasticsearch/backup'
    ulimits:
      nproc: 65535
      memlock:
        soft: -1
        hard: -1
    cap_add:
      - ALL
    privileged: true
    volumes:
      - esdata2:/usr/share/elasticsearch/data
      - elastic-backup:/opt/elasticsearch/backup
    restart: always
  elasticsearch3:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.5.1
    container_name: elasticsearch3
    environment:
      - node.name=es03
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms28g -Xmx28g"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
      - 'path.repo=/opt/elasticsearch/backup'
    ulimits:
      nproc: 65535
      memlock:
        soft: -1
        hard: -1
    cap_add:
      - ALL
    privileged: true
    volumes:
      - esdata3:/usr/share/elasticsearch/data
      - elastic-backup:/opt/elasticsearch/backup
    restart: always

Anybody that can help me out there?

[fribse (Kenneth Fribert)]

Weehee, I made progress, I shut down most of it, and then started the first Elasticsearch, waited for it settle before starting the others, and then it worked. Wheew.
What a releif.
I still need to get Kibana flying again, but at least now it's talking to me.