Elasticsearch no longer writing to index(s) after upgrading from 7.6 to 7.7

Elastic Stack Elasticsearch
1

Last night we upgraded from 7.6 to 7.7. Since then we have not had any logs coming into Elasticsearch.

Our pipeline looks like so:

filebeat on nodes ship to a single ELK server

I enabled ruby debug logging in logstash and logs are coming into logstash, logs look fine and I am not getting any errors.

I enabled debug logging in elasticsearch and I see what looks like healthy connections from logstash delivering logs:

[2020-06-11T18:30:20,989][TRACE][o.e.t.T.tracer           ] [ps-dev-elk] [841][indices:data/write/bulk[s][p]] received response from [{ps-dev-elk}{qmfFc9wtTn20jWjM6mCQMw}{iExEVJUJSJa36HZgIL-gOw}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=16808701952, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]
[2020-06-11T18:30:20,990][TRACE][o.e.t.T.tracer           ] [ps-dev-elk] [843][indices:admin/seq_no/global_checkpoint_sync[p]] sent to [{ps-dev-elk}{qmfFc9wtTn20jWjM6mCQMw}{iExEVJUJSJa36HZgIL-gOw}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=16808701952, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] (timeout: [null])
[2020-06-11T18:30:20,991][TRACE][o.e.t.T.tracer           ] [ps-dev-elk] [843][indices:admin/seq_no/global_checkpoint_sync[p]] received request
[2020-06-11T18:30:20,991][TRACE][o.e.t.T.tracer           ] [ps-dev-elk] [842][indices:data/write/bulk[s][p]] sent response
[2020-06-11T18:30:20,991][TRACE][o.e.t.T.tracer           ] [ps-dev-elk] [842][indices:data/write/bulk[s][p]] received response from [{ps-dev-elk}{qmfFc9wtTn20jWjM6mCQMw}{iExEVJUJSJa36HZgIL-gOw}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=16808701952, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]
[2020-06-11T18:30:20,991][TRACE][o.e.t.T.tracer           ] [ps-dev-elk] [843][indices:admin/seq_no/global_checkpoint_sync[p]] sent response
[2020-06-11T18:30:20,991][TRACE][o.e.t.T.tracer           ] [ps-dev-elk] [843][indices:admin/seq_no/global_checkpoint_sync[p]] received response from [{ps-dev-elk}{qmfFc9wtTn20jWjM6mCQMw}{iExEVJUJSJa36HZgIL-gOw}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=16808701952, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]

However I am not seeing these logs hit any indexes in elasticsearch. No index documents are increasing. Here is my current indexes:

root@ps-dev-elk:/var/log/logstash# curl -XGET localhost:9200/_cat/shards?v
index                    shard prirep state          docs   store ip        node
.kibana_8                0     p      STARTED          91 126.6kb 127.0.0.1 ps-dev-elk
.kibana_task_manager_1   0     p      STARTED           2   7.2kb 127.0.0.1 ps-dev-elk
logstash-2020.06.03      0     p      STARTED    14712122   7.3gb 127.0.0.1 ps-dev-elk
logstash-2020.06.03      0     r      UNASSIGNED
logstash-2020.06.02      0     p      STARTED    14813812   7.3gb 127.0.0.1 ps-dev-elk
logstash-2020.06.02      0     r      UNASSIGNED
logstash-2020.06.10      0     p      STARTED    12666539   5.7gb 127.0.0.1 ps-dev-elk
logstash-2020.06.10      0     r      UNASSIGNED
logstash                 0     p      STARTED     3057574   1.8gb 127.0.0.1 ps-dev-elk
logstash                 0     r      UNASSIGNED
.kibana_task_manager_3   0     p      STARTED           6    21kb 127.0.0.1 ps-dev-elk
.apm-custom-link         0     p      STARTED           0    230b 127.0.0.1 ps-dev-elk
.apm-agent-configuration 0     p      STARTED           0    283b 127.0.0.1 ps-dev-elk
logstash-2020.06.08      0     p      STARTED    12663722   5.7gb 127.0.0.1 ps-dev-elk
logstash-2020.06.08      0     r      UNASSIGNED
.kibana_task_manager_2   0     p      STARTED           3  25.9kb 127.0.0.1 ps-dev-elk
.kibana_10               0     p      STARTED         241   169kb 127.0.0.1 ps-dev-elk
logstash-2020.06.04      0     p      STARTED    14423582   7.1gb 127.0.0.1 ps-dev-elk
logstash-2020.06.04      0     r      UNASSIGNED
.async-search            0     p      STARTED           2 685.9kb 127.0.0.1 ps-dev-elk
.kibana_9                0     p      STARTED         227 220.2kb 127.0.0.1 ps-dev-elk
logstash-2020.06.06      0     p      STARTED    12435442   5.5gb 127.0.0.1 ps-dev-elk
logstash-2020.06.06      0     r      UNASSIGNED
logstash-2020.06.07      0     p      STARTED    12436608   5.5gb 127.0.0.1 ps-dev-elk
logstash-2020.06.07      0     r      UNASSIGNED
.kibana-6                0     p      STARTED          75  78.3kb 127.0.0.1 ps-dev-elk
.kibana-6                0     r      UNASSIGNED
logstash-2020.06.05      0     p      STARTED    12650352   5.7gb 127.0.0.1 ps-dev-elk
logstash-2020.06.05      0     r      UNASSIGNED
.tasks                   0     p      STARTED           1   6.6kb 127.0.0.1 ps-dev-elk
logstash-2020.06.09      0     p      STARTED    12686744   5.8gb 127.0.0.1 ps-dev-elk
logstash-2020.06.09      0     r      UNASSIGNED
.kibana_7                0     p      STARTED          86  87.2kb 127.0.0.1 ps-dev-elk

As you can see we use daily indexes and use the logstash-* pattern. I deleted the index for today. Normally this would result in it instantly getting recreated as new logs for today come in but this did not happen, it never recreated it.

Can anyone help me determine where these logs are going?

2

According to apt.log I was actually on version 7.5.1 prior to this upgrade. I tried downgrading from 7.7.1 to 7.6.0 but I got some errors on startup saying IndexFormatTooNewException. I was able to downgrade to 7.6.2 without errors but I am still seeing the same issue. No errors however nothing is being written by elasticsearch.

I attempted to file a bug with elasticsearch but because I have no error logs to show the bug was closed. Any help would be much appreciated.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.