Upgraded beats (7.10) still writing to old version indices

[rgeisman (Rachel)]

Hello all. A maddening problem. We upgraded our cluster to 7.10 (ES, Logstash, Metricbeat, Auditbeat, Filebeat). Since we use Logstash I manually loaded each index template for the beats and default dashboards, then I removed the old index templates for earlier versions, and closed the indices for the old ones.

When we start logstash it tries to write to the old beat index! Here is our output below, along with the error we see in Logstash. Anyone have a hint on how to fix this?

Not writing to new 7.10 index. Trying to use old one!

[2020-12-02T11:48:05,438][WARN ][logstash.outputs.elasticsearch][main][a95592bf084961207f2e0f22aae7a98b42d2e6b6aee1995c6a0b944fffefabb6] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"auditbeat-7.4.0-2020.11.06", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x5c0b1e5c>], :response=>{"index"=>{"_index"=>"auditbeat-7.4.0-2020.11.06", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"index_closed_exception", "reason"=>"closed", "index_uuid"=>"gaWjJRBERhSifVG82Hu9qg", "index"=>"auditbeat-7.4.0-2020.11.06"}}}}

In our output we set a dynamic index to write the correct beat index...from this it should write a new 7.10 index correct? But it doesn't. It goes to the old one. I checked the individual servers and it has the correct 7.10 beat version on it.

output {
    elasticsearch {
      hosts => ["hostname (secret)"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      #index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      manage_template => false
    }
 }

[rgeisman (Rachel)]

When looking at the records coming in we see this:

_index:"filebeat-7.4.2-2020.12.02"

but with setting our index in Logstash to dynamically pick up new 7.10 beats:

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

it should be:
_index :filebeat-7.10.0-2020.12.02

But it's not!

[warkolm (Mark Walkom)]

Can you post an example document that Filebeat is sending?

[rgeisman (Rachel)]

Sure thing. Here is an example of the record filebeat is now sending from the filebeat 107 beat:

{
  "_index": "filebeat-7.4.2-2020.12.03",
  "_type": "_doc",
  "_id": "M8N9JnYBy_AFci_sJL7s",
  "_version": 1,
  "_score": null,
  "_source": {
    "@version": "1",
    "log": {
      "file": {
        "path": "/var/log/messages"
      },
      "offset": 13676358
    },
    "input": {
      "type": "log"
    },
    "host": {
      "os": {
        "name": "CentOS Linux",
        "version": "7 (Core)",
        "family": "redhat",
        "codename": "Core",
        "platform": "centos",
        "kernel": "3.10.0-957.1.3.el7.x86_64"
      },
      "name": "accre_fb",
      "architecture": "x86_64",
      "hostname": "hostname.secret",
      "containerized": false,
      "id": "fa8859409a0b45f89c8e4176cf1c38ab"
    },
    "message": "Nov 21 11:41:06 host(secret) systemd: ipmi-getty.service holdoff time over, scheduling restart.",
    "ecs": {
      "version": "1.1.0"
    },
    "agent": {
      "name": "accre_fb",
      "version": "7.4.2",
      "ephemeral_id": "db25b54f-15d9-4806-8322-2ba7d8f12420",
      "hostname": "hostname (secret)",
      "type": "filebeat",
      "id": "9bab1b60-9d92-460a-bc68-a97b96c10e63"
    },
    "tags": [
      "beats_input_codec_plain_applied",
      "_grokparsefailure",
      "_geoip_lookup_failure"
    ],
    "@timestamp": "2020-12-03T02:24:26.927Z"
  },
  "fields": {
    "@timestamp": [
      "2020-12-03T02:24:26.927Z"
    ],
    "suricata.eve.timestamp": [
      "2020-12-03T02:24:26.927Z"
    ]
  },
  "sort": [
    1606962266927
  ]
}

[rgeisman (Rachel)]

As. you can see when I loaded the template manually (by disconnecting Logstash, and then using curl injected the new 10.7 template into ES and restarted...but Logstash/Beat just keeps on making old index names. The Logstash config is set up to dynamically name the beat by beat-version-time as is normal.

!

[rgeisman (Rachel)]

It's also showing ILM errors, and I turned off lifecyle management to try and troubleshoot. Here is the exception.

[warkolm (Mark Walkom)]

Are you 100% sure Filebeat is upgraded? Did you restart it after the upgrade?

[Badger]

@rgeisman

Here is an example of the record filebeat is now sending from the filebeat 107 beat

Can you show the output from

output { stdout { codec => rubydebug { metadata => true } } }

for such an event.

[rgeisman (Rachel)]

It's managed through an orchestration platform. It's on 1000 nodes. Yes, I do a filebeat version on many of then and have 10.7 returned? There aren't two version on the boxes. I thought that perhaps some of the boxes didn't upgrade, but going into kibana I look at the records and all of them have 7.4 in the _index. Then I check the hostname box where it came from and 10.7 is running.

[rgeisman (Rachel)]

Sure, but I am running Logstash as a service (systemctl). I know, I will stop the service and start logstash manually (/bin/logstash) with this. Thanks for debug hint.