Question regarding filebeat indexes for version 8.11.x and ILM

Elastic Stack Beats
,
1

Hi @all

I am seeing an issue on the ELK production server w.r.t to ILM.

We have a production server (standalone) where ELK stack is installed and has the version 7.17.15.
Recently ILM policy was implemented and is running successfully. Please find the details below.

# curl -XGET http://localhost:9200
{
  "name" : "ip-x.x.x.x",
  "cluster_name" : "xxxx",
  "cluster_uuid" : "xxxxx",
  "version" : {
    "number" : "7.17.15",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "0b8ecfb4378335f4689c4223d1f1115f16bef3ba",
    "build_date" : "2023-11-10T22:03:46.987399016Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

# filebeat version
filebeat version 7.17.15 (arm64), libbeat 7.17.15 [b474d2803ed2961f23f614d7213d9099fb0b4354 built 2023-11-08 19:08:34 +0000 UTC]

GET _ilm/status

{
  "operation_mode" : "RUNNING"
}

However, currently I am seeing the filebeat indexes of different versions from other client servers and are utilizing high disk space e.g. as below.

yellow open   filebeat-8.11.1                    J9GUiOhkThilgzBp0i0RPQ   1   1  399800458            0    182.8gb        182.8gb
yellow open   filebeat-8.11.2                    hp8zHe1hTKmfuAJnTsy5fg   1   1   39897815            0     12.9gb         12.9gb
yellow open   filebeat-8.10.4                    HbQfL0pGS_ClJLLBWGFYUQ   1   1    4042769            0    920.3mb        920.3mb

I tried to look for a solution from forums and other sources but I could not find the right one. Please need to know on how to fix this issue and is there a way out, if we can assign to the existing ILM policy runnning (for beat 7.17.15) and also it should not conflict with the existing ILM policy because of the filebeat versions on the main ELK server and others from the client servers.

Thanks,
Ravi

2

Hello,

Please can someone help with a solution on the issue I posted.

Thanks,
Ravi

3

Hello,

I have tried reading from other sources and I am still not able to come to the conclusion on how to fix the current issue. could someone please help with their inputs?

Thanks,
Ravi

4

Hi @Ravi_Pattar

apologies but I'm not clear what you are asking?

Is it that you don't know where those other indices are coming from?

Is it that you want to apply ILM to those indices?

Is it that you want each of them on their own? ILM policy?

Did you run set up for each one of those?

What are you trying to accomplish?

5

Hi @stephenb

Thanks for the reply on this post. Please find the details mentioned inline.

As per my analysis these indices are coming from the other hosts.
Yes, I want to apply ILM for these indices. More details on the such indices with size is share below.

health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   filebeat-8.11.1                    neEszcRuQ3WFEdk_lZw7Ng   1   1  233861425            0    107.7gb        107.7gb
yellow open   filebeat-8.11.2                    ytS9uDDvQDSeoLCtinUNFQ   1   1   22845860            0      7.3gb          7.3gb
yellow open   filebeat-8.10.4                    HbQfL0pGS_ClJLLBWGFYUQ   1   1    6129880            0      1.3gb          1.3gb
yellow open   filebeat-8.10.4-2024.01.12         7ah5nAK6ShSf8oQhCSar3w   1   1      51352            0     10.2mb         10.2mb

Yes, can have separate ILM policy created for filebeat version 8.x? or its possible to add these to the existing ILM which in place?

ILM implemented for filebeat on the master ELK (stand alone) and the filebeat version is 7.17.15 and ILM is in operation mode successfully.

# filebeat version
filebeat version 7.17.15 (arm64), libbeat 7.17.15 [b474d2803ed2961f23f614d7213d9099fb0b4354 built 2023-11-08 19:08:34 +0000 UTC]

GET _ilm/status

{
  "operation_mode" : "RUNNING"
}

I am trying to check if there's a way to that these filebeat indices for version 8.x can be added to the existing ILM policy or we have to separate and create a new ILM policy.

Since its 8.x, it may work differently as per my earlier understanding.

Thanks,
Ravi

6

Can you run the following?

GET filebeat-8.11.1/_ilm/explain

Also, do you have something like logstash in between filebeat and elasticsearch?

Generally beats with version ahead of Elasticsearch do not work without making extra changes... I think you are in a weird state.

I suspect those indices are not under any ILM because they were not initiated / setup correctly.

With some work you can get them on separate ILM but I would work to get them on the default / same first.

What ILM policy are you trying to apply?

I have a couple ideas but need a little bit more information.

7

Hi @stephenb

My apologies for late response. Please find the details below, as I have tried to put the comments inline.

GET filebeat-8.11.1/_ilm/explain

{
  "indices" : {
    "filebeat-8.11.1" : {
      "index" : "filebeat-8.11.1",
      "managed" : false
    }
  }
}

Question: Also, do you have something like logstash in between filebeat and elasticsearch?
Ravi: Yes, we do have a logstash in between filebeat and elasticsearch

What ILM policy are you trying to apply?
Ravi: Same as what is in place for filebeat (7.x) currently.

GET filebeat-7.17.15-2024.02.19-000070/_ilm/explain

{
  "indices" : {
    "filebeat-7.17.15-2024.02.19-000070" : {
      "index" : "filebeat-7.17.15-2024.02.19-000070",
      "managed" : true,
      "policy" : "filebeat",
      "lifecycle_date_millis" : 1708344792258,
      "age" : "2.08d",
      "phase" : "hot",
      "phase_time_millis" : 1708344792540,
      "action" : "rollover",
      "action_time_millis" : 1708344792740,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1708344792740,
      "phase_execution" : {
        "policy" : "filebeat",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "10gb",
              "max_primary_shard_size" : "10gb",
              "max_age" : "7d"
            }
          }
        },
        "version" : 13,
        "modified_date_in_millis" : 1705325399371
      }
    }
  }
}

Thanks,
Ravi

8

Hello,

Please let me know if we have any solutions for my recent post.

Thanks,
Ravi

9

Hi @Ravi_Pattar Please be patient.

Well.... there are a number of issues...

First filebeat-8.11.1 is not under any form of ILM so it will continue to grow forever... probably the others indices are the same.

This is because filebeat setup was not run before the data came in so logtsash is just writing to an index that is not under ILM.

What makes it worse... the data is being written to filebeat-8.11.1, which SHOULD be a data stream, etc... which will manage the rollover, etc..etc...

but instead, it is just writing to a "concrete" normal index that is not under ILM

How to Fix? ... not easy.... and if you users just keep sending new beats data straight through with no prep work ahead

In short if you want to get this working correct...

You will need to move / reindex the existing indices.

Then run setup for 8.X beats so that they will write to the correct data streams

This COULD break the logstash output because the settings for data streams is a little different ...

So to fix this you have a lot of work....

All this could be avoided by having your users let you know what beats version they are going to use and run setup before they use them....

10

Hi @stephenb

I am in discussions with my internal team. Will keep you posted on how to handle this.

Thanks,
Ravi

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.