Latest rolled over data not searchable in Discover menu

[Geeboy]

Im new to ELK (Im using td-agent as log forwarder) I have created ILM rollover in my dev environment, my index was successfully rolling over, but I can't search the new index in the latest rollover in the discover. i have also added my index in Data Views. sharing my config below.

GET cinqv2*/_alias/cinqv2-alias?pretty
{
  "cinqv2" : {
    "aliases" : {
      "cinqv2-alias" : { }
    }
  },
  "cinqv2-000003" : {
    "aliases" : {
      "cinqv2-alias" : {
        "is_write_index" : true
      }
    }
  },
  "cinqv2-000002" : {
    "aliases" : {
      "cinqv2-alias" : {
        "is_write_index" : false
      }
    }
  },
  "cinqv2-000001" : {
    "aliases" : {
      "cinqv2-alias" : {
        "is_write_index" : false
      }
    }
  }
}

GET _index_template/default-template
{
  "index_templates" : [
    {
      "name" : "default-template",
      "index_template" : {
        "index_patterns" : [
          "cinqv2-*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "lifecycle" : {
                "name" : "default_policy",
                "rollover_alias" : "cinqv2-alias"
              },
              "number_of_shards" : "1",
              "number_of_replicas" : "0"
            }
          },
          "mappings" : {
            "_source" : {
              "enabled" : true
            }
          }
        },
        "composed_of" : [ ]
      }
    }
  ]
}

GET _ilm/policy/default_policy
{
  "default_policy" : {
    "version" : 2,
    "modified_date" : "2023-08-29T23:32:59.573Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "200kb",
              "max_age" : "30m",                   ---->> for quick rollover scenario
              "max_docs" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "1d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [
        "cinqv2-000001",
        "cinqv2-000003",
        "cinqv2-000002"
      ],
      "data_streams" : [ ],
      "composable_templates" : [
        "default-template"
      ]
    }
  }
}

on this image. trying to search the _index (cinqv2-000001) but can't find it via discover menu.

[Geeboy]

here is my index rollover

[Geeboy]

Any help in understanding where I could have gone wrong in my setup or formatting would be greatly appreciated. Thanks in advance

[stephenb (Stephen Brown)]

How did you create the index pattern/ data view?

Is the index pattern

cinqv2-*

Do any of the indices show?

[Geeboy]

thank you very much for your reply. I used this command (executed via kibana console). the only index I can see in the discover (_index) is cinqv2, nothing more.

PUT cinqv2/_alias/cinqv2-alias

PUT cinqv2-000001
{
   "aliases": {
        "cinqv2-alias":{
            "is_write_index": true
        }
    }
}

I created the data view using the kibana UI (Create data view botton)

here is the list of rollover indexes

[stephenb (Stephen Brown)]

Your index pattern is only matching the very first index cinqv2 it is not matching i.e. then you are not searching any of the other indices,

That is what I'm trying to say

The index pattern needs to be
cinqv2* as the matching pattern to match all the indices or you will need to put cinqv2,cinqv2-* to account for both patterns

it is not best practice to have
cinqv2
cinqv2-000001

[Geeboy]

good day Sir, is this the only parameter that I need to change? from "cinqv2-" change to "cinqv2"

GET _index_template/default-template
{
  "index_templates" : [
    {
      "name" : "default-template",
      "index_template" : {
        "index_patterns" : [
          "cinqv2*"
        ],

[stephenb (Stephen Brown)]

That is the template you don't need to change that, the template is on the index creation process.

You need to change the index pattern which is used for the search / Discover/ visualization

Stack Management -> Index Pattern...that is what you need to fix.

This one

Delete it a recreate with

cinqv2*

The check Discover

[Geeboy]

thanks for this Sir. i already create new index (cinqv2*) in "data views"

do i need to delete the old index (cinqv2) in data views and delete all in index management?

[stephenb (Stephen Brown)]

I would get rid of the old data view. You don't need to get rid of the index unless you want to

Technically in the data view you could define the index pattern

cinqv2,cinqv2-*

To cover both that original index and the ones with the numbers

One gotcha. Don't put a space after the comms

[Geeboy]

GREAT! I can now view the content of the rollover index in the discover. your a great help @stephenb thanks a lot