Recently rolled index does process data (0 docs)

Elastic Stack Elasticsearch
1

Hi People,

I'm having an issue with our ILM set up. Well, I think it's related to our ILM policy. We use ElasticSearch 7.

The policy rolled from 00001 to 00002 and suddenly it stopped processing data from our applications. See:

image
image2338×402 42.4 KB

The applications use SeriLog to send its logs, they use the following config:

        "logging": {
        "Serilog": {
          "WriteTo": [
            {
              "Name": "Elasticsearch",
              "Args": {
                "nodeUris": "http://es-cluster:9200/",
                "autoRegisterTemplate": "true",
                "autoRegisterTemplateVersion": "ESv6",
                "templateName": "app-serilog-events-template",
                "indexFormat": "app-index",
                "bufferBaseFilename": "/opt/app/logs/buffer-app",
                "restrictedToMinimumLevel": "Debug"
              }
            }
          ]
        },

The ILM policy looks like:

    PUT _ilm/policy/hot-warm-app
    {
      "policy": {
        "phases": {
          "hot": {
            "min_age": "0ms",
            "actions": {
              "rollover": {
                "max_size": "50gb",
                "max_age": "30d"
              },
              "set_priority": {
                "priority": 100
              }
            }
          },
          "warm": {
            "min_age": "2d",
            "actions": {
              "forcemerge": {
                "max_num_segments": 1
              },
              "set_priority": {
                "priority": 50
              },
              "shrink": {
                "number_of_shards": 1
              }
            }
          },
          "delete": {
            "min_age": "60d",
            "actions": {
              "wait_for_snapshot": {
                "policy": "Weekly-App-Indices"
              }
            }
          }
        }
      }
    }

I couldn't find any errors in the logs of ElasticSearch nor in the application.

Thanks in advance!

2

Can you check the status of the policy - https://www.elastic.co/guide/en/elasticsearch/reference/7.10/ilm-explain-lifecycle.html?

1 Like
3

Hi Warkolm,

Yes.

This is the output:

{
  "indices" : {
    "logging-app-000002" : {
      "index" : "logging-app-000002",
      "managed" : true,
      "policy" : "hot-warm-app",
      "lifecycle_date_millis" : 1610541813496,
      "age" : "4.8d",
      "phase" : "hot",
      "phase_time_millis" : 1610541815123,
      "action" : "rollover",
      "action_time_millis" : 1610542413458,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1610542413458,
      "phase_execution" : {
        "policy" : "hot-warm-app",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1607949667311
      }
    }
  }
}

This is the index (alias):

{
  "logging-app-000002" : {
    "aliases" : {
      "logging-app" : { }
    },
    "mappings" : { },
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "hot-warm-app",
          "rollover_alias" : "logging-app"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "logging-app-000002",
        "creation_date" : "1610541813496",
        "priority" : "100",
        "number_of_replicas" : "1",
        "uuid" : "KRYPd4cKTjm__sy3-546iQ",
        "version" : {
          "created" : "7100099"
        }
      }
    }
  }
}
4

What do you think?

5

@Zeroc0pter if you manually index a document into the alias, do you get an error? If you don't get an error, which index does the document get indexed into?

6

Hey Dakrone,

Thanks for the reply. So stupid of me that I did not try to index a doc myself.. This actually works. I'm thinking of Serilog having issues but not being logged properly. I'll ask the app team to update the Serilog library.

7

We've tried Filebaet but we've got the exact same issue. It does not write any logs although a manual index POST does work..

8

When you manually indexed the document, what index did it go into? (you should be able to see the concrete index name from the response JSON when indexing the document)

9

Hi @dakrone and @Zeroc0pter ,
I'm facing the same problem. Here's my scenario:

  1. If I create the index from Serilog WITHOUT any ILM it works fine, I see all docs being logged.
  2. If I create the index from Serilog WITH a ILM template, it doesn't log any docs to the index. It basically creates the index, apply the Lifecycle policy BUT doesn't add docs to the index.
  3. If I create the index from a third party tool (Postman), with or without ILM, it works as expected.

Have you found any solution?
Thanks!!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.