Issue with rollover

Elastic Stack Elasticsearch
1

Hi,

I am trying to bootstrap an index to write after and getting the below error:

{
"error" : {
"root_cause" : [
{
"type" : "resource_already_exists_exception",
"reason" : "index [logs-sbk-000001/qa4ur3ZUQCmqugKN9mXLuQ] already exists",
"index_uuid" : "qa4ur3ZUQCmqugKN9mXLuQ",
"index" : "logs-sbk-000001"
}
],
"type" : "resource_already_exists_exception",
"reason" : "index [logs-sbk-000001/qa4ur3ZUQCmqugKN9mXLuQ] already exists",
"index_uuid" : "qa4ur3ZUQCmqugKN9mXLuQ",
"index" : "logs-sbk-000001"
},
"status" : 400
}

Steps:

  1. Created index lifecycle policy.
  2. Created a Index template for pattern "logs*" with alias to "logs-sbk" and attached the created ILP with rollover enable.
  3. Logstash is sending the logs to created an index in elastic with name "logs-sbk-000001".
  4. Index gets created.
  5. When i try to bootstrap the alias to write it fails.

What is the right way to do it?

When the index gets created it doesnt take the alias that why i have to boostrap for the index to take up the alias.

Here is my setting for template:

{
  "index": {
    "lifecycle": {
      "name": "log_small_size",
      "rollover_alias": "logs-sbk"
    },
    "codec": "best_compression",
    "mapping": {
      "total_fields": {
        "limit": "2000"
      }
    },
    "number_of_shards": "1",
    "number_of_replicas": "1"
  }
}

Can some one please let me know if this is being done correctly?

2

I went through some trial and error in getting ILM working for indices of rsyslog events in a small dev/test enclave, and part of that was a realization that I just wasn't figuring out how to get ILM policy applied to an existing index - so I went with a "blank sheet" approach. Your steps look good and are real close to what I did:

· create the policy
· create the template

PUT _template/syslog_ilm_template {
  “index_patterns”: [“syslog-*”],
  “settings”: {
    “number_of_shards”: 1,
    “number_of_replicas”: 1,
    “index.lifecycle.name”: “syslog_ilm_policy”,
    “index.lifecycle.rollover_alias”: “syslog-”
  }
}

· stop Logstash and modify the output that sends syslog events to Elasticsearch to send to the (not yet existing) syslog- index
· create the bootstrap index syslog-000001 with an alias of syslog-

PUT syslog-000001
{
  “aliases”: {
    “syslog-”: {
      “is_write_index”: true
    }
  }
}

· and then restart Logstash.
· Logstash then starts sending to syslog-, which is "intercepted" and sent to first syslog-000001, which then later rolls over to syslog-000002, etc.

I don't know if that's the way it's supposed to be done but it worked.

3

Hi Bob,

Thanks for taking time to respond.

I tried the same but sometimes when it is required to restart the logstash which writes the index as below. When there is a rollover lets say for a 2 cycle's and the third is active per say something like -000003 , logstash tries to recreate the index from suffix -000001, this is like a mess so i have to recreate everything all over. There must be a write way to do it.

Logstash pipeline sample config:

input {
  beats {
    port => 5045
  }

}
filter {
    if ( [@metadata][beat] == "filebeat" ) {
        if ( [log-type] == "sbk-application" ) {
	  mutate {
            add_field => {
              "[@metadata][target_index]" => "log-%{[service]}-000001"
            }
          }
        }  else if ( [log-type] == "sbk-gc" ) {
	     	mutate {
                            add_field => {
                                        "[@metadata][target_index]" => "log-%{[service]}-gc-000001"
                                }
                        }
		} else if ( [log-type] == "sbk-access" ) {
			mutate {
                                add_field => {
                                        "[@metadata][target_index]" => "log-%{[service]}-access-000001"
                                }
                        }
		} else if ( [service] == "sbk-nginx" ) {
                        mutate {
                                add_field => {
                                        "[@metadata][target_index]" => "log-%{[service]}-000001"
                                }
                        }
                }

    }

}
output {
  elasticsearch {
    hosts => ["http://instance1:9200", "http://instance2:9200", "http://instance3:9200"]
    user => logstash_host
    password => ******
    index => "%{[@metadata][target_index]}"
  }
}

It would be great if someone from elastic team can look into this and respond.

4

@warkolm @shaunak @chrisronline @ElasticStewart

Hey Guys, Can anybody please check on my above query. Would really appreciate it.

Thanks!

Hey @RLPowellJr

Please let me know if your logstash looks something like responded to your earlier comment. Appreciate your help.

5

Another observation to the above--

After the index rollsover (based on ILMP) lets say *000002. I don't see any data getting write to the index.

ILM/explain:

{
  "indices" : {
    "log-sbk-000003" : {
      "index" : "log-sbk-000003",
      "managed" : true,
      "policy" : "logging_small_size",
      "lifecycle_date_millis" : 1599458246094,
      "age" : "4.92d",
      "phase" : "hot",
      "phase_time_millis" : 1599458346982,
      "action" : "rollover",
      "action_time_millis" : 1599458880465,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1599458880465,
      "phase_execution" : {
        "policy" : "logging_small_size",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "1gb",
              "max_age" : "7d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 5,
        "modified_date_in_millis" : 1599022060429
      }
    },
    "log-sbk-000002" : {
      "index" : "log-sbk-000002",
      "managed" : true,
      "policy" : "logging_small_size",
      "lifecycle_date_millis" : 1599458245941,
      "age" : "4.92d",
      "phase" : "warm",
      "phase_time_millis" : 1599631613646,
      "action" : "complete",
      "action_time_millis" : 1599632804490,
      "step" : "complete",
      "step_time_millis" : 1599632804490,
      "phase_execution" : {
        "policy" : "logging_small_size",
        "phase_definition" : {
          "min_age" : "2d",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 1,
              "include" : { },
              "exclude" : { },
              "require" : { }
            },
            "forcemerge" : {
              "max_num_segments" : 1
            },
            "set_priority" : {
              "priority" : 50
            },
            "shrink" : {
              "number_of_shards" : 1
            }
          }
        },
        "version" : 5,
        "modified_date_in_millis" : 1599022060429
      }
    },
    "log-sbk-000001" : {
      "index" : "log-sbk-000001",
      "managed" : true,
      "policy" : "logging_small_size",
      "lifecycle_date_millis" : 1598852832678,
      "age" : "11.93d",
      "phase" : "cold",
      "phase_time_millis" : 1599717074974,
      "action" : "complete",
      "action_time_millis" : 1599717351698,
      "step" : "complete",
      "step_time_millis" : 1599717351698,
      "phase_execution" : {
        "policy" : "logging_small_size",
        "phase_definition" : {
          "min_age" : "10d",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 1,
              "include" : { },
              "exclude" : { },
              "require" : { }
            },
            "freeze" : { },
            "set_priority" : {
              "priority" : 0
            }
          }
        },
        "version" : 5,
        "modified_date_in_millis" : 1599022060429
      }
    }
  }
}
6

GET log-sbk-*

{
  "log-sbk-000001" : {
    "aliases" : {
      "log-sbk" : {
        "is_write_index" : false
      }
    },
    "mappings" : {
      "dynamic" : "true",
      "_meta" : { },
      "_source" : {
        "includes" : [ ],
        "excludes" : [ ]
      },
      "dynamic_date_formats" : [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic_templates" : [ ],
      "date_detection" : true,
      "numeric_detection" : false,
      "properties" : {
        "@timestamp" : {
          "type" : "date",
          "format" : "strict_date_optional_time"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "agent" : {
          "properties" : {
            "ephemeral_id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "cloud" : {
          "properties" : {
            "availability_zone" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "instance" : {
              "properties" : {
                "id" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "machine" : {
              "properties" : {
                "type" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "project" : {
              "properties" : {
                "id" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "provider" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "ecs" : {
          "properties" : {
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "host" : {
          "properties" : {
            "architecture" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "containerized" : {
              "type" : "boolean"
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "os" : {
              "properties" : {
                "codename" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "family" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "kernel" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "platform" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "version" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            }
          }
        },
        "input" : {
          "properties" : {
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "log" : {
          "properties" : {
            "file" : {
              "properties" : {
                "path" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "flags" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "offset" : {
              "type" : "long"
            }
          }
        },
        "log-type" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "service" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "tags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "2000"
          }
        },
        "blocks" : {
          "write" : "true"
        },
        "provided_name" : "log-sbk-000001",
        "frozen" : "true",
        "creation_date" : "1598247712306",
        "priority" : "0",
        "number_of_replicas" : "1",
        "uuid" : "cmkJg1WwT-qC1LUKiyhXwg",
        "version" : {
          "created" : "7060299"
        },
        "lifecycle" : {
          "name" : "logging_small_size",
          "rollover_alias" : "log-sbk",
          "indexing_complete" : "true"
        },
        "codec" : "best_compression",
        "search" : {
          "throttled" : "true"
        },
        "number_of_shards" : "1"
      }
    }
  },
  "log-sbk-000002" : {
    "aliases" : {
      "log-sbk" : {
        "is_write_index" : false
      }
    },
    "mappings" : {
      "dynamic" : "true",
      "_meta" : { },
      "_source" : {
        "includes" : [ ],
        "excludes" : [ ]
      },
      "dynamic_date_formats" : [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic_templates" : [ ],
      "date_detection" : true,
      "numeric_detection" : false
    },
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "logging_small_size",
          "rollover_alias" : "log-sbk",
          "indexing_complete" : "true"
        },
        "codec" : "best_compression",
        "mapping" : {
          "total_fields" : {
            "limit" : "2000"
          }
        },
        "number_of_shards" : "1",
        "blocks" : {
          "write" : "true"
        },
        "provided_name" : "log-sbk-000002",
        "creation_date" : "1598852832667",
        "priority" : "50",
        "number_of_replicas" : "1",
        "uuid" : "FPgQZZ0bQWuwKGURRFXXbg",
        "version" : {
          "created" : "7060299"
        }
      }
    }
  },
  "log-sbk-000003" : {
    "aliases" : {
      "log-sbk" : {
        "is_write_index" : true
      }
    },
    "mappings" : {
      "dynamic" : "true",
      "_meta" : { },
      "_source" : {
        "includes" : [ ],
        "excludes" : [ ]
      },
      "dynamic_date_formats" : [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic_templates" : [ ],
      "date_detection" : true,
      "numeric_detection" : false
    },
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "logging_small_size",
          "rollover_alias" : "log-sbk"
        },
        "codec" : "best_compression",
        "mapping" : {
          "total_fields" : {
            "limit" : "2000"
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "log-sbk-000003",
        "creation_date" : "1599458246094",
        "priority" : "100",
        "number_of_replicas" : "1",
        "uuid" : "zfamI-4zTXWXGQl0Uc1hkA",
        "version" : {
          "created" : "7060299"
        }
      }
    }
  }
}
7

Before the index creating i have the below bootstrapped and when the logstash start up the bootstrapped index gets filled but the rollover index *000002 fails to write.

Bootstrap api:

PUT  log-sbk-000001
{
  "aliases": {
    "log-sbk":{
      "is_write_index": true 
    }
  }
}
8

I can see a few potential issues with what you are doing.

As far as I know you should direct Logstash and Filebeat to write to the write alias, in this case log-sbk not the initial index.

As far as I know each write alias and ILM pattern need to be set up explicitly so you can not use dynamic patterns like "log-%{[service]}-gc-000001" in your config as these does not match an existing write alias.

9

I do have it set up the pattern and alias explicitly. The above configuration is just a sample from my config is only the details of 1 index pattern log-sbk.

I will try to change my logstash config to the alias and then see if it works.

10

There is another observation i noticed so i have about 150 pattern templates how does the order apply , currently i have 0 set to all so the explicit alias set for each pattern gets confused taking someother alias name which actually needs to be something else.

so my question is if each template pattern is explicit what should be the order if i have 150 templates with 150 aliases?

1-150?

11

I am not sure I understand. Can you please provide some examples?

I would expect you to have an index template and write alias for every index pattern and ILM policy if you are indexing into multiple index patterns.

12

Template 1:

PUT _template/kafka-topic-logging-kafka
{
  "order": 0,
  "index_patterns": [
    "kafka-topic-logging-kafka*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "kafka-topic_small_size",
        "rollover_alias": "kafka-topic-logging-kafka"
      },
      "codec": "best_compression",
      "mapping": {
        "total_fields": {
          "limit": "2000"
        }
      },
      "number_of_shards": "1",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "_doc": {
      "_routing": {
        "required": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "_meta": {},
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic": true,
      "dynamic_templates": [],
      "date_detection": true,
      "properties": {}
    }
  }
}

===================================================================

Template 2:

PUT _template/kafka-topic-metrics-jmx-kafka
{
  "order": 0,
  "index_patterns": [
    "kafka-topic-metrics-jmx-kafka*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "kafka-topic_large_size",
        "rollover_alias": "kafka-topic-metrics-jmx-kafka"
      },
      "codec": "best_compression",
      "mapping": {
        "total_fields": {
          "limit": "2000"
        }
      },
      "number_of_shards": "3",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "_doc": {
      "_routing": {
        "required": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "_meta": {},
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic": true,
      "dynamic_templates": [],
      "date_detection": true,
      "properties": {}
    }
  }
}
13

Screen Shot 2020-09-12 at 12.42.43 PM
Screen Shot 2020-09-12 at 12.42.43 PM1246×146 16.3 KB

14

Another example:

Screen Shot 2020-09-12 at 12.44.31 PM
Screen Shot 2020-09-12 at 12.44.31 PM1452×326 30.6 KB

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.