Convert field type for ILM managed indices

I'm running into what I would assume is a common problem. Apologies in advanced for the long post, I just reviewed my steps and I realize that there are a lot of steps involved. Hopefully fleshing out this process will result in something simpler for myself and others.

I'm trying to transform my field types to better use my existing data. I've seen a lot of posts about this but still I'm struggling to get to a resolution, so figured I'd document my steps where others can re-use the information.

I have an index alias httpd that refers to the last 35d worth of indices. I have corrected the type in my logstash filter so that new documents are coming in with the proper field type using the grok "int" suffix on field processing time like so %{INT:processing_time:int}. Previously had tested mutate convert to the same effect, for those following along at home who aren't using grok, but do note that the convert statement should be in a separate mutate block than any existing mutate blocks. This appears to be working, as now I have a mapping conflict on the httpd-* index pattern.

Now of course, I need to convert my previous indices so that I can properly aggregate on this new field type. I found a post which outlined the steps:

  1. Define an ingest pipeline which uses convert processor.
  2. Use this ingest pipeline in the reindex operation.

After some more research to tailor this to my needs, I ended up with the following, partially working API requests:

PUT _ingest/pipeline/make-integer
  "description": "converts the field type of the defined field to an integer",
  "processors" : [
      "convert" : {
        "field" : "processing_time",
        "type": "integer", 

POST _reindex
  "source": {
    "index": "httpd-2019.10.01-000002"
  "dest": {
    "index": "httpd-2019.10.01-000002-convert-integer-processing_time_1",
    "pipeline": "make-integer"
  , "conflicts": "proceed"

Please note that you should not need the "ignore_failure":true or "conflicts": "proceed" statements if your data is more standardized than mine, I just ran into some trouble with a very small percentage of my log not containing this value and failing halfway through the job's batches.

These 2 requests and a couple hours later, I have a replica of the index with what I assume is a corrected field type. I have tested the mapping with GET /httpd-2019.10.01-000002-convert-integer-processing_time_4/_mapping/field/processing_time which shows the new mapping type that I expect. Now, the problem that I am having is that the new index is throwing ILM errors. This initially looked like index [httpd] is not the write index for alias [httpd-2019.10.11-000012] but I found a guide that described how to create a new policy to age out old indices and apply it without giving any rollover or active policy. This appears to fit my needs, as I had already rolled the index manually once the type change was implemented. However, now my index has the error illegal_argument_exception: index.lifecycle.rollover_alias [httpd] does not point to index [httpd-2019.10.01-000002-convert-integer-processing_time_4]. I see that others have corrected this with Update index alias API, but my query doesn't seem to do the trick - there doesn't appear to be any change in the error.

POST /_aliases
    "actions" : [
        { "add" : { "index" : "httpd-2019.10.01-000002-convert-integer-processing_time_4", "alias" : "httpd" } }

So for now, that's about where I'm stuck. I'd also like to get this index name back to its original httpd-2019.10.01-000002 which I assume is a simple fix, but I'd like to validate the type conversion before taking this step.

<edit - mistook block quotes for multiline preformatted text.

Doing some more research, it appears that renaming an index is actually NOT simple. Any way that it is done is by copying an index and deleting the old one. Is it really not possible to make this field conversion happen without deleting the original data in one way or another? I understand why it must be done in the long run but is there some way that I can test that everything is healthy before committing to a delete? At this point it is seeming like my best next step is to delete the original index and re-copy the converted index back over to the original name while hoping for the best.

Seems like I'm not allowed to make edits any more for some reason... I'll just reply even though no one has responded yet.

I've had an idea to run another reindex against the same convert processor, but give it a name that matches the alias at the top of the range (999999) which I should never hit. This should result in my new index with the proper field type which matches the alias pattern. The job is running now so I'll get back on the results of that soon. I'm still curious what happens when I choose 000017 when we have 16 as the current hot index, but not wanting to test it here in production.

So, that didn't work. I am still getting the error illegal_argument_exception: index.lifecycle.rollover_alias [httpd] does not point to index [httpd-2019.10.01-999999]. At this point, I'm taking a moment to compare the configuration of each of these indices. In the settings, the new index indicates that indexing is not complete, and it has a static provided name vs the old one i.e. "httpd-2019.10.01-999999" vs "<httpd-{now/d}-000002>". For the mapping, the only difference is the expected type change. For the settings, the new index is again, missing the indexing_complete: "true". Researching...

So, I dug into ILM some and found out how to correct this. You need to manually correct the setting with the update index settings API, followed by an ILM retry. This will clear the Alias error.

PUT /<index>/_settings
  "index" : {
    "lifecycle": {
      "indexing_complete" : "true"

POST <index>/_ilm/retry

I'm going to build a couple more queries and double-check the configuration comparison again to test that everything looks okay before I start deleting the old indices. Fingers crossed that was the last hurdle.

Seems like I can sum and bucket the suspect field now. I want to finish writing up my process so that I can make a nice clean guide. I'll try to get back to this tomorrow.

I have another remaining mapping conflict but I know the source, because I changed another field in the pipeline as well. I'll try to get that fixed tomorrow, along with the guide.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.