Index template with only one field across multiple indices

Elastic Stack Elasticsearch
1

Hi,

I have multiple indices like mailbox-xx- xxxx .The xx-xxxx values are assigned dynamically. Each index contains various fields with date as a common field. The problem that date field has been read as long type by default , but I need to keep it as date type.

I tried to use _index_template API and specify only the date field in mappings, but I saw that data is not written to those indices since the mapping mismaches between the index template and actual query.

Is there anything can be done here?

Many thanks

2

What is the template you created? You need to share it.

Also, did you deleted the old indices and created new ones? Templates only work when creating the index.

3

Hi,

THank you for your reply.

Here is my template:

PUT _index_template/mailbox
{
   "index_patterns":[
      "mailbox*"
   ],
   "priority":"100",
   "template":{
      "mappings":{
         "properties":{
            "date":{
               "type":"date",
               "format":"epoch_second"
            }
         }
      }
   }
}

Now, after the template is created I start loading data to elasticsearch. The query has more fields than a date. The data is not written to the index.

4

And what does your date string looks like?

Please share a sample of your log messages.

5

Hi,

Date is long type, I need to force it into date type.
Below is the exmaple

 {
        "_index" : "mailbox-ps-2022.10",
        "_type" : "PS",
        "_id" : "A2C1418190071_227264124_1666616748",
        "_score" : 1.0,
        "_source" : {
          "emetteur" : "fxd600",
          "comment" : "defaut systeme",
          "reference" : "A2C1418190071",
          "SN" : "227264124",
          "date" : 1666616748,
          "site" : "trwwv"
        }
      }

Wil dynamic template within index template work for this?

6

No, your mapping is correct, it should work.

Did you deleted the old indices or created a new index?

How are you loading your data?

7

I'am on the version 7.16.
I want to create mapping for one common filed across multiple indices.
I deleted all the indices and created the mapping.
I started loading the data. I use _search api to see if the date field is mapped accordingly, but the indices are empty. If I remove the _index_template the loading works fine.

8

I understand what you are trying to do, but you didn't explained how you are loading the data.

If after the template was applied you got no data in the index, this could mean some kind of mapping issue, and you will get an error while trying to index the data in elasticsearch.

What are you using to index the data in elasticsearch?

9

Thank you,

I see, thank you for your reply.
I am not sure what client is used to load the data. It uses AWS Lambda python. The system is managed by another person so unfortunately I do not have much details.
THe loading json looks like that:

{
  [
  {"index": {
        "_index": "mailbox-p2-2022.10",
        "_type": "P2",
        "_id": "AAA201834000003_826644600_1666615919"
    }
}
 \n
{
    "emetteur": "fxd9100n0002",
    "comment": "BiSac_lace",
    "reference": "AAA201834000003",
    "SN": "826644600",
    "operation": "092002",
    "face": "FACE2",
    "equipement": "000001",
    "programme": "E41B019FSB",
    "config": "Tog2",
    "nb_ci": "3",
    "date": 1666615919,
    "site": "fdsex",
    "Ligne": "smd0",
    "Designation": "Pc "
}
\n'
.......
]

Is there any other way to force a date type instead of default long?

10

No, the only way to force a data type to a field is using a index template.

Your index template is correct and it is expected to work, but if you are not getting any data you need to check in the source of the data, the response that the AWS Lambda is getting to see if there is any error.

You may also check in your Elasticsearch logs to see if there is any hint.

I suspect that this may be causing some issues:

"_type": "P2"

Types are deprecates in Elasticsearch 7.X and should not be used any more, but I do not know if this would cause an issue in your case.

11

Hi,

Yes, you are right!
I reviewed the reindexed indices, and saw that, when reindexed the original _type is replaced by _doc.

I need customized _type for kibana visualization lib filters.
At the same time some visualizations require date field in date format not long for average aggregation on date filed otherwise I am getting an error:

Can't get text on a VALUE_NULL at 1

Is there anyway, you think I can address this issue. The system in production and, I would eally like to avoid lots of changes?

Really appreciate your help, many thanks

12

So, you reindexed your indices after applying the template? You didn't shared this information before. The mapping is correct in the reindexed indices? If so, this means that the template is working.

You should create a custom field for it or use the tags field, types are deprecated in version 7.X and removed in version 8.X you need to move away from it if you want to have less issues when upgrading.

You may try to reindex it again, but this time specify the type destination index.

{
  "source": {
      "index": "sourceindex"
  },
  "dest": {
      "index": "destinationindex",
      "type": "destinationtype"
  }
}

Not sure if this will work, but it should according to the documentation for this version.

13

Thank you so much for your help.
Now I see it is version compatibility issue.

I tried multiple options, one was to manually reindex the available indices. The problem with this approach: I do not know the index pattern that may appear. Yes , the mapping is correct, but the new data is not written because of the _type field ( I have not yet tried your suggestions setting the _type while reindexing)

I have probably the last question, at least at this stage. I reviewed the indices that have been restored from the snapshot. They have versions between 7.0 - 7.13.

If I decide to downgrade to 6.8, will they work ? or at least back to 7.0?
The system is quite complex, was done by other person, I would like to skip rewriting all the queries, like replacing _type, setting field.keyword /s

14

Elasticsearch does not support downgrade, you would need to install an entirely new cluster in a version that has reached EOL and will not receive more updates, you also cannot restore a snapshot in a cluster with a lower version than the cluster that created the snapshot.

I think that in your case, you may be able to temporarily solve your problem using a runtime field, but be aware that this has an impact in performance.

A runtime field is a field mapping applied while reading, which means, while making queries, you may have your date field mapped as long in your index, but change it to a runtime field of the type date.

You will basically need to make the following request to your indices:

PUT index_name/
{
  "mappings": {
    "runtime": {
      "date": {
        "type": "date",
        "format": "epoch_second"
      }
    }
  }
}

And if you need to remove the runtime mapping for some reason, just use:

PUT index_name/
{
  "mappings": {
    "runtime": {
      "date": null
    }
  }
}

This may help your case, but it should not be the definitive solution, you need to change your system to not depend on the _type field.

15

Hi,
I really appreciate your help, thanks a lot.
I think I will try to remove the _type field.

The one possible issue I may face is duplicating document _id's. I understood specifying different _type values allows having multiple documents with the same _id.

I consider the option of setting different indices with patterns like : initialIndex_initialType_month and setting a custom tag so I can use filters on index pattern like initialIndex* .

Anyway, thank you so much for your help, I helped a lot, without that I would definitely be blocked without seeing light at the end of the tonnel.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.