Uptades to template not being applied

Elastic Stack Elasticsearch
1

I'm getting an error when trying to post a document. The error I think is that the format I used for a date property is incorrect. I initially used YYYY-MM-DD'T'hh:mm:ssZ, but since then I have made like 10 calls to update the template, and deleted the template and created it again to have a format of yyyy-MM-dd'T'HH:mm:ssZ. But the problem is that no matter what I do to update the template, when I try to post a document, elastic is still using the initial format I used. I can see it in the error message I get.

When I make the call to get all templates, I see that the template has the right format. So why is elastic still using the original format I used? What can I do to apply the new format?

2

Hi @somatusjn Welcome to the community .

You're going to need to provide your template and a sample command that posts document to the index so we can check.

Most likely your document is not matching the index pattern in the template or there's another template that you're not aware of.

3

Did you also recreated the index with a name that matches the index_pattern?

As asked, you need to share your template and a sample of the document and also check if there is another template that could match the same index_pattern and override it.

4

Templates are applied when indices are created. Any changes made to templates will not affect existing indices where fields have already been mapped.

5

The template is too big to share, but I fetched all the templates in our cluster by calling /_template. There are only two templates that we created. The rest seem to be default templates that elastic uses. One template has pattern:

"index_patterns": [
"transcription_documents_*"
],

the other has:

"index_patterns": [
"transcription_logs_*"
],

I updated the template for "transcription_documents_*". I can see the right format when I fetched the templates. When I post documents to that index, I use:

es_url = f"{es_endpoint}/transcription_documents_{today_str}/_doc"

And I get the error that a property in the template is still using the old format I created:

Old format:

"blob_created": {
"format": "YYYY-MM-DD'T'hh:mm:ssZ",
"type": "date"
}

new format:

"blob_created": {
"format": "yyyy-MM-dd'T'HH:mm:ssZ||yyyy-MM-dd'T'HH:mm:ss'Z'",
"type": "date"
}

error:

"reason":"failed to parse date field [2022-07-06T23:32:21Z] with format [YYYY-MM-DD'T'hh:mm:ssZ]"

I'm at a loss. Any help will be greatly appreciated.

6

Has a new index been created since you updated the index template? Does the mapping for the field in this differ compared to earlier indices?

7

Actually, this is brand new cluster. There are no documents or indices with that pattern. I called:

/transcription_documents_*/_search?pretty=true

And I get:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits":
}
}

I can't input any docs because of that issue.

8

But that means the mapping is already defined!

What does this show

GET /transcription_documents_{today_str}/

looks for that "blob_created"

Changing the template does not change the mapping.. if the index is already created... which it appears to be

Also what does
GET _cat/indices/transcription_documents_*/?v

Delete the index and try again.

9

Yeah I found the issue. It looks like although the documents fail to be indexed, for whatever reason an index transcription_documents_20221006 is still created. I got that by calling /_cat/indices. So I guess any document that tries to be indexed with that index uses the format that was used for that index that failed. So I had to run DELETE /transcription_documents_20221006 and that worked.

1 Like
10

I just read what you wrote. That was the solution. Thanks!

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.