I have a mapping that dynamically detects date and stores. Fields are dynamically created, and I don't have any configuration about date formatting. so dates are detected and stored as their formats. They are generally stored as strict_date_optional_time
However, If I receive date with different format than default, such as yyyy/MM/dd, strict_date_optional_time is not set as the format of dynamically created field. In fact it is stored as yyyy/MM/dd. I can then get that field and apply some filters on it.
My problem is that I have a general filtering that creates date query with the format of yyyy-MM-dd. In this case, fields that have different formats (yyyy/MM/dd in our example) are filtered out.
Adding strict_date_optional_time to all possible date fields. Any future field can be queried with default format yyyy-MM-dd. Regardless of the name of field, if I can handle this, it will be amazing for me.
Query date columns regardless of their mapping. Because when I send query above, docs in index-002 will throw the following error. It may also be okay if I solve this error somehow.
failed to parse date field [2021-01-01] with format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis]: [failed to parse date field [2021-01-01] with format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis]]
I want to be able to query dates with yyyy-MM-dd format. Most of the dates we store is already in the yyyy-MM-dd (strict_date_optional_time) format. But we may have some cases that the date field is formatted with some other types.
ff your requirement / expectation is to only define those two formats in the dynamic template (which I don't understand why because you could be much more flexible by adding multiple patterns which many do for just this case) then you will need to match / convert those unmatched formats into one of the matched formats when you query there's no automatic way to do that As far as I know.
Or to put in a more simple way
You can only search by the formats that you allow to store the date.
For Example, So if you only accept the 2 formats to store the date you can only search by those formats because it uses the same analyzer when it stores the date as when it searches the date.
That does not mean that you have to search by the exact format that you originally stored the date just by one of the formats that you store the date.
So suppose you store the date with three different formats supported and you used format one when you stored the date You could use any of the three formats to search the date but you can not search with a format that is not one of the three defined formats.
I think I could not express my need. I do not want to restrict my formats. Alongside date formats can be any of them, I want all my date fields should contain a date format that I can manage. Because in my automated query, I want to be able to use specific date formatting. So I may want to have this:
PUT /index-001
POST /index-001
{
"field1": "2021/01/01"
}
PUT /index-002
POST /index-002
{
"field1": "2021-01-01"
}
PUT /index-003
POST /index-003
{
"field1": 1609462800
}
PUT /index-004
POST /index-004
{
"field1": "20210101"
}
And at the end, I need the following query will work for all indices.
To make it clear, I can use any date format in my query. If I can manage all date fields to contain one certain format, I can use 2021-01-01 as 2021/01/01 or 20210101 or 1609462800 or anything else.
You are responding great, thank you for that. And there is no need for any kind of apology. In fact, I am sorry for making my explanations hard to observe.
This solution may make most of my work, yes. However:
I have one big problem: field name is not known at the beginning. it can be my_date_fields or some_other_field_name or etc.
I don't want to restrict date formats:
will bound my possible date fields. I only wish to have strict_date_optional_time in all of my date fields.
Hi @Mehmet_Gencol I suggest you read the docs carefully.
Then you will need to only pass in that... and search that
I think you do not perhaps understand how elastisearch stores dates... internally all dates are stored exactly the same this all about being ables to pass in and search the dates with different formats no matter What Format you Pass In ... the date is ALWAYS stored the same in elastic!
Internally, dates are converted to UTC (if the time-zone is specified) and stored as a long number representing milliseconds-since-the-epoch.
If you do not know the name of the fields you will need to use Dynamic Date Detection as you stated Which is discussed here ...
and what I showed above hereBUT there is an issue with that
The issue with that is the long millis that will be detected as a number not a date.... that is why an explicit mapping will work better and you generally will not want to turn of numeric date detection.
If you know your date fields have a naming convention you can try dynamic_templates and match on a name
You have lots of options...I am sure you will figure one out that works
Actually I read all and could not find a way to solve my problem. To repeat, without restricting date formats by defining dynamic_date_format or some other ways, I want all my date fields to have strict_date_optional_time formatting. So that I can query them in the yyyy-MM-dd format. I will go with this solution:
PUT index-001
{
"mappings": {
# Maybe some other types to. I should consider all possible date formats here.
"dynamic_date_formats": ["yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||strict_date_optional_time||epoch_millis||epoch_second"]
}
}
This can be a solution but not my expectation. Thank you for your help anyway.
There is no magic.. if you want ALL your dates to be strict_date_optional_time then you / your app will need to format / convert them all to strict_date_optional_time before inserting and searching you can not have it both ways.
Your requirement can not be met as you write it as far as I know, perhaps someone else can help.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.