We are migrating our applications from 6.8 ES cluster to 7.17.10 ES cluster . The one thing which we identified is that the negative values are not supported for date type fields( "format": "epoch_millis") . Is there any way to make it support in 7.17.10 ? because we see this PR to support this support negative epoch_millis timestamps by rkophs · Pull Request #80208 · elastic/elasticsearch · GitHub which looks like fixed in ES 8.XX versions . Can someone guide me how we can resolve this issue . we can't change the data type of this field . Also is there any way in ES to lets say disable template validation specific to one field . Please suggest which is the better way to handle this issue ?
Digging through the issue you cite and the related documentation issue it looks like negative timestamps are supported as of version 8.2.0. Is there a reason you are not able to upgrade?
The other option I can think of is using the
ignore_malformed attribute on your field until you can upgrade.
Thanks @carly.richmond for the comments . Yes we are planning for 8.XX upgrade soon but few other apps still need support from ES 8.XX which we are waiting to complete . Once thats done the idea is to move to latest Elastic search . This is an intermediate solution we are looking for. Also lets say we have a index in ES 6.8 where the date field has negative values and when we do the rolling upgrade to 7.17 , the old index will have some problem right since the old index had negative values and new ES 7.17 doesn't support it . or is it like the existing data in ES 6.8 will not have any issue during rolling upgrade and new data insertion to ES 7.17 with negative values for date fields will have a problem
Yes you will have a validation problem on the field with the negative values. For a temporary workaround I would have a look at
ignore_malformed as listed above.
yes thanks @carly.richmond . So what am thinking , may be before the rolling upgrade , i will add
ignore_malformed property to the current index template in 6.8 ES and then do the rolling upgrade to 7.17 ES. is this approach good ? so in that way during rolling upgrade i shouldn't get any issues . what do you think
I would check the data is being indexing in version 6.8 with the
ignore_malformed for the fields in question as you expect and also take a snapshot first. But yes you can do a rolling upgrade between 6.8 and 7.17.10. There's details for rolling upgrades in the below sections of the documentation:
@carly.richmond have did some digging and find out that when we set ignore_malformed for the date i was able to insert negative values to my new index . But the problem comes when i make a query with that specific field . Elastic search throws exception saying that i wont be able to access this field . Is it because , when we set ignore_malformed field , ES removed this field from its indexing which caused later query failures ? can you please confirm whether my understanding is correct . Attaching the query failure
That understanding does sound correct to me. If you're needing to query these fields I strongly recommend you upgrade to get the fix, where you won't need
Hope that helps!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.