Hi Guys
We are migrating our applications from 6.8 ES cluster to 7.17.10 ES cluster . The one thing which we identified is that the negative values are not supported for date type fields( "format": "epoch_millis") . Is there any way to make it support in 7.17.10 ? because we see this PR to support this support negative epoch_millis timestamps by rkophs · Pull Request #80208 · elastic/elasticsearch · GitHub which looks like fixed in ES 8.XX versions . Can someone guide me how we can resolve this issue . we can't change the data type of this field . Also is there any way in ES to lets say disable template validation specific to one field . Please suggest which is the better way to handle this issue ?
Hi @Abhilashsr2008,
Digging through the issue you cite and the related documentation issue it looks like negative timestamps are supported as of version 8.2.0. Is there a reason you are not able to upgrade?
The other option I can think of is using the ignore_malformed attribute on your field until you can upgrade.
Thanks @carly.richmond for the comments . Yes we are planning for 8.XX upgrade soon but few other apps still need support from ES 8.XX which we are waiting to complete . Once thats done the idea is to move to latest Elastic search . This is an intermediate solution we are looking for. Also lets say we have a index in ES 6.8 where the date field has negative values and when we do the rolling upgrade to 7.17 , the old index will have some problem right since the old index had negative values and new ES 7.17 doesn't support it . or is it like the existing data in ES 6.8 will not have any issue during rolling upgrade and new data insertion to ES 7.17 with negative values for date fields will have a problem
Yes you will have a validation problem on the field with the negative values. For a temporary workaround I would have a look at ignore_malformed as listed above.
yes thanks @carly.richmond . So what am thinking , may be before the rolling upgrade , i will add ignore_malformed property to the current index template in 6.8 ES and then do the rolling upgrade to 7.17 ES. is this approach good ? so in that way during rolling upgrade i shouldn't get any issues . what do you think
I would check the data is being indexing in version 6.8 with the ignore_malformed for the fields in question as you expect and also take a snapshot first. But yes you can do a rolling upgrade between 6.8 and 7.17.10. There's details for rolling upgrades in the below sections of the documentation:
@carly.richmond have did some digging and find out that when we set ignore_malformed for the date i was able to insert negative values to my new index . But the problem comes when i make a query with that specific field . Elastic search throws exception saying that i wont be able to access this field . Is it because , when we set ignore_malformed field , ES removed this field from its indexing which caused later query failures ? can you please confirm whether my understanding is correct . Attaching the query failure
Hi @Abhilashsr2008,
That understanding does sound correct to me. If you're needing to query these fields I strongly recommend you upgrade to get the fix, where you won't need ignore_malformed.
Hope that helps!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.