# Elastic Search : Data Migration for Dates **URL:** \ **Category:** Elasticsearch\ **Created:** [May 31, 2018, 4:30am UTC](https://discuss.elastic.co/t/elastic-search-data-migration-for-dates/133965 "2018-05-31T04:30:07Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Nakshathri](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@Nakshathri](https://discuss.elastic.co/u/Nakshathri)\ **Post date:** [May 31, 2018, 4:30am UTC](https://discuss.elastic.co/t/elastic-search-data-migration-for-dates/133965/1 "2018-05-31T04:30:07Z")
Hello All, We have our documents indexed in Elastic search. Each of these documents has date field captured without the timezone (By default, we are capturing the date in IST timezone, without the timezone). For obvious reasons, this is not right. We are planning to capture date in UTC timezone and migrate all existing documents to capture the date in UTC timezone. After going through a couple of documents, I did not get any definitive guide to do this. Need some expert opinion on how to solve this issue. Really appreciate every opinion. Regards, Manjunath ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 31, 2018, 5:14am UTC](https://discuss.elastic.co/t/elastic-search-data-migration-for-dates/133965/2 "2018-05-31T05:14:13Z")
I wonder if you could use the reindex API with an ingest pipeline and a date processor? Would that work? ---
**Author:** ![Nakshathri](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@Nakshathri](https://discuss.elastic.co/u/Nakshathri)\ **Post date:** [June 12, 2018, 12:18pm UTC](https://discuss.elastic.co/t/elastic-search-data-migration-for-dates/133965/3 "2018-06-12T12:18:59Z")
Hello David, Thanks for that suggestion. We are able to get this working with Reindex API with ingest pipeline with date, rename, remove, script processors. ``` Request Data : { "startTime" : "2017-12-30 10:00:00", "endTime" : "2017-12-30 11:00:00", "activities" : [ { "startTime" : "2017-12-30 10:00:00", "endDtime" : "2017-12-30 10:15:00" } ] } ``` Script Processor : Used to add default timezone for the Date fields. Date Processor : Used for converting the timezone and copy the value to new field Rename and Remove Processors : For getting back the original document structure without the intermediate fields which got created during processing. But facing one challenge. We have the date field captured at the document root level and at the nested level as well. How can we use the script processor for setting the default timezone value at the nested activities level? Really appreciate your quick suggestion on this. Regards, Manjunath ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 12, 2018, 12:55pm UTC](https://discuss.elastic.co/t/elastic-search-data-migration-for-dates/133965/4 "2018-06-12T12:55:05Z")
I'm not a painless user so I don't really know. Can't you iterate on the nested objects as well? ---
**Author:** ![Nakshathri](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@Nakshathri](https://discuss.elastic.co/u/Nakshathri)\ **Post date:** [June 12, 2018, 1:20pm UTC](https://discuss.elastic.co/t/elastic-search-data-migration-for-dates/133965/5 "2018-06-12T13:20:16Z")
I can iterate over activities with foreach processor like below : ``` { "foreach": { "field": "activities", "processor": { "date": { "field": "_ingest._value.startTime", "target_field": "_ingest._value.startTime_utc", "formats": ["dd/MM/yyyy hh:mm:ss Z"], "timezone": "UTC" } } } } ``` But what I need is : ``` { "foreach": { "field": "activities", "processor": { "script": { "lang": "painless", "source": "ctx.startTimeWithOffset = ctx.startTime + params.default_offset", "params": { "default_offset": " +0530" } } } } } ``` And after this transformation apply the date processor on the modified field like below : ``` { "foreach": { "field": "activities", "processor": { "date": { "field": "_ingest._value.startTimeWithOffset", "target_field": "_ingest._value.startTime_utc", "formats": ["dd/MM/yyyy hh:mm:ss Z"], "timezone": "UTC" } } } } ``` Where is the issue : ForEach processor with the script processor is not able to iterate through the normal way and we get exceptions in ``` "source": "ctx.startTimeWithOffset = ctx.startTime + params.default_offset", ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 12, 2018, 9:06pm UTC](https://discuss.elastic.co/t/elastic-search-data-migration-for-dates/133965/6 "2018-06-12T21:06:55Z")
@Igor_Motov do you know by any chance? ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [June 13, 2018, 1:10am UTC](https://discuss.elastic.co/t/elastic-search-data-migration-for-dates/133965/7 "2018-06-13T01:10:50Z")
@Nakshathri which error are you getting? Could you post here the [simulate pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/master/simulate-pipeline-api.html) command with your test data and pipeline, so I can take a look? ---
**Author:** ![Nakshathri](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@Nakshathri](https://discuss.elastic.co/u/Nakshathri)\ **Post date:** [June 13, 2018, 4:03am UTC](https://discuss.elastic.co/t/elastic-search-data-migration-for-dates/133965/8 "2018-06-13T04:03:38Z")
@Igor_Motov here are the scenarios: 1. Able to iterate over activities.startTime; But the updated field, activityStartTimeWithOffset, is present as part of parent object, above activities. 2. If we change the script to as given below, then it gives error: Error : ``` { "error": { "root_cause": [ { "type": "script_exception", "reason": "compile error", "script_stack": [ "_ingest._value.activitySt ...", "^---- HERE" ], "script": "_ingest._value.activityStartTimeWithOffset = _ingest._value.startTime + params.default_offset", "lang": "painless", "header": { "processor_type": "foreach", "property_name": "source" } } ], "type": "script_exception", "reason": "compile error", "script_stack": [ "_ingest._value.activitySt ...", "^---- HERE" ], "script": "_ingest._value.activityStartTimeWithOffset = _ingest._value.startTime + params.default_offset", "lang": "painless", "caused_by": { "type": "illegal_argument_exception", "reason": "Variable [_ingest] is not defined." }, "header": { "processor_type": "foreach", "property_name": "source" } }, "status": 500 } ``` I think I am doing some syntax issue in the second case. But with the available documents, I was not able to get the right syntax for my need. Really appreciate your feedback. ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [June 13, 2018, 1:43pm UTC](https://discuss.elastic.co/t/elastic-search-data-migration-for-dates/133965/9 "2018-06-13T13:43:02Z")
@Nakshathri could you help me here and instead of describing the issue piece by piece post the [simulate pipeline command](https://www.elastic.co/guide/en/elasticsearch/reference/master/simulate-pipeline-api.html) with your test data and pipeline that reproduces the issue, so I can run it locally to see what's going on? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 11, 2018, 1:43pm UTC](https://discuss.elastic.co/t/elastic-search-data-migration-for-dates/133965/10 "2018-07-11T13:43:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.