# Merging transform indices to create new summary index
**URL:** https://discuss.elastic.co/t/merging-transform-indices-to-create-new-summary-index/244720
**Category:** Elasticsearch
**Created:** [August 12, 2020, 1:18pm UTC](https://discuss.elastic.co/t/merging-transform-indices-to-create-new-summary-index/244720 "2020-08-12T13:18:23Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@InfiniteDreamer](https://discuss.elastic.co/u/InfiniteDreamer)
#### Post date: [August 12, 2020, 1:18pm UTC](https://discuss.elastic.co/t/merging-transform-indices-to-create-new-summary-index/244720/1 "2020-08-12T13:18:23Z")
we want to create a summary transform index from two main index named salesdata and visitdata respectively,both indices mapping is different where CustomerCode is like primary key present in both index.
SalesData index we will aggregate below property
> `CustomerCode VolumeOfSale GrossSale`
VisitData index we will aggregate below property
> `CustomerCode LastVisitDate DayOfWeek MonthOFYear`
and transform index we want to have something like that -
> `CustomerCode VolumeOfSale GrossSale LastVisitDate DayOfWeek MonthOFYear`
we are able to create 2 separate transform index 1 for sales data and other for visit data
but how can we have 1 transform index which will be pivot on CustomerCode to have both indices fields.
---
### Author:  [@ylasri](https://discuss.elastic.co/u/ylasri)
#### Post date: [August 12, 2020, 2:57pm UTC](https://discuss.elastic.co/t/merging-transform-indices-to-create-new-summary-index/244720/2 "2020-08-12T14:57:20Z")
Did you tried to use the same index pattern for both index as input for your tranform ?
Example :
```
Sales Data are stored into indice named : customers-sales-data-YYYY-MM-DD
Visits Data are stored into indice named : customers-visits-data-YYYY-MM-DD
```
Then use index pattern `customers-*` as data source of your tranform
---
### Author:  [@InfiniteDreamer](https://discuss.elastic.co/u/InfiniteDreamer)
#### Post date: [August 17, 2020, 7:02pm UTC](https://discuss.elastic.co/t/merging-transform-indices-to-create-new-summary-index/244720/3 "2020-08-17T19:02:53Z")
Thanks @ylasri,
The above mentioned logic works for the scenario, but the transform job fails frequently,as its not even crossing 5% of actual index.
Following Logs FYI, is there any we can solve this,kindly help.
> Experienced at least [6] bulk index failures. See the logs of the node running the transform for details. failure in bulk execution: [324]: index [nx003], type [\_doc], id [MjExM2ByzyKF5G5Vz2MDoQzSAAAAAAAA], message [MapperParsingException[failed to parse field [last\_sales\_date.max] of type [date] in document with id 'MjExM2ByzyKF5G5Vz2MDoQzSAAAAAAAA'. Preview of field's value: '1.5967386E12']; nested: IllegalArgumentException[failed to parse date field [1.5967386E12] with format [strict\_date\_optional\_time||epoch\_millis]]; nested: NotSerializableExceptionWrapper[date\_time\_parse\_exception: Failed to parse with all enclosed parsers];] [325]: index [nx003], type [\_doc], id [MjFcImrsR4hufG6Qx4ic-CWcAAAAAAAA], message [MapperParsingException[failed to parse field [last\_sales\_date.max] of type [date] in document with id 'MjFcImrsR4hufG6Qx4ic-CWcAAAAAAAA'. Preview of field's value: '1.5965658E12']; nested: IllegalArgumentException[failed to parse date field [1.5965658E12] with format [strict\_date\_optional\_time||epoch\_millis]]; nested: NotSerializableExceptionWrapper[date\_time\_parse\_exception: Failed to parse with all enclosed parsers];] [326]: index [nx003], type [\_doc], id [MjGmFq44hrnEe6LOpf8rSQ2OAAAAAAAA], message [MapperParsingException[failed to parse field [last\_sales\_date.max] of type [date] in document with id 'MjGmFq44hrnEe6LOpf8rSQ2OAAAAAAAA'. Preview of field's value: '1.5967386E12']; nested: IllegalArgumentException[failed to parse date field [1.5967386E12] with format [strict\_date\_optional\_time||epoch\_millis]]; nested: NotSerializableExceptionWrapper[date\_time\_parse\_exception: Failed to parse with all enclosed parsers];] [327]: index [nx003], type [\_doc], id [MjE\_sD-JyLTwlV472S1yc2UBAAAAAAAA], message [MapperParsingException[failed to parse field [last\_sales\_date.max] of type [date] in document with id 'MjE\_sD-JyLTwlV472S1yc2UBAAAAAAAA'. Preview of field's value: '1.5967386E12']; nested: IllegalArgumentException[failed to parse date field [1.5967386E12] with format [strict\_date\_optional\_time||epoch\_millis]]; nested: NotSerializableExceptionWrapper[date\_time\_parse\_exception: Failed to parse with all enclosed parsers];] [328]: index [nx003], type [\_doc], id [MjHHULqBwpCaq3ltMGeiOhCLAAAAAAAA], message [MapperParsingException[failed to parse field [last\_sales\_date.max] of type [date] in document with id 'MjHHULqBwpCaq3ltMGeiOhCLAAAAAAAA'. Preview of field's value: '1.5965658E12']; nested: IllegalArgumentException[failed to parse date field [1.5965658E12] with format [strict\_date\_optional\_time||epoch\_millis]]; nested: NotSerializableExceptionWrapper[date\_time\_parse\_exception: Failed to parse with all enclosed parsers];] [348]: index [nx003], type [\_doc], id [MjHXWyFsOPzBbfUxBwUcKGzpAAAAAAAA], message [MapperParsingException[failed to parse field [last\_sales\_date.max] of type [date] in document with id 'MjHXWyFsOPzBbfUxBwUcKGzpAAAAAAAA'. Preview of field's value: '1.5965658E12']; nested: IllegalArgumentException[failed to parse date field [1.5965658E12] with format [strict\_date\_optional\_time||epoch\_millis]]; nested: NotSerializableExceptionWrapper[date\_time\_parse\_exception: Failed to parse with all enclosed parsers];]
> Transform encountered an exception: Bulk index experienced failures. See the logs of the node running the transform for details. Will attempt again at next scheduled trigger.
>
> task encountered more than 10 failures; latest failure: Bulk index experienced failures. See the logs of the node running the transform for details.
---
### Author:  [@ylasri](https://discuss.elastic.co/u/ylasri)
#### Post date: [August 17, 2020, 7:46pm UTC](https://discuss.elastic.co/t/merging-transform-indices-to-create-new-summary-index/244720/4 "2020-08-17T19:46:23Z")
Check the mapping for field `last_sales_date.max`
`index [nx003], type [_doc], id [MjExM2ByzyKF5G5Vz2MDoQzSAAAAAAAA], message [MapperParsingException[failed to parse field [last_sales_date.max] of type [date] in document with id 'MjExM2ByzyKF5G5Vz2MDoQzSAAAAAAAA'. Preview of field's value: '1.5967386E12']; nested: IllegalArgumentException[failed to parse date field [1.5967386E12] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];]`
---
### Author:  [@InfiniteDreamer](https://discuss.elastic.co/u/InfiniteDreamer)
#### Post date: [August 18, 2020, 8:20am UTC](https://discuss.elastic.co/t/merging-transform-indices-to-create-new-summary-index/244720/5 "2020-08-18T08:20:32Z")
Hi @ylasri,
Here we are creating transform index from input of 2 Transformed indices, one of the input transform index `customer-visits-data` have scripted metric to create a new field `last_sales_date.max` with some business logic and that index have expected data.
Also previously mentioned transform index job fails after 3% of job running successfully.
3% generated data is accurate compared with input after that we are getting that error message.
Your thoughts on the same please.
---
### Author:  [@ylasri](https://discuss.elastic.co/u/ylasri)
#### Post date: [August 18, 2020, 1:39pm UTC](https://discuss.elastic.co/t/merging-transform-indices-to-create-new-summary-index/244720/6 "2020-08-18T13:39:36Z")
I guess your index `customer-visits-data` has the following mapping for date
```
"last_sales_date": {
"max": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
...
```
But a value of 1.5967386E12 will not be parsed to date
The second tranform will create an index with automtic mapping, and the first value will fix the mapping, and later if a new value come with différent type will cause issues ....
Can you provide the mapping for `customer-visits-data` and an example of doc
```
GET customer-visits-data/_mapping
GET customer-visits-data/_doc/MjExM2ByzyKF5G5Vz2MDoQzSAAAAAAAA
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 15, 2020, 1:39pm UTC](https://discuss.elastic.co/t/merging-transform-indices-to-create-new-summary-index/244720/7 "2020-09-15T13:39:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.