Merging transform indices to create new summary index

Elastic Stack Elasticsearch
1

we want to create a summary transform index from two main index named salesdata and visitdata respectively,both indices mapping is different where CustomerCode is like primary key present in both index.

SalesData index we will aggregate below property

CustomerCode VolumeOfSale GrossSale

VisitData index we will aggregate below property

CustomerCode LastVisitDate DayOfWeek MonthOFYear

and transform index we want to have something like that -

CustomerCode VolumeOfSale GrossSale LastVisitDate DayOfWeek MonthOFYear

we are able to create 2 separate transform index 1 for sales data and other for visit data
but how can we have 1 transform index which will be pivot on CustomerCode to have both indices fields.

2

Did you tried to use the same index pattern for both index as input for your tranform ?
Example :

Sales Data are stored into indice named : customers-sales-data-YYYY-MM-DD
Visits Data are stored into indice named : customers-visits-data-YYYY-MM-DD

Then use index pattern customers-* as data source of your tranform

3

Thanks @ylasri,
The above mentioned logic works for the scenario, but the transform job fails frequently,as its not even crossing 5% of actual index.

Following Logs FYI, is there any we can solve this,kindly help.

Experienced at least [6] bulk index failures. See the logs of the node running the transform for details. failure in bulk execution: [324]: index [nx003], type [_doc], id [MjExM2ByzyKF5G5Vz2MDoQzSAAAAAAAA], message [MapperParsingException[failed to parse field [last_sales_date.max] of type [date] in document with id 'MjExM2ByzyKF5G5Vz2MDoQzSAAAAAAAA'. Preview of field's value: '1.5967386E12']; nested: IllegalArgumentException[failed to parse date field [1.5967386E12] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];] [325]: index [nx003], type [_doc], id [MjFcImrsR4hufG6Qx4ic-CWcAAAAAAAA], message [MapperParsingException[failed to parse field [last_sales_date.max] of type [date] in document with id 'MjFcImrsR4hufG6Qx4ic-CWcAAAAAAAA'. Preview of field's value: '1.5965658E12']; nested: IllegalArgumentException[failed to parse date field [1.5965658E12] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];] [326]: index [nx003], type [_doc], id [MjGmFq44hrnEe6LOpf8rSQ2OAAAAAAAA], message [MapperParsingException[failed to parse field [last_sales_date.max] of type [date] in document with id 'MjGmFq44hrnEe6LOpf8rSQ2OAAAAAAAA'. Preview of field's value: '1.5967386E12']; nested: IllegalArgumentException[failed to parse date field [1.5967386E12] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];] [327]: index [nx003], type [_doc], id [MjE_sD-JyLTwlV472S1yc2UBAAAAAAAA], message [MapperParsingException[failed to parse field [last_sales_date.max] of type [date] in document with id 'MjE_sD-JyLTwlV472S1yc2UBAAAAAAAA'. Preview of field's value: '1.5967386E12']; nested: IllegalArgumentException[failed to parse date field [1.5967386E12] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];] [328]: index [nx003], type [_doc], id [MjHHULqBwpCaq3ltMGeiOhCLAAAAAAAA], message [MapperParsingException[failed to parse field [last_sales_date.max] of type [date] in document with id 'MjHHULqBwpCaq3ltMGeiOhCLAAAAAAAA'. Preview of field's value: '1.5965658E12']; nested: IllegalArgumentException[failed to parse date field [1.5965658E12] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];] [348]: index [nx003], type [_doc], id [MjHXWyFsOPzBbfUxBwUcKGzpAAAAAAAA], message [MapperParsingException[failed to parse field [last_sales_date.max] of type [date] in document with id 'MjHXWyFsOPzBbfUxBwUcKGzpAAAAAAAA'. Preview of field's value: '1.5965658E12']; nested: IllegalArgumentException[failed to parse date field [1.5965658E12] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];]

Transform encountered an exception: Bulk index experienced failures. See the logs of the node running the transform for details. Will attempt again at next scheduled trigger.

task encountered more than 10 failures; latest failure: Bulk index experienced failures. See the logs of the node running the transform for details.

4

Check the mapping for field last_sales_date.max

index [nx003], type [_doc], id [MjExM2ByzyKF5G5Vz2MDoQzSAAAAAAAA], message [MapperParsingException[failed to parse field [last_sales_date.max] of type [date] in document with id 'MjExM2ByzyKF5G5Vz2MDoQzSAAAAAAAA'. Preview of field's value: '1.5967386E12']; nested: IllegalArgumentException[failed to parse date field [1.5967386E12] with format [strict_date_optional_time||epoch_millis]]; nested: NotSerializableExceptionWrapper[date_time_parse_exception: Failed to parse with all enclosed parsers];]

5

Hi @ylasri,

Here we are creating transform index from input of 2 Transformed indices, one of the input transform index customer-visits-data have scripted metric to create a new field last_sales_date.max with some business logic and that index have expected data.

Also previously mentioned transform index job fails after 3% of job running successfully.
3% generated data is accurate compared with input after that we are getting that error message.

Your thoughts on the same please.

6

I guess your index customer-visits-data has the following mapping for date

"last_sales_date": {
"max": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
...

But a value of 1.5967386E12 will not be parsed to date

The second tranform will create an index with automtic mapping, and the first value will fix the mapping, and later if a new value come with différent type will cause issues ....

Can you provide the mapping for customer-visits-data and an example of doc

GET customer-visits-data/_mapping
GET customer-visits-data/_doc/MjExM2ByzyKF5G5Vz2MDoQzSAAAAAAAA
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.