Every hour text files ar ingested to update the index. I have the following code of a process that transforms the data to seconds.
"version": "10.0.0",
"create_time": 1714766696988,
"source": {
"index": [
"trx_hours_new"
],
"query": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte": "now-6h"
}
}
}
]
}
}
},
"dest": {
"index": "trx_hours_history_segundos_sp"
},
"frequency": "20m",
"sync": {
"time": {
"field": "@timestamp",
"delay": "60s"
}
},
"pivot": {
"group_by": {
"@timestamp": {
"date_histogram": {
"field": "@timestamp",
"calendar_interval": "1s",
"missing_bucket": true
}
},
"nodo": {
"terms": {
"field": "nodo"
}
},
"sp": {
"terms": {
"field": "sp"
}
}
},
"aggregations": {
"segundo_prom": {
"avg": {
"field": "tiempo"
}
},
"cant_trn_seg": {
"value_count": {
"field": "trn"
}
}
}
},
"description": "trx_hours_hist_seg_sp",
"settings": {
"max_page_search_size": 19000
}
}
1 hour of data could has 1.2 millions of documents and can take more than 1 hours to execute.
Any advice ?
Thanks in advance.