Hello guys, I have an index with a lot of records, like these:
"_source": {
"terminal_number": " 123456",
"date": "2023-10-18 12:02:31.676",
"iin": " 111111111 ",
"service_type": "o.t.s.transactions.trm.TerminalService",
"log_level": "INFO",
"microservice": "microservices-transaction",
"iin_data": "transaction-service,7ec6ea6dyc1e3f11,7ec6eq6ddc1e3f11",
"@timestamp": "2023-10-18T08:10:34.267Z"
}
and other record:
"_source": {
"date": "2023-10-18 12:14:31.163",
"role": " ROLE_PSP",
"service_type": "o.t.s.t.security.TokenService",
"log_level": "INFO",
"microservice": "microservices-transaction-svc",
"iin_data": "transaction-service,7ec6ea6dyc1e3f11,7ec6eq6ddc1e3f11",
"@timestamp": "2023-10-18T08:987:34.266Z"
}
iin_data in some of records maybe same,
now I want to find same of iin_data and aggregate them in an index
Can anyone help?
Hi @reza_sabz,
Welcome back! Have you had a look at transforms to see if this can give you the summarized index that you need? It's not a feature I've played with much but might be worth a look.
Let us know how you get on!
Thank you for reply 
I tried to solve my issue with enrich policy, for example:
I have two index:
index_1 :
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "index-1-2023.10.22",
"_id": "BRxNV4sB4G6JLjkwUiLj",
"_score": 1,
"_source": {
"terminal_number": "12245657",
"iin": " 111111123 ",
"microservice": "microservices-transaction-svc",
"date": "2023-10-22 15:42:45.136",
"@timestamp": "2023-10-22T12:12:45.417Z",
"iin_data": "a1552135cd3d16db",
"log_level": "INFO",
"service_type": "o.t.s.transactions.trm.TerminalService"
}
},
{
"_index": "index-1-2023.10.22",
"_id": "U_NNV4sB-5gOz7rQUrHj",
"_score": 1,
"_source": {
"terminal_number": "12245657",
"iin": " 111111123 ",
"microservice": "microservices-transaction-svc",
"date": "2023-10-22 15:42:44.646",
"@timestamp": "2023-10-22T12:12:45.417Z",
"iin_data": "8fc72d10c3952962",
"log_level": "INFO",
"service_type": "o.t.s.transactions.trm.TerminalService"
}
},
{
"_index": "index-1-2023.10.22",
"_id": "BxxNV4sB4G6JLjkwWCKQ",
"_score": 1,
"_source": {
"terminal_number": "12245657",
"iin": " 111111123 ",
"microservice": "microservices-transaction-svc",
"date": "2023-10-22 15:42:45.672",
"@timestamp": "2023-10-22T12:12:47.414Z",
"iin_data": "1d1f7bb4e4fd7f28",
"log_level": "INFO",
"service_type": "o.t.s.transactions.trm.TerminalService"
}
}
]
}
}
index-2 :
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "index-2-2023.10.22",
"_id": "UvNNV4sB-5gOz7rQUrGp",
"_score": 1,
"_source": {
"microservice": "microservices-transaction-svc",
"date": "2023-10-22 15:42:44.634",
"@timestamp": "2023-10-22T12:12:45.416Z",
"uri_query": "/itemorder",
"iin_data": "8fc72d10c3952962",
"log_level": "INFO"
}
}
]
}
}
my enrich policy:
PUT _enrich/policy/merge
{
"match": {
"indices": "index-2-2023.10.22",
"match_field": "iin_data",
"enrich_fields": ["uri_query"]
}
}
POST /_enrich/policy/merge/_execute
PUT _ingest/pipeline/enrich
{
"processors": [
{
"enrich": {
"description": "Add 'uri_query' data based on 'iin_data'",
"policy_name": "merge",
"field": "iin_data",
"target_field": "new",
"max_matches": "1"
}
}
]
}
POST _reindex
{
"source": {
"index": "index-1-2023.10.22"
},
"dest": {
"index": "merge",
"pipeline": "enrich"
}
}
But it doesn't work.
I would be interested to hear what you mean by it doesn't work. But enrich policies are intended to enrich documents by adding fields from another index based on a particular field match rather than against the same index, so I would expect some interesting results when trying to enrich against the same index.
At first I tried to collect some of their records as well, but I didn't get good results. Because of this, I made 2 indexes and tried to collect them
1 Like
Ok, did splitting into two indices work for you in the end?
No, this is the result:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "merge",
"_id": "BRxNV4sB4G6JLjkwUiLj",
"_score": 1,
"_source": {
"date": "2023-10-22 15:42:45.136",
"service_type": "o.t.s.transactions.trm.TerminalService",
"@timestamp": "2023-10-22T12:12:45.417Z",
"iin_data": "a1552135cd3d16db",
"terminal_number": "12245657",
"microservice": "microservices-transaction-svc",
"log_level": "INFO",
"iin": " 111111123 "
}
},
{
"_index": "merge",
"_id": "U_NNV4sB-5gOz7rQUrHj",
"_score": 1,
"_source": {
"date": "2023-10-22 15:42:44.646",
"service_type": "o.t.s.transactions.trm.TerminalService",
"@timestamp": "2023-10-22T12:12:45.417Z",
"iin_data": "8fc72d10c3952962",
"terminal_number": "12245657",
"microservice": "microservices-transaction-svc",
"log_level": "INFO",
"iin": " 111111123 "
}
},
{
"_index": "merge",
"_id": "BxxNV4sB4G6JLjkwWCKQ",
"_score": 1,
"_source": {
"date": "2023-10-22 15:42:45.672",
"service_type": "o.t.s.transactions.trm.TerminalService",
"@timestamp": "2023-10-22T12:12:47.414Z",
"iin_data": "1d1f7bb4e4fd7f28",
"terminal_number": "12245657",
"microservice": "microservices-transaction-svc",
"log_level": "INFO",
"iin": " 111111123 "
}
}
]
}
}
It returned the same records to me!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.