I use transformation for data preprocessing, in my case every second rps are created for each host
{
"id": "appgw-rps-host",
"source": {
"index": [
"logs-azure.platformlogs-default"
]
},
"dest": {
"index": "appgw-rps-host"
},
"frequency": "10m",
"sync": {
"time": {
"field": "@timestamp",
"delay": "5m"
}
},
"pivot": {
"group_by": {
"@timestamp": {
"date_histogram": {
"field": "@timestamp",
"calendar_interval": "1s"
}
}
},
"aggregations": {
"host": {
"terms": {
"field": "event.host"
}
}
}
}
}
everything works perfectly and I get the data I need. event example:
{
"_index": "appgw-rps-host",
"_id": "ADUZ-GZffUzMkUJXMRaP3wgAAAAAAAAA",
"_version": 1,
"_score": 1,
"_source": {
"@timestamp": "2022-07-12T09:57:41.000Z",
"host": {
"host1.com": 2,
"host2.com": 4,
"host3.com": 42
}
}
}
but this "host" type is flatten
{
"mappings": {
"_meta": {
"created_by": "transform",
},
"properties": {
"@timestamp": {
"type": "date"
},
"host": {
"type": "flattened"
}
}
}
}
and I can't work with them. For example, I can’t make a graph in kibana or perform any aggregation like with a numeric value. Maybe there is some possibility not to do the flattened type or do some convert using ingest pipeline?
Best Regards,
Dmitri