Hello,
I've been trying to get geoip working on field with dot notation without much luck.
EDIT: So this is happening even when not using nested fields. Same error either way. I'm using Fluentd so maybe the way it outputs the data is somehow not jiving with ES?
The field is coming in as "source.ip" as a single field. Since this won't work with the geoip processor, I reindexed a test index using dot expander so the source.ip field shows up like below in the _doc:
{
"_index": "test.infr.event.network-paloalto-expand",
"_source": {
"destination.port": "0",
"source": {
"ip": "11.111.111"
},
"destination.ip": "111.11.111",
"network.transport": "udp",
"event.action": "allow",
"@timestamp": "2020-01-28T23:43:29.000000000+00:00",
"source.port": "53",
"host.name": "hostname"
},
"fields": {
"@timestamp": [
"2020-01-28T23:43:29.000Z"
]
},
"sort": [
1580255009000
]
}
I then tried to run these dot expanded documents through a geoip processor to no luck. It's still giving me the field [source] not present as part of path [source.ip]" error.
Here is my processor:
"sourcegeoip" : {
"description" : "Add geoip info",
"processors" : [
{
"geoip" : {
"field" : "source.ip"
}
}
]
}
And this is the reindex I ran:
POST _reindex
{
"source": {
"index": "test.infr.event.network-paloalto-expand"
},
"dest": {
"index": "test.infr.event.network-paloalto-expand-geo",
"pipeline" : "sourcegeoip"
}
}
It is properly being recognized as an IP with this in the index template:
"source": {
"properties": {
"address": {
"type": "keyword",
"ignore_above": 1024
},
...
},
"ip": {
"type": "ip"
I was under the impression that this would work similar to nested conditionals, but that doesn't seem to be the case.
Is what I am trying just not possible? In that case, is my only option to clone the field and create the geoip from there?
edit:
It works fine when I manually enter a document using
PUT test.infr.event.network-paloalto-expand/_doc/my_id?pipeline=sourcegeoip
{
"source" : {
"ip": "8.8.8.8"
}
}