Hi,
This is a weird problem I am facing. I am using the IIS logs input for filebeat, which creates an ingest-pipeline in ES, thus I can see that the data leaving filebeat is an unmodified string.
I created some add_fields processors in filebeat, which are properly detected at the startup.
Since there is no cidrMatch or network processor for painless available, I cannot rewrite the IIS ingest pipeline to append my private geolocation.
In my head the current flow is the following:
- Filebeat read line from file
- filebeat does not touch that line and sends it as a string to ES using the IIS ingest_pipeline
- ES does index pipeline magic
- data is properly indexed
If I add pipeline: xyz in the output.elasticsearch config, is this pipeline skipped? How do you chain pipelines using the default filebeat settings for such modules?
Possible solutions:
-
Modify the maxmind geoip database. ==> I do not like that idea...
-
Use some painless magic ==> No idea how to properly solve a CIDR mapping, without writing a lot of things by hand. E.g. I would have to create a mapping for /8, /16, /... to resolve to
192.168.0.0/16to192.168.0.0 - 192.168.255.255 -
Use logstash ==> Not possible would mean I have to port the IIS ingest pipeline to logstash.
-
Rewrite the IIS ingest pipeline to a local beats javascript processor ==> well, could be done, but has to be tested every time the release notes mention something about processors and IIS module.
All the relevant posts linked in this "mega" thread Creating geoip data for internal networks do not really apply to my use case, as I have to deal with an ES ingest pipeline.
For the sake of it:
Filebeat, ES, Kibana, everything basically version 7.8.0
startup log
2020-06-22T12:57:11.210+0200 DEBUG [conditions] conditions/conditions.go:98 New condition network:(source.ip:192.168.0.0/16)
2020-06-22T12:57:11.210+0200 DEBUG [conditions] conditions/conditions.go:98 New condition network:(destination.ip:192.168.0.0/16)
filebeat processor
- add_fields:
when.network.source.ip: 192.168.0.0/16
fields:
source.geo.location:
lat: 47.060658
lon: 15.447569
target: ''
- add_fields:
when.network.destination.ip: 192.168.0.0/16
fields:
source.geo.location:
lat: 47.060658
lon: 15.447569
target: ''
debug json from filebeat
2020-06-22T12:33:21.436+0200 DEBUG [processors] processing/processors.go:187 Publish event: {
"@timestamp": "2020-06-22T10:33:21.436Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.8.0",
"pipeline": "filebeat-7.8.0-iis-access-pipeline"
},
"message": "2020-06-22 06:00:34 POST /mapi/emsmdb/ MailboxId=9d3364e3-7765-4682-1234-b5c24462f997@domain.com&CorrelationID=<empty>;&ClientId=EMIV9JOYUOLCPPCRXXHD&ClientRequestInfo=R:{F3E4EFAE-AC36-4B46-ABCD-32A3940FE8E1}:72;CI:{34FE85E5-DBC8-4D89-ABCD-BC5E4AE4C473}:18;RT:NotificationWait&cafeReqId=bd7f6780-abcd-41da-8178-a5b103bfaaa5; domain\\pkahr 192.168.1.1 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.4266;+Pro) ClientId=EMIV9JOYUOLCPPCRABCD;MapiContext=MAPIAAAAAPaj+87+yf3e7Nzu3vPDahgzrqjruqtNnBwAAAAAA;MapiSequence=34-nBbhJA==;X-BackEndCookie=9d3364e3-abcd-ddef-aaaa-b5c24462f997=u56Lnp2ej/SnZ6ZyNLLyp7M0p3I0s3Nq8/KxcvHxczH 200",
"input": {
"type": "log"
},
"fileset": {
"name": "access"
},
"agent": {
"version": "7.8.0",
"hostname": "my-iss-server",
"ephemeral_id": "3f685b87-abcd-4de3-81ce-adec2d8c4292",
"id": "fba4df89-abcd-41e6-b776-7fbb7a760a1f",
"name": "my-iss-server",
"type": "filebeat"
},
"log": {
"offset": 910469,
"file": {
"path": "C:\\inetpub\\logs\\LogFiles\\1\\u_ex20062206.log"
}
},
"tags": [
"exchange"
],
"service": {
"type": "iis"
},
"event": {
"module": "iis",
"dataset": "iis.access"
},
"ecs": {
"version": "1.5.0"
}
}
raw document
{
"_index": "filebeat-7.8.0-2020.06.19-000001",
"_type": "_doc",
"_id": "Ff-t23IBaPYAommVthG0",
"_version": 1,
"_score": null,
"_source": {
"agent": {
"version": "7.8.0",
"hostname": "my-iss-server",
"ephemeral_id": "3f685b87-abcd-4de3-81ce-adec2d8c4292",
"id": "fba4df89-abcd-41e6-b776-7fbb7a760a1f",
"name": "my-iss-server",
"type": "filebeat"
},
"temp": {},
"log": {
"file": {
"path": "C:\\inetpub\\logs\\LogFiles\\W3SVC2\\u_ex20062210.log"
},
"offset": 564178678
},
"destination": {
"address": "192.168.1.100",
"port": 444,
"ip": "192.168.1.100"
},
"source": {
"address": "192.168.1.1",
"ip": "192.168.1.1"
},
"fileset": {
"name": "access"
},
"url": {
"path": "/mapi/emsmdb/",
"query": "MailboxId=9d3364e3-7765-4682-1234-b5c24462f997@domain.com&CorrelationID=<empty>;&ClientId=EMIV9JOYUOLCPPCRXXHD&ClientRequestInfo=R:{F3E4EFAE-AC36-4B46-ABCD-32A3940FE8E1}:72;CI:{34FE85E5-DBC8-4D89-ABCD-BC5E4AE4C473}:18;RT:NotificationWait&cafeReqId=bd7f6780-abcd-41da-8178-a5b103bfaaa5; domain\\pkahr 192.168.1.1 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.4266;+Pro) ClientId=EMIV9JOYUOLCPPCRABCD;MapiContext=MAPIAAAAAPaj+87+yf3e7Nzu3vPDahgzrqjruqtNnBwAAAAAA;MapiSequence=34-nBbhJA==;X-BackEndCookie=9d3364e3-abcd-ddef-aaaa-b5c24462f997=u56Lnp2ej/SnZ6ZyNLLyp7M0p3I0s3Nq8/KxcvHxczH"
},
"tags": ["exchange"],
"input": {
"type": "log"
},
"iis": {
"access": {
"sub_status": 0,
"win32_status": 0
}
},
"@timestamp": "2020-06-22T10:58:14.000Z",
"ecs": {
"version": "1.5.0"
},
"related": {
"user": ["domain\\\\pkahr"]
},
"service": {
"type": "iis"
},
"http": {
"request": {
"method": "POST"
},
"response": {
"status_code": 200
}
},
"user": {
"name": "domain\\pkahr"
},
"user_agent": {
"original": "Microsoft+Outlook+16.0.4266",
"os": {
"name": "Microsoft Outlook",
"version": "16.0.4266",
"full": "Microsoft Outlook 16.0.4266"
}
}
}
}