I have successfully configured filebeats to ingest iis logfiles from a clients server to my elasticsearch. I can view the entries through discovery, but every message has the following error:
Provided Grok expressions do not match field value: [
2020-07-26 10:31:49 W3SVC6 SRV-XXX 99.99.99.99 GET /api/List Usuario=xxx.xx@xx.com&Senha=password&Placa=UUU9999 80 - 88.88.88.88 Mozilla/3.0+(compatible;+Indy+Library) - 200 0 0 2244 246 1218]"
My pipeline.yml for iis ingestion is:
description: Pipeline for parsing IIS access logs. Requires the geoip and user_agent
plugins.
processors:
- grok:
field: message
patterns:
- '%{TIMESTAMP_ISO8601:iis.access.time} (?:-|%{IPORHOST:destination.address}) (?:-|%{WORD:http.request.method})
(?:-|%{NOTSPACE:url.path}) (?:-|%{NOTSPACE:url.query}) (?:-|%{NUMBER:destination.port:long}) (?:-|%{NOTSPACE:user.name})
(?:-|%{IPORHOST:source.address}) (?:-|%{NOTSPACE:user_agent.original}) (?:-|%{NOTSPACE:http.request.referrer})
(?:-|%{NUMBER:http.response.status_code:long}) (?:-|%{NUMBER:iis.access.sub_status:long})
(?:-|%{NUMBER:iis.access.win32_status:long}) (?:-|%{NUMBER:temp.duration:long})'
- '%{TIMESTAMP_ISO8601:iis.access.time} (?:-|%{NOTSPACE:iis.access.site_name}) (?:-|%{WORD:http.request.method})
(?:-|%{NOTSPACE:url.path}) (?:-|%{NOTSPACE:url.query}) (?:-|%{NUMBER:destination.port:long}) (?:-|%{NOTSPACE:user.name})
(?:-|%{IPORHOST:source.address}) (?:-|%{NOTSPACE:user_agent.original}) (?:-|%{NOTSPACE:iis.access.cookie})
(?:-|%{NOTSPACE:http.request.referrer}) (?:-|%{NOTSPACE:destination.domain}) (?:-|%{NUMBER:http.response.status_code:long})
(?:-|%{NUMBER:iis.access.sub_status:long}) (?:-|%{NUMBER:iis.access.win32_status:long})
(?:-|%{NUMBER:http.response.body.bytes:long}) (?:-|%{NUMBER:http.request.body.bytes:long})
(?:-|%{NUMBER:temp.duration:long})'
- '%{TIMESTAMP_ISO8601:iis.access.time} (?:-|%{NOTSPACE:iis.access.site_name}) (?:-|%{NOTSPACE:iis.access.server_name})
(?:-|%{IPORHOST:destination.address}) (?:-|%{WORD:http.request.method}) (?:-|%{NOTSPACE:url.path})
(?:-|%{NOTSPACE:url.query}) (?:-|%{NUMBER:destination.port:long}) (?:-|%{NOTSPACE:user.name})
(?:-|%{IPORHOST:source.address}) (?:-|HTTP/%{NUMBER:http.version}) (?:-|%{NOTSPACE:user_agent.original})
(?:-|%{NOTSPACE:iis.access.cookie}) (?:-|%{NOTSPACE:http.request.referrer}) (?:-|%{NOTSPACE:destination.domain})
(?:-|%{NUMBER:http.response.status_code:long}) (?:-|%{NUMBER:iis.access.sub_status:long})
(?:-|%{NUMBER:iis.access.win32_status:long}) (?:-|%{NUMBER:http.response.body.bytes:long})
(?:-|%{NUMBER:http.request.body.bytes:long}) (?:-|%{NUMBER:temp.duration:long})'
- '%{TIMESTAMP_ISO8601:iis.access.time} \[%{IPORHOST:destination.address}\]\(http://%{IPORHOST:destination.address}\)
(?:-|%{WORD:http.request.method}) (?:-|%{NOTSPACE:url.path}) (?:-|%{NOTSPACE:url.query}) (?:-|%{NUMBER:destination.port:long})
(?:-|%{NOTSPACE:user.name}) \[%{IPORHOST:source.address}\]\(http://%{IPORHOST:source.address}\)
(?:-|%{NOTSPACE:user_agent.original}) (?:-|%{NUMBER:http.response.status_code:long}) (?:-|%{NUMBER:iis.access.sub_status:long})
(?:-|%{NUMBER:iis.access.win32_status:long}) (?:-|%{NUMBER:temp.duration:long})'
- '%{TIMESTAMP_ISO8601:iis.access.time} (?:-|%{IPORHOST:destination.address}) (?:-|%{WORD:http.request.method})
(?:-|%{NOTSPACE:url.path}) (?:-|%{NOTSPACE:url.query}) (?:-|%{NUMBER:destination.port:long}) (?:-|%{NOTSPACE:user.name})
(?:-|%{IPORHOST:source.address}) (?:-|%{NOTSPACE:user_agent.original}) (?:-|%{NUMBER:http.response.status_code:long})
(?:-|%{NUMBER:iis.access.sub_status:long}) (?:-|%{NUMBER:iis.access.win32_status:long})
(?:-|%{NUMBER:temp.duration:long})'
ignore_missing: true
- remove:
field: message
- rename:
field: '@timestamp'
target_field: event.created
- date:
field: iis.access.time
target_field: '@timestamp'
formats:
- yyyy-MM-dd HH:mm:ss
- remove:
field: iis.access.time
- script:
lang: painless
source: ctx.event.duration = Math.round(ctx.temp.duration * params.scale)
params:
scale: 1000000
if: ctx.temp?.duration != null
- remove:
field: temp.duration
ignore_missing: true
- urldecode:
field: user_agent.original
ignore_missing: true
- user_agent:
field: user_agent.original
ignore_missing: true
- grok:
field: destination.address
ignore_failure: true
patterns:
- '%{NOZONEIP:destination.ip}'
pattern_definitions:
NOZONEIP: '[^%]*'
- grok:
field: source.address
ignore_failure: true
patterns:
- '%{NOZONEIP:source.ip}'
pattern_definitions:
NOZONEIP: '[^%]*'
- geoip:
field: source.ip
target_field: source.geo
ignore_missing: true
- geoip:
database_file: GeoLite2-ASN.mmdb
field: source.ip
target_field: source.as
properties:
- asn
- organization_name
ignore_missing: true
- rename:
field: source.as.asn
target_field: source.as.number
ignore_missing: true
- rename:
field: source.as.organization_name
target_field: source.as.organization.name
ignore_missing: true
- set:
field: event.kind
value: event
- append:
field: event.category
value: web
- append:
field: event.category
value: network
if: "ctx?.source?.ip != null && ctx?.destination?.ip != null"
- append:
field: event.type
value: connection
if: "ctx?.source?.ip != null && ctx?.destination?.ip != null"
- append:
field: related.ip
value: "{{source.ip}}"
if: "ctx?.source?.ip != null"
- append:
field: related.ip
value: "{{destination.ip}}"
if: "ctx?.destination?.ip != null"
- append:
field: related.user
value: "{{user.name}}"
if: "ctx?.user?.name != null"
- set:
field: event.outcome
value: success
if: "ctx?.http?.response?.status_code != null && ctx.http.response.status_code < 400"
- set:
field: event.outcome
value: failure
if: "ctx?.http?.response?.status_code != null && ctx.http.response.status_code >= 400"
on_failure:
- set:
field: error.message
value: '{{ _ingest.on_failure_message }}'
Are these errors related to the url.query having some special characters like the "@" or the "&"? Is there a way to correct this in the ingest file?
Thanks in advance
Wolf