Hi,
I have IHS in my env, so I have 3 types of logs: error, access and ncsa.
I am currently using modules: apache2/error for error logs and apache2/access for ncsa logs, but access logs are different. I tried to duplicate apache2/access dir and just change pattern in ingest/default.json but I am getting error in kibana: Provided Grok expressions do not match field value.
example log:
[09/Jul/2019:00:00:36 -0500] 200 10.10.10.10 - GET HTTP/1.1 "Server: servername.net:443" "RequestedURL: int.webserver:443/here/is/path" "Referer: -" "User-Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com" 6 sec 6849696 usec "JSESSIONID: xxxxxxxxxxxxx_xxxxxxxxxxxxx:-1" "True-Client-IP: 90.90.90.90" "X-Source: test, test" "X-Forwarded-For: 90.90.90.90, 2.22.222.22" 70411 bytes
and default.json with my expresion:
{
"description": "Custom pipeline for parsing Apache2 access logs. Requires the geoip and user_agent plugins.",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"\\[%{HTTPDATE:apache2.access.time}\\] %{NUMBER:apache2.access.response_code} %{IPORHOST:apache2.access.lbalancer_ip} %{DATA:apache2.access.user_name} %{WORD:apache2.access.method} HTTP/%{NUMBER:apache2.access.http_version} "Server: %{DATA:apache2.access.webserver}" "RequestedURL: %{DATA:apache2.access.url}" "Referer: %{DATA:apache2.access.referrer}" "User-Agent: %{DATA:apache2.access.agent}" %{NUMBER:apache2.access.sec} sec %{NUMBER:apache2.access.usec} usec "JSESSIONID: %{DATA:apache2.access.jsessionid}" "True-Client-IP: %{IPORHOST:apache2.access.remote_ip}" "X-Source: %{DATA:apache2.access.xsource}" "X-Forwarded-For: %{DATA:apache2.access.xforwardedfor}" %{NUMBER:apache2.access.sizeinbytes} bytes"
],
"ignore_missing": true
}
},{
"remove":{
"field": "message"
}
}, {
"rename": {
"field": "@timestamp",
"target_field": "read_timestamp"
}
}, {
"date": {
"field": "apache2.access.time",
"target_field": "@timestamp",
"formats": ["dd/MMM/YYYY:H:m:s Z"]
}
}, {
"remove": {
"field": "apache2.access.time"
}
}, {
"user_agent": {
"field": "apache2.access.agent",
"target_field": "apache2.access.user_agent",
"ignore_failure": true
}
}, {
"remove": {
"field": "apache2.access.agent",
"ignore_failure": true
}
}, {
"geoip": {
"field": "apache2.access.remote_ip",
"target_field": "apache2.access.geoip"
}
}],
"on_failure" : [{
"set" : {
"field" : "error",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}
My pattern is wrong? or maybe new module cannot be created just by copying existing one?