Custom module from default

Hi,
I have IHS in my env, so I have 3 types of logs: error, access and ncsa.
I am currently using modules: apache2/error for error logs and apache2/access for ncsa logs, but access logs are different. I tried to duplicate apache2/access dir and just change pattern in ingest/default.json but I am getting error in kibana: Provided Grok expressions do not match field value.

example log:
[09/Jul/2019:00:00:36 -0500] 200 10.10.10.10 - GET HTTP/1.1 "Server: servername.net:443" "RequestedURL: int.webserver:443/here/is/path" "Referer: -" "User-Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com" 6 sec 6849696 usec "JSESSIONID: xxxxxxxxxxxxx_xxxxxxxxxxxxx:-1" "True-Client-IP: 90.90.90.90" "X-Source: test, test" "X-Forwarded-For: 90.90.90.90, 2.22.222.22" 70411 bytes

and default.json with my expresion:
{
"description": "Custom pipeline for parsing Apache2 access logs. Requires the geoip and user_agent plugins.",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"\\[%{HTTPDATE:apache2.access.time}\\] %{NUMBER:apache2.access.response_code} %{IPORHOST:apache2.access.lbalancer_ip} %{DATA:apache2.access.user_name} %{WORD:apache2.access.method} HTTP/%{NUMBER:apache2.access.http_version} "Server: %{DATA:apache2.access.webserver}" "RequestedURL: %{DATA:apache2.access.url}" "Referer: %{DATA:apache2.access.referrer}" "User-Agent: %{DATA:apache2.access.agent}" %{NUMBER:apache2.access.sec} sec %{NUMBER:apache2.access.usec} usec "JSESSIONID: %{DATA:apache2.access.jsessionid}" "True-Client-IP: %{IPORHOST:apache2.access.remote_ip}" "X-Source: %{DATA:apache2.access.xsource}" "X-Forwarded-For: %{DATA:apache2.access.xforwardedfor}" %{NUMBER:apache2.access.sizeinbytes} bytes"
],
"ignore_missing": true
}
},{
"remove":{
"field": "message"
}
}, {
"rename": {
"field": "@timestamp",
"target_field": "read_timestamp"
}
}, {
"date": {
"field": "apache2.access.time",
"target_field": "@timestamp",
"formats": ["dd/MMM/YYYY:H:m:s Z"]
}
}, {
"remove": {
"field": "apache2.access.time"
}
}, {
"user_agent": {
"field": "apache2.access.agent",
"target_field": "apache2.access.user_agent",
"ignore_failure": true
}
}, {
"remove": {
"field": "apache2.access.agent",
"ignore_failure": true
}
}, {
"geoip": {
"field": "apache2.access.remote_ip",
"target_field": "apache2.access.geoip"
}
}],
"on_failure" : [{
"set" : {
"field" : "error",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}

My pattern is wrong? or maybe new module cannot be created just by copying existing one?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.