Hello All,
Long story made shorter. I've created on my network a DNS black hole for ad server dns resolution. It works perfectly. My DNS serves up a dummy internal address for all of the adhosts in a list.
But now I want to start Analysing these requests. So I created an Nginx configuration for logging that looks like this. And it works perfectly.
log_format adblock '[$time_local] $remote_addr $host "$request"';
server {
...
access_log /var/log/nginx/adblock_access.log adblock;
...
}
The log sample is:
[02/Jul/2018:13:45:51 +1000] 192.168.0.55 s.webtrends.com "GET /js/webtrends.replicate.js HTTP/1.1"
[02/Jul/2018:13:45:52 +1000] 192.168.0.55 cdn-gl.imrworldwide.com "GET /novms/js/2/ggcmb510.js HTTP/1.1"
[02/Jul/2018:13:45:52 +1000] 192.168.0.55 secure-au.imrworldwide.com "GET /v60.js HTTP/1.1"
So All good here.
Now I've cloned the module nginx and I have renamed most vars to be adblock instead of nginx. All good. I'm seeing documents hit elasticsearch. So all comms are good. I am NOT using logstash. I'm going straight to elasticsearch.
But I have an issue. I get a grok error in filebeat. Which show up in elastic search. When I test my grok pattern in Kibana devtools. It all works. But when I move it over to the default.json file for ingestion it fails.
OK So here is the Kibana Dev Tools
Everything works in devtools.
FYI here is the groc pattern in text:
\[%{HTTPDATE:adblock.access.time}\] %{IP:adblock.access.requester} %{DATA:adblock.access.host} "%{GREEDYDATA:adblock.access.info}"
But When I transpose this to the default.json file and add the appropriate escapes as required. To this.
"description": "Pipeline for parsing Tailored Nginx access logs used for adblocking.",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"\\[%{HTTPDATE:adblock.access.time}\\] %{IP:adblock.access.requester} %{DATA:adblock.access.host} \"%{GREEDYDATA:adblock.access.info}\""
],
I end up with a grok error on every document but I can't track down what I did wrong.
error.message Provided Grok expressions do not match field value: [[01/Jul/2018:21:48:53 +1000] 192.168.0.62 t.dailymail.co.uk \"POST /s/sa HTTP/1.1\"]
I can put the same message into the dev tools and it groks just fine.
Any Ideas?