Regex for redis logs

Elastic Stack Logstash
1

I have made a regex for redis logs. I have tested it at http://regexr.com/ but it is not working in logstash.

Here is a log entry from redis logs,

30200:C 06 May 21:25:10.186 * RDB: 6 MB of memory used by copy-on-write

Here is the regex pattern file , location is /opt/logstash/patterns/redis

INTO (?:[+-]?(?:[0-9]+))\:[A-Z]
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
HOUR (?:2[0123]|[01]?[0-9])
MINUTE (?:[0-5][0-9])
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
REDISTIMESTAMP %{MONTHDAY} %{MONTH} %{TIME}
GREEDYDATA .*
REDISALPHALOG %{INTO:pid} %{REDISTIMESTAMP:timestamp} %{GREEDYDATA:action}

Here is the logstash file,

filter {
   if[type] == "redis" {
    grok {
        patterns_dir => ["/opt/logstash/patterns"]
        match => ["message" , "%{REDISALPHALOG:message}" ]
        overwrite => ["message"]
          }
   }
 }

But it is always stored like this,

2017/05/08 18:38:23.950473 client.go:214: DBG  Publish: {
"@timestamp": "2017-05-08T18:38:18.950Z",
"beat": {
    "hostname": "DHARI-Inspiron-3542",
    "name": "DHARI-Inspiron-3542",
    "version": "5.4.0"
  },
  "input_type": "log",
   "message": "30200:C 06 May 21:25:10.186 * RDB: 6 MB of memory used by copy-on-write",
   "offset": 249,
  "source": "/var/log/alpharedis.log",
  "type": "redis"
 }

What am I missing here ? It is my first experience with custom log formats. Any kind of help is welcomed.

2

That's a Filebeat log. Have you configured Filebeat to send to Logstash? What do things look like after Logstash has processed the events?

Not that it has anything to do with your problem, but why are you duplicating so many grok patterns in your own file?

3

Yes I have configured filebeat and logstash.

Here are the full files,

filebeat.prospectors:
- input_type: log
paths:
    - /var/log/alpharedis.log
  document_type: redis
output.logstash:
  hosts: ["127.0.0.1:5043"]

Here is the logstash full conf file,

input {
beats {
     port => "5043"
   }
}
filter {
   if[type] == "redis" {
    grok {
        patterns_dir => ["/opt/logstash/patterns"]
        match => ["message" , "%{REDISALPHALOG:message}" ]
        overwrite => ["message"]
         }
    }
 }
output {
   elasticsearch {
    hosts => [ "localhost:9200" ]
    index => "%{type}_indexer"
    }
}

I run this command,

luvpreet@DHARI-Inspiron-3542:/usr/bin$ sudo ./filebeat.sh -e -c /etc/filebeat/filebeat.yml -d "publish"

and then get this output,

 2017/05/08 18:38:23.950473 client.go:214: DBG  Publish: {
 "@timestamp": "2017-05-08T18:38:18.950Z",
 "beat": {
    "hostname": "DHARI-Inspiron-3542",
    "name": "DHARI-Inspiron-3542",
    "version": "5.4.0"
  },
  "input_type": "log",
   "message": "30200:C 06 May 21:25:10.186 * RDB: 6 MB of memory used by copy-on-write",
   "offset": 249,
  "source": "/var/log/alpharedis.log",
  "type": "redis"
 }

Just a precautionary mind.:grin:

4

I run this command, [...] and then get this output,

Once again, that's what Filebeat sends to Logstash. Since that occurs prior to the Logstash filters it obviously won't reflect any transforms that your Logstash filters make. You should look at what comes out from Logstash, for example by looking at Elasticsearch.