# Regex for redis logs

**URL:** <https://discuss.elastic.co/t/regex-for-redis-logs/84968>\
**Category:** Logstash\
**Created:** [May 8, 2017, 6:45pm UTC](https://discuss.elastic.co/t/regex-for-redis-logs/84968 "2017-05-08T18:45:50Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sLuvpreet33](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sluvpreet33/32/50626_2.png) [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)\
**Post date:** [May 8, 2017, 6:45pm UTC](https://discuss.elastic.co/t/regex-for-redis-logs/84968/1 "2017-05-08T18:45:50Z")

</div>

I have made a regex for redis logs. I have tested it at [http://regexr.com/](http://regexr.com/) but it is not working in logstash.

Here is a log entry from redis logs,

```
30200:C 06 May 21:25:10.186 * RDB: 6 MB of memory used by copy-on-write

```

Here is the regex pattern file , location is `/opt/logstash/patterns/redis`

```
INTO (?:[+-]?(?:[0-9]+))\:[A-Z]
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
HOUR (?:2[0123]|[01]?[0-9])
MINUTE (?:[0-5][0-9])
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
REDISTIMESTAMP %{MONTHDAY} %{MONTH} %{TIME}
GREEDYDATA .*
REDISALPHALOG %{INTO:pid} %{REDISTIMESTAMP:timestamp} %{GREEDYDATA:action}

```

Here is the logstash file,

```
filter {
   if[type] == "redis" {
    grok {
        patterns_dir => ["/opt/logstash/patterns"]
        match => ["message" , "%{REDISALPHALOG:message}"]
        overwrite => ["message"]
          }
   }
 }

```

But it is always stored like this,

```
2017/05/08 18:38:23.950473 client.go:214: DBG Publish: {
"@timestamp": "2017-05-08T18:38:18.950Z",
"beat": {
    "hostname": "DHARI-Inspiron-3542",
    "name": "DHARI-Inspiron-3542",
    "version": "5.4.0"
  },
  "input_type": "log",
   "message": "30200:C 06 May 21:25:10.186 * RDB: 6 MB of memory used by copy-on-write",
   "offset": 249,
  "source": "/var/log/alpharedis.log",
  "type": "redis"
 }

```

What am I missing here ? It is my first experience with custom log formats. Any kind of help is welcomed.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 8, 2017, 7:44pm UTC](https://discuss.elastic.co/t/regex-for-redis-logs/84968/2 "2017-05-08T19:44:54Z")

</div>

That's a Filebeat log. Have you configured Filebeat to send to Logstash? What do things look like after Logstash has processed the events?

Not that it has anything to do with your problem, but why are you duplicating so many grok patterns in your own file?

---

<div class="post-metadata">

**Author:** ![sLuvpreet33](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sluvpreet33/32/50626_2.png) [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)\
**Post date:** [May 8, 2017, 7:53pm UTC](https://discuss.elastic.co/t/regex-for-redis-logs/84968/3 "2017-05-08T19:53:54Z")

</div>

Yes I have configured filebeat and logstash.

Here are the full files,

```
filebeat.prospectors:
- input_type: log
paths:
    - /var/log/alpharedis.log
  document_type: redis
output.logstash:
  hosts: ["127.0.0.1:5043"]

```

Here is the logstash full conf file,

```
input {
beats {
     port => "5043"
   }
}
filter {
   if[type] == "redis" {
    grok {
        patterns_dir => ["/opt/logstash/patterns"]
        match => ["message" , "%{REDISALPHALOG:message}"]
        overwrite => ["message"]
         }
    }
 }
output {
   elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{type}_indexer"
    }
}

```

I run this command,

```
luvpreet@DHARI-Inspiron-3542:/usr/bin$ sudo ./filebeat.sh -e -c /etc/filebeat/filebeat.yml -d "publish"

```

and then get this output,

```
 2017/05/08 18:38:23.950473 client.go:214: DBG Publish: {
 "@timestamp": "2017-05-08T18:38:18.950Z",
 "beat": {
    "hostname": "DHARI-Inspiron-3542",
    "name": "DHARI-Inspiron-3542",
    "version": "5.4.0"
  },
  "input_type": "log",
   "message": "30200:C 06 May 21:25:10.186 * RDB: 6 MB of memory used by copy-on-write",
   "offset": 249,
  "source": "/var/log/alpharedis.log",
  "type": "redis"
 }

```

> [@magnusbaeck](#):
>
> Not that it has anything to do with your problem, but why are you duplicating so many grok patterns in your own file?

Just a precautionary mind.😁

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 8, 2017, 8:30pm UTC](https://discuss.elastic.co/t/regex-for-redis-logs/84968/4 "2017-05-08T20:30:36Z")

</div>

> I run this command, [...] and then get this output,

Once again, that's what Filebeat sends to Logstash. Since that occurs prior to the Logstash filters it obviously won't reflect any transforms that your Logstash filters make. You should look at what comes _out_ from Logstash, for example by looking at Elasticsearch.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 5, 2017, 8:34pm UTC](https://discuss.elastic.co/t/regex-for-redis-logs/84968/5 "2017-06-05T20:34:46Z")

</div>

Closed on June 5, 2017, 8:34pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.