# Parse logs

**URL:** <https://discuss.elastic.co/t/parse-logs/29225>\
**Category:** Logstash\
**Created:** [September 14, 2015, 9:28am UTC](https://discuss.elastic.co/t/parse-logs/29225 "2015-09-14T09:28:34Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![silviupanaite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silviupanaite/32/4766_2.png) [@silviupanaite](https://discuss.elastic.co/u/silviupanaite)\
**Post date:** [September 14, 2015, 9:28am UTC](https://discuss.elastic.co/t/parse-logs/29225/1 "2015-09-14T09:28:34Z")

</div>

I'm trying to parse:  
{"log":"86.121.176.201 - - [09/Sep/2015:14:14:06 +0000] "GET /css/style.less HTTP/1.1" 304 0 "[http://developer.unificationengine.com/login](http://developer.unificationengine.com/login)" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_10\_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36" "-"\r\n","stream":"stdout","time":"2015-09-09T14:14:06.112442077Z"}

if [type] == "docker-logs" {  
grok {  
match =\> ["message", "%{QS}:%{QS:message\_log},%{QS}:%{QS:log\_type},%{QS}:%{QS:time}"]  
match =\> ["message", "\A{"log":"%{IP} - - [%{HTTPDATE:time}] \"%{WORD} %{PATH} %{PROG}%{NOTSPACE}%{SPACE}%{NUMBER} 0 \"%{URI}%{NOTSPACE}%{SPACE}%{GREEDYDATA}"]  
}  
mutate {  
rename =\> ["log", "message"]  
}  
date {  
match =\> ["time", "ISO8601"]  
remove\_field =\> ["time"]  
}  
}

But I'm not able to, the message output is still the same. Can you help me out, I don't know what I'm doing wrong?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 14, 2015, 10:46am UTC](https://discuss.elastic.co/t/parse-logs/29225/2 "2015-09-14T10:46:59Z")

</div>

Don't use grok to parse JSON. If possible use the [json codec](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json.html) for the input. Failing that, use the [json filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html). Once that's in place, see if you can use the [COMBINEDAPACHELOG pattern](https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns) instead of writing your own pattern from scratch.

---

<div class="post-metadata">

**Author:** ![silviupanaite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silviupanaite/32/4766_2.png) [@silviupanaite](https://discuss.elastic.co/u/silviupanaite)\
**Post date:** [September 15, 2015, 10:20am UTC](https://discuss.elastic.co/t/parse-logs/29225/3 "2015-09-15T10:20:55Z")

</div>

Tried what you suggested... nothing changed. Tried with input codec and also filter.  
Here is the json pulled from kibana4  
{  
"\_index": "logstash-2015.09.15",  
"\_type": "docker-logs",  
"\_id": "AU\_QgbrYMc0WVE2CwNCU",  
"\_score": null,  
"\_source": {  
"message": "{"log":"sendmail\r\n","stream":"stdout","time":"2015-09-15T10:16:38.353745638Z"}",  
"@version": "1",  
"@timestamp": "2015-09-15T10:16:46.144Z",  
"type": "docker-logs",  
"file": "/var/lib/docker/containers/bfc1026c1e00a25cd145d449f031e00c26abb120e8160c20a12198e0822ec365/bfc1026c1e00a25cd145d449f031e00c26abb120e8160c20a12198e0822ec365-json.log",  
"host": "bs1.uib.co",  
"offset": "10824",  
"tags": "unifiedinbox/ue-developer",  
"serivce": "map[io.rancher.project.name:unificationengine-developer-portal io.rancher.project\_service.name:unificationengine-developer-portal/ue-developer io.rancher.scheduler.affinity:container\_label\_soft:io.rancher.service.deployment.unit=12495afc-716d-41ec-9152-d652c390d09e io.rancher.service.deployment.unit:12495afc-716d-41ec-9152-d652c390d09e io.rancher.stack.name:unificationengine-developer-portal io.rancher.container.ip:10.42.137.137/16 io.rancher.container.uuid:b7fc233f-cabf-4a97-8102-005ded7feb93 io.rancher.service.launch.config:io.rancher.service.primary.launch.config io.rancher.stack\_service.name:unificationengine-developer-portal/ue-developer]"  
},  
"fields": {  
"@timestamp": [  
1442312206144  
]  
},  
"sort": [  
1442312206144  
]  
}

and here's the filter used:  
filter {  
if "unifiedinbox/ue-developer" in [tags] {  
json {  
source =\> "message"  
date {  
match =\> ["time", "ISO8601"]  
remove\_field =\> ["time"]  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 15, 2015, 11:22am UTC](https://discuss.elastic.co/t/parse-logs/29225/4 "2015-09-15T11:22:26Z")

</div>

Try isolating things. Skip the `if "unifiedinbox/ue-developer" in [tags]` conditional, for example. I can't reproduce what you're seeing with either of these minimal examples:

```
$ cat data
{"log":"sendmail\r\n","stream":"stdout","time":"2015-09-15T10:16:38.353745638Z"}
$ cat json-codec.config 
input { stdin { codec => json } }
output { stdout { codec => rubydebug } }
$ /opt/logstash/bin/logstash -f json-codec.config < data 
Logstash startup completed
{
           "log" => "sendmail\r\n",
        "stream" => "stdout",
          "time" => "2015-09-15T10:16:38.353745638Z",
      "@version" => "1",
    "@timestamp" => "2015-09-15T11:20:13.965Z",
          "host" => "lnxmagnusbk"
}
Logstash shutdown completed
$ cat json-filter.config 
input { stdin { } }
output { stdout { codec => rubydebug } }
filter {
  json {
    source => "message"
  }
}
$ /opt/logstash/bin/logstash -f json-filter.config < data
Logstash startup completed
{
       "message" => "{\"log\":\"sendmail\\r\\n\",\"stream\":\"stdout\",\"time\":\"2015-09-15T10:16:38.353745638Z\"}",
      "@version" => "1",
    "@timestamp" => "2015-09-15T11:20:39.075Z",
          "host" => "lnxmagnusbk",
           "log" => "sendmail\r\n",
        "stream" => "stdout",
          "time" => "2015-09-15T10:16:38.353745638Z"
}
Logstash shutdown completed
```

---

<div class="post-metadata">

**Author:** ![silviupanaite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silviupanaite/32/4766_2.png) [@silviupanaite](https://discuss.elastic.co/u/silviupanaite)\
**Post date:** [September 15, 2015, 2:13pm UTC](https://discuss.elastic.co/t/parse-logs/29225/5 "2015-09-15T14:13:59Z")

</div>

The full description:  
I use [docker-gen](https://github.com/jwilder/docker-gen) to generate json files that I send using logstash forwarder to logstash server. I wasn't able to add the new field, with or without the if condition.

The docker gen template is:  
{{/\* this is a docker-gen template. See docker-gen for more details \*/}}  
{

# The network section covers network configuration 🙂

"network": {  
# A list of downstream servers listening for our messages.  
# logstash-forwarder will pick one at random and only switch if  
# the selected one appears to be dead or unresponsive  
"servers": ["[ec2-52-16-45-21.eu-west-1.compute.amazonaws.com:5000](http://ec2-52-16-45-21.eu-west-1.compute.amazonaws.com:5000)" ],

```
# The path to your trusted ssl CA file. This is used
# to authenticate your downstream server.
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",

# Network timeout in seconds. This is most important for
# logstash-forwarder determining whether to stop waiting for an
# acknowledgement from the downstream server. If an timeout is reached,
# logstash-forwarder will assume the connection or server is bad and
# will connect to a server chosen at random from the servers list.
"timeout": 15

```

},

# The list of files configurations

## "files": [# An array of hashes. Each hash tells what paths to watch and # what fields to annotate on events from those paths. {{ range $key, $value := . }} { "paths": [ "/var/lib/docker/containers/{{ $value.ID }}/{{ $value.ID }}-json.log"], "fields": { "type": "docker-logs", "tags":"{{ $value.Image.Repository }}", "serivce": "{{ $value.Labels }}" } }, {{ end }} { "paths": ["/var/log/syslog"], "fields": { "type": "syslog" } } ] }

logstash filter:  
filter{  
if [type] == "docker-logs" {  
json {  
source =\> "message"  
grok {  
match =\> ["message", "%{COMBINEDAPACHELOG}"]  
}  
add\_field =\> {  
"new\_field" =\> "new\_static\_value"  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 15, 2015, 2:18pm UTC](https://discuss.elastic.co/t/parse-logs/29225/6 "2015-09-15T14:18:56Z")

</div>

Your Logstash configuration is broken and I'm not sure what it's supposed to mean. You're starting the grok filter before closing the json filter, and the add\_field parameter doesn't belong to any filter at all. Is this what you mean?

```
filter{
  if [type] == "docker-logs" {
    json {
      source => "message"
    }
    grok {
      match => ["message", "%{COMBINEDAPACHELOG}"]
      add_field => {
        "new_field" => "new_static_value"
      }
    } 
  }
}
```

---

<div class="post-metadata">

**Author:** ![silviupanaite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silviupanaite/32/4766_2.png) [@silviupanaite](https://discuss.elastic.co/u/silviupanaite)\
**Post date:** [September 17, 2015, 6:55am UTC](https://discuss.elastic.co/t/parse-logs/29225/7 "2015-09-17T06:55:34Z")

</div>

Yes, the config you gave back is the one that I'm using.  
Looking into the stdout there is no grok parse error or any indication that the filter works.

I don't know how to go about this.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 17, 2015, 7:32am UTC](https://discuss.elastic.co/t/parse-logs/29225/8 "2015-09-17T07:32:29Z")

</div>

> Yes, the config you gave back is the one that I'm using.

So... you're manually typing in your configuration instead of copying & pasting it? Because the configuration you posted earlier wouldn't be accepted by Logstash.

> Looking into the stdout there is no grok parse error or any indication that the filter works.

What _do_ you get to stdout? Please use the rubydebug codec to make things more readable. Either comment out all filters or copy/paste them.

---

<div class="post-metadata">

**Author:** ![silviupanaite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silviupanaite/32/4766_2.png) [@silviupanaite](https://discuss.elastic.co/u/silviupanaite)\
**Post date:** [September 17, 2015, 8:58am UTC](https://discuss.elastic.co/t/parse-logs/29225/9 "2015-09-17T08:58:22Z")

</div>

I was cleaning up my config as there were some clutter.  
This is the stdout:  
{  
"message" =\> "{"log":"2015-09-17 08:55:02,975 INFO spawned: 'cassandra' with pid 5954\r\n","stream":"stdout","time":"2015-09-17T08:55:02.975891882Z"}",  
"@version" =\> "1",  
"@timestamp" =\> "2015-09-17T08:55:08.122Z",  
"type" =\> "docker-logs",  
"file" =\> "/var/lib/docker/containers/9c0c7cf5f0cfb88dcd84144db6f216c95e74f7a24be0c560848ad9ff8e4f8fe4/9c0c7cf5f0cfb88dcd84144db6f216c95e74f7a24be0c560848ad9ff8e4f8fe4-json.log",  
"host" =\> "xxx",  
"offset" =\> "3666273",  
"tags" =\> "xxx",  
"serivce" =\> "xxxx"  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 17, 2015, 9:06am UTC](https://discuss.elastic.co/t/parse-logs/29225/10 "2015-09-17T09:06:31Z")

</div>

And this is without any filter? What if you enable the json and grok filters posted above?

---

<div class="post-metadata">

**Author:** ![silviupanaite](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/silviupanaite/32/4766_2.png) [@silviupanaite](https://discuss.elastic.co/u/silviupanaite)\
**Post date:** [September 17, 2015, 9:21am UTC](https://discuss.elastic.co/t/parse-logs/29225/11 "2015-09-17T09:21:03Z")

</div>

Well they work...  
It turn out I was editing another file, not actual config (did update logstash several times).

I feel like an 🍅.

Thank you Magnus for helping me out.

---

<div class="post-metadata">

**Author:** ![cmittana](https://avatars.discourse-cdn.com/v4/letter/c/e9a140/32.png) [@cmittana](https://discuss.elastic.co/u/cmittana)\
**Post date:** [January 14, 2016, 3:49pm UTC](https://discuss.elastic.co/t/parse-logs/29225/12 "2016-01-14T15:49:25Z")

</div>

Hi ,  
I saw the kibana parsed Json response of yours .  
{  
"\_index": "logstash-2015.09.15",  
"\_type": "docker-logs",  
"id": "AUQgbrYMc0WVE2CwNCU",  
"\_score": null,  
"\_source": {  
"message": "{"log":"sendmail\r\n","stream":"stdout","time":"2015-09-15T10:16:38.353745638Z"}",  
"@version": "1",  
"@timestamp": "2015-09-15T10:16:46.144Z",  
"type": "docker-logs",  
"file": "/var/lib/docker/containers/bfc1026c1e00a25cd145d449f031e00c26abb120e8160c20a12198e0822ec365/bfc1026c1e00a25cd145d449f031e00c26abb120e8160c20a12198e0822ec365-json.log",  
"host": "[bs1.uib.co](http://bs1.uib.co)",  
"offset": "10824",  
"tags": "unifiedinbox/ue-developer",  
"serivce": "map[[io.rancher.project.name](http://io.rancher.project.name):unificationengine-developer-portal io.rancher.project\_service.name:unificationengine-developer-portal/ue-developer io.rancher.scheduler.affinity:container\_label\_soft:io.rancher.service.deployment.unit=12495afc-716d-41ec-9152-d652c390d09e io.rancher.service.deployment.unit:12495afc-716d-41ec-9152-d652c390d09e [io.rancher.stack.name](http://io.rancher.stack.name):unificationengine-developer-portal io.rancher.container.ip:10.42.137.137/16 io.rancher.container.uuid:b7fc233f-cabf-4a97-8102-005ded7feb93 io.rancher.service.launch.config:io.rancher.service.primary.launch.config io.rancher.stack\_service.name:unificationengine-developer-portal/ue-developer]"  
},  
"fields": {  
"@timestamp": [  
1442312206144  
]  
},  
"sort": [  
1442312206144  
]  
}

Questions is how did you get the rancher metadata in your reponse .What config changes you had to make to get the rancher metadata like  
"serivce": "map[[io.rancher.project.name](http://io.rancher.project.name):unificationengine-developer-portal io.rancher.project\_service.name:unificationengine-developer-portal/ue-developer io.rancher.scheduler.affinity:container\_label\_soft:io.rancher.service.deployment.unit=12495afc-716d-41ec-9152-d652c390d09e io.rancher.service.deployment.unit:12495afc-716d-41ec-9152-d652c390d09e [io.rancher.stack.name](http://io.rancher.stack.name):unificationengine-developer-portal io.rancher.container.ip:10.42.137.137/16 io.rancher.container.uuid:b7fc233f-cabf-4a97-8102-005ded7feb93 io.rancher.service.launch.config:io.rancher.service.primary.launch.config io.rancher.stack\_service.name:unificationengine-developer-portal/ue-developer]"

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:15am UTC](https://discuss.elastic.co/t/parse-logs/29225/13 "2017-07-06T05:15:42Z")

</div>