# Appending app field to coreos logs

**URL:** <https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312>\
**Category:** Logstash\
**Created:** [March 28, 2017, 2:44pm UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312 "2017-03-28T14:44:25Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ticktockhouse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ticktockhouse/32/16801_2.png) [@ticktockhouse](https://discuss.elastic.co/u/ticktockhouse)\
**Post date:** [March 28, 2017, 2:44pm UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312/1 "2017-03-28T14:44:25Z")

</div>

I have logs coming in from coreos nodes via filebeat, the "source" field is of the general form: /var/log/containers/\<name of container\>-\<some alphanumeric string I don't care about\>

I'd like to parse the part after /var/log/containers/, and have found that the regex ^(\w+\b-\w+\b) should pull out the part I need. I have put this into a file in the pattern\_dir.

So my grok filter currently looks like this:

```
filter {
  grok {
    match => { "source" => "%{GREEDYDATA}/%{CONTAINERAPP:app}*" }
    patterns_dir => ["/patterns"]
  }
}

```

The patterns directory has the a file "containerlogs" containing:

```
CONTAINERAPP ^(\w+\b-\w+\b)

```

...but I currently don't even see an "app" field. According to logstash's output, it seems happy with this pattern.

Can anybody point me in the right direction?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 28, 2017, 2:51pm UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312/2 "2017-03-28T14:51:28Z")

</div>

Don't include `^` in the grok pattern. Do the resulting events have a `_grokparsefailure` tag?

---

<div class="post-metadata">

**Author:** ![ticktockhouse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ticktockhouse/32/16801_2.png) [@ticktockhouse](https://discuss.elastic.co/u/ticktockhouse)\
**Post date:** [March 28, 2017, 3:01pm UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312/3 "2017-03-28T15:01:46Z")

</div>

Thank you so much! That worked a treat!

Now, I've noticed that I have to match a different regex in the \<name of container\> part of the string, but I guess that's beyond the scope of this thread 🙂

---

<div class="post-metadata">

**Author:** ![ticktockhouse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ticktockhouse/32/16801_2.png) [@ticktockhouse](https://discuss.elastic.co/u/ticktockhouse)\
**Post date:** [March 28, 2017, 3:11pm UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312/4 "2017-03-28T15:11:38Z")

</div>

Actually, one further question - why does this work? We only want the alphanumeric words at the beginning, so why doesn't it match everything like the various regex checkers I've found online say it should?

OK, after some playing around, I need more help 😊

I changed my regex to this:

```
CONTAINERAPP .+?(?=-\d) 

```

As I need to match everything up to some digits. I now don't even get an "app" tag being sent to logstash..

Any chance of a bit more help?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 29, 2017, 5:28am UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312/5 "2017-03-29T05:28:58Z")

</div>

Off the top of my head I'm not sure what's going on. Your positive lookahead looks correct.

---

<div class="post-metadata">

**Author:** ![ticktockhouse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ticktockhouse/32/16801_2.png) [@ticktockhouse](https://discuss.elastic.co/u/ticktockhouse)\
**Post date:** [March 29, 2017, 8:49am UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312/6 "2017-03-29T08:49:41Z")

</div>

Thanks for looking. Do you think this may be a bug? Should I create an issue on github? Should I take more debugging steps first?

Thanks

Jerry

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 29, 2017, 8:51am UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312/7 "2017-03-29T08:51:18Z")

</div>

Statistically very few grok problems are caused by bugs so I'd debug this further.

---

<div class="post-metadata">

**Author:** ![ticktockhouse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ticktockhouse/32/16801_2.png) [@ticktockhouse](https://discuss.elastic.co/u/ticktockhouse)\
**Post date:** [April 3, 2017, 12:12pm UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312/8 "2017-04-03T12:12:42Z")

</div>

I have looked at this again, and grok still seem s not to work with this particular regex. with others it works, but of course it does not match the pattern I want it to.

I have looked at [Grok debugger](http://grokdebug.herokuapp.com/), plugging in the relevant values, and it gives me the output:

```
{
  "GREEDYDATA": [
    [
      "/var/log/containers"
    ]
  ],
  "container": [
    [
      "my-really-even-more-pukka-container"
    ]
  ]
}

```

...but the field still does not appear in the output. Could it be a bug?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 3, 2017, 12:18pm UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312/9 "2017-04-03T12:18:27Z")

</div>

If you need to match everything up to the first digit, why not use a basic not digit pattern (`\D+`) ?

---

<div class="post-metadata">

**Author:** ![ticktockhouse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ticktockhouse/32/16801_2.png) [@ticktockhouse](https://discuss.elastic.co/u/ticktockhouse)\
**Post date:** [April 3, 2017, 2:48pm UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312/10 "2017-04-03T14:48:02Z")

</div>

Thanks @Christian_Dahlqvist, almost there - I end up with:

```
{
  "GREEDYDATA": [
    [
      "/var/log/containers"
    ]
  ],
  "container": [
    [
      "my-really-even-more-pukka-container-"
    ]
  ]
}

```

(note the trailing hyphen). Even better, it works when I test in logstash. That's what's confusing me here: why does the previous one work in the grok debugger, but fail when I try it in logstash?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 1, 2017, 2:48pm UTC](https://discuss.elastic.co/t/appending-app-field-to-coreos-logs/80312/11 "2017-05-01T14:48:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.