# Enrich filebeat content using ingest pipeline

**URL:** <https://discuss.elastic.co/t/enrich-filebeat-content-using-ingest-pipeline/68182>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [December 6, 2016, 3:00pm UTC](https://discuss.elastic.co/t/enrich-filebeat-content-using-ingest-pipeline/68182 "2016-12-06T15:00:30Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dwjvaughan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dwjvaughan/32/10648_2.png) [@dwjvaughan](https://discuss.elastic.co/u/dwjvaughan)\
**Post date:** [December 6, 2016, 3:00pm UTC](https://discuss.elastic.co/t/enrich-filebeat-content-using-ingest-pipeline/68182/1 "2016-12-06T15:00:31Z")

</div>

I want to use filebeat to send Oracle alert logs to Elasticsearch. So far I've managed to get the data in (using some multiline magic). Ideally I want to use a pipeline on the ES side to extract the date, errors (ORA-), keywords etc. from the message and have them as separate fields in the indexes. I vaguely remember seeing something similar demonstrated at Elastic{ON} London, wish I'd been paying more attention!

If anyone knows if/how this can be done, I'd really appreciate some guidance.

Many thanks,  
Dave

EDIT: OK, so I found some documentation: [https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html)

So now I have this:

```
post _ingest/pipeline/_simulate
{
  "pipeline": {
  "description" : "grok test",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{DATESTAMP_OTHER:ora-date}"]
      }
    }
  ]
},
"docs":[
  {
    "_source": {
      "message": "Wed Aug 26 16:29:35 2015\nStarting background process CJQ0"
    }
  }
  ]
}

```

The issue I have is that the Oracle date doesn't appear to be in a standard setting, can I extract this using Grok?

The above gives me the following (presumable because the date is in the wrong format):

`java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [Wed Aug 26 16:29:35 2015\nStarting background process CJQ0`

---

<div class="post-metadata">

**Author:** ![dwjvaughan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dwjvaughan/32/10648_2.png) [@dwjvaughan](https://discuss.elastic.co/u/dwjvaughan)\
**Post date:** [December 7, 2016, 8:36am UTC](https://discuss.elastic.co/t/enrich-filebeat-content-using-ingest-pipeline/68182/2 "2016-12-07T08:36:52Z")

</div>

Just in case anyone stumbles upon this, I've managed to extract the date successfully. I'm no regexpert (haha!), and I'm sure there's a more graceful way, but it is working:

```
post _ingest/pipeline/_simulate
{
  "pipeline": {
  "description" : "grok test",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{ORADATE:ora-date}"],
        "pattern_definitions": {
          "ORADATE" :
          "^[A-Z]{1}[a-z]{2} [A-Z]{1}[a-z]{2} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{4}"
          
        }
      }
    },
      {
      "date": {
        "field": "ora-date",
        "formats": ["EEE MMM dd HH:mm:ss YYYY"]
      }
    }
  ]
},
"docs":[
  {
    "_source": {
      "message": "Wed Aug 26 16:29:35 2015\nStarting background process CJQ0"
    }
  }
  ]
}
```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [December 7, 2016, 8:49am UTC](https://discuss.elastic.co/t/enrich-filebeat-content-using-ingest-pipeline/68182/3 "2016-12-07T08:49:05Z")

</div>

@dwjvaughan Glad you found a solution and thanks for sharing it with the community.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [December 7, 2016, 12:42pm UTC](https://discuss.elastic.co/t/enrich-filebeat-content-using-ingest-pipeline/68182/4 "2016-12-07T12:42:27Z")

</div>

Great you got this working.

ingest node grok ships some default grok patterns. [see github](https://github.com/elastic/elasticsearch/tree/master/modules/ingest-common/src/main/resources/patterns). Not sure how you could re-use these patterns or enhance by installing your own ones.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 4, 2017, 12:42pm UTC](https://discuss.elastic.co/t/enrich-filebeat-content-using-ingest-pipeline/68182/5 "2017-01-04T12:42:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.