# Logstash equivalents with ingest pipeline

**URL:** <https://discuss.elastic.co/t/logstash-equivalents-with-ingest-pipeline/83938>\
**Category:** Elasticsearch\
**Created:** [April 27, 2017, 9:01pm UTC](https://discuss.elastic.co/t/logstash-equivalents-with-ingest-pipeline/83938 "2017-04-27T21:01:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![messageinabottle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/messageinabottle/32/17549_2.png) [@messageinabottle](https://discuss.elastic.co/u/messageinabottle)\
**Post date:** [April 27, 2017, 9:01pm UTC](https://discuss.elastic.co/t/logstash-equivalents-with-ingest-pipeline/83938/1 "2017-04-27T21:01:00Z")

</div>

We are researching our ability to remove Logstash and instead have Filebeats send directly to Elasticsearch's new ingest pipelines.

I am trying to take the following Logstash filter and create an ingest pipeline.

```
grok {
  match => ["message", "^:%{DATESTAMP:timestamp} %{DATA:thread} %{LOGLEVEL:log_level} +%{DATA:log_name} (%{DATA:agent_id} )?session_id:(%{DATA:sid})? time_id:(%{DATA:time_id})? referrer:(%{DATA:referrer})? %{GREEDYDATA:msg}$"]
  overwrite => ["timestamp"]
  add_tag => ["application_log", "%{type}"]
}
if [msg] =~ /.*Publishing following data to SNS topic.*/ {
  grok { match => ["msg", "(?<sns_queue>Publishing[^{]+) %{GREEDYDATA:snsMsg}"] }
  json {
    source => "snsMsg"
    target => "parsedMsg"
  }
}
if [msg] =~ /^.+Exception|Error.+$/ {
  mutate {
    add_tag => ["error"]
  }
}
mutate {
  remove_field => "snsMsg"
}
date {
  match => ["timestamp", "MM-dd-yyyy HH:mm:ss.SSS"]
}

```

So far all I have is:

```
{
    "description": "Application Logs",
    "processors": [
        {
            grok {
              "field": "message",
              "patterns": ["^:%{DATESTAMP:timestamp} %{DATA:thread} %{LOGLEVEL:log_level} +%{DATA:log_name} (%{DATA:agent_id} )?session_id:(%{DATA:sid})? time_id:(%{DATA:time_id})? referrer:(%{DATA:referrer})? %{GREEDYDATA:msg}$"]
    }
]
}

```

I have no clue how to handle the overwrite, add\_tag, and add\_field items. Can the append processor be used for the add\_tag?

A point in the right direction would be very much appreciated.

---

<div class="post-metadata">

**Author:** ![talevy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talevy/32/44896_2.png) [@talevy](https://discuss.elastic.co/u/talevy)\
**Post date:** [May 5, 2017, 6:57pm UTC](https://discuss.elastic.co/t/logstash-equivalents-with-ingest-pipeline/83938/2 "2017-05-05T18:57:32Z")

</div>

Do you mind providing a sample input document to your pipeline?

Also, a note about tagging. Ingest does not have a notion of tags, tags can be added into a field called "tags" if you wish, and the SetProcessor can help you achieve this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 2, 2017, 7:02pm UTC](https://discuss.elastic.co/t/logstash-equivalents-with-ingest-pipeline/83938/3 "2017-06-02T19:02:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.