# How to use pipelines and processors

**URL:** <https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719>\
**Category:** Elasticsearch\
**Created:** [May 31, 2019, 12:12pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719 "2019-05-31T12:12:47Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![nicks1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicks1993/32/84627_2.png) [@nicks1993](https://discuss.elastic.co/u/nicks1993)\
**Post date:** [May 31, 2019, 12:12pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/1 "2019-05-31T12:12:47Z")

</div>

if I want to extract the numeric duration value from  
words::words::words::words (duration=432, words)

would this grok pipeline work  
PUT \_ingest/pipeline/parse  
{  
"description" : "parses the duration field",  
"processors" : [  
{  
"grok": {  
"field": "message",  
"patterns": ["%{NUMBER:duration}"]  
}  
}  
]  
}

Is there anything else I have to do after creating this pipeline in order to use it?

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [May 31, 2019, 1:17pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/2 "2019-05-31T13:17:44Z")

</div>

You can [\_simulate](https://www.elastic.co/guide/en/elasticsearch/reference/7.1/simulate-pipeline-api.html) the pipeline to verify that it works how you intend.

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
      "processors" : [
         {
            "grok": {
               "field": "message",
               "patterns": ["%{NUMBER:duration}"]
            }
         }
      ]
   },
  "docs": [
       {"_id": "asdf", "_source": {"message": "words::words::words::words (duration=432, words)"}}
  ]
}

```

Results in

```auto
{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "asdf",
        "_source" : {
          "duration" : "432",
          "message" : "words::words::words::words (duration=432, words)"
        },
        "_ingest" : {
          "timestamp" : "2019-05-31T13:04:20.599Z"
        }
      }
    }
  ]
}

```

As for using a pipeline, you can use one at ingest time by specifying the `pipeline` url parameter when `POST`ing docs, e.g. `POST my_index/_doc/?pipeline=my_pipeline`

Recently in `6.5` a new index setting was added: `default_pipeline`. That allows a pipeline to be ran on all docs without you having to specify it.

---

<div class="post-metadata">

**Author:** ![nicks1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicks1993/32/84627_2.png) [@nicks1993](https://discuss.elastic.co/u/nicks1993)\
**Post date:** [May 31, 2019, 1:20pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/3 "2019-05-31T13:20:59Z")

</div>

Great that was a lot of help. My pipeline works as expected. Now my logs come in automatically through fluentd I believe and I have version 6.3 of Kibana unfortunately. Is their a way to apply this pipeline to documents coming in or is their something I can do to get this working on its own?

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [May 31, 2019, 1:23pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/4 "2019-05-31T13:23:05Z")

</div>

@nicks1993 you need to change whatever is POST'ing the docs to the ES cluster to use the pipeline.

> [@BenTrent](#):
>
> POST my\_index/\_doc/?pipeline=my\_pipeline

---

<div class="post-metadata">

**Author:** ![nicks1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicks1993/32/84627_2.png) [@nicks1993](https://discuss.elastic.co/u/nicks1993)\
**Post date:** [May 31, 2019, 1:26pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/5 "2019-05-31T13:26:54Z")

</div>

Running that gives me this error

{  
"error": {  
"root\_cause": [  
{  
"type": "parse\_exception",  
"reason": "request body is required"  
}  
],  
"type": "parse\_exception",  
"reason": "request body is required"  
},  
"status": 400  
}

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [May 31, 2019, 1:42pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/6 "2019-05-31T13:42:05Z")

</div>

@nicks1993 whatever you are using to POST the documents to the index, that needs to be changed to use the pipeline.

e.g. logstash:

```auto
output {
  elasticsearch {
    /// es settings
    pipeline => "rename_hostname" <--- your pipeline
  }
}

```

posting docs manually

```auto
POST my_index/_doc/?pipeline=my_pipeline
{
// doc body
}

```

This blog covers some use cases: [https://www.elastic.co/blog/new-way-to-ingest-part-1](https://www.elastic.co/blog/new-way-to-ingest-part-1)

---

<div class="post-metadata">

**Author:** ![nicks1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicks1993/32/84627_2.png) [@nicks1993](https://discuss.elastic.co/u/nicks1993)\
**Post date:** [May 31, 2019, 1:49pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/7 "2019-05-31T13:49:30Z")

</div>

Okay cool, it looks like I'll have to add that to fluentd because I believe we are using EFK here. Thanks I'll try and pass the information along to someone who can give me access to fluentd.

Thanks for all the help, hopefully this will solve my problem.  
Now do I need to create another field named duration? because I did that already, will that effect whether it gets stored in that field?  
And once it is stored I can do normal aggregations on it like any other field right?

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [May 31, 2019, 2:14pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/8 "2019-05-31T14:14:39Z")

</div>

> [@nicks1993](#):
>
> Now do I need to create another field named duration?

You should not have to create another field named duration. You can see from the `_simulate` call that the `grok` processor added the field. If you want to see its behavior given an existing duration field, I suggest running that simulation.

> [@nicks1993](#):
>
> And once it is stored I can do normal aggregations on it like any other field right?

Yep, it is just another field.

---

<div class="post-metadata">

**Author:** ![nicks1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicks1993/32/84627_2.png) [@nicks1993](https://discuss.elastic.co/u/nicks1993)\
**Post date:** [May 31, 2019, 2:26pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/9 "2019-05-31T14:26:40Z")

</div>

Sweet man thanks. I'll have to run it by management now to get it running.  
If we decide to upgrade to version 6.5 what would the process look like, I would just set the default pipeline from dev tools on kibana?

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [May 31, 2019, 2:36pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/10 "2019-05-31T14:36:41Z")

</div>

@nicks1993 since it is an index setting, you will need to update the settings.

```auto
PUT /my_index/_settings
{
    "index" : {
        "default_pipeline" : "my_pipeline"
    }
}

```

> **[Index modules | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#dynamic-index-settings)**

> `index.default_pipeline`  
> The default [ingest node](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) pipeline for this index. Index requests will fail if the default pipeline is set and the pipeline does not exist. The default may be overridden using the `pipeline` parameter. The special pipeline name `_none` indicates no ingest pipeline should be run.

---

<div class="post-metadata">

**Author:** ![nicks1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicks1993/32/84627_2.png) [@nicks1993](https://discuss.elastic.co/u/nicks1993)\
**Post date:** [May 31, 2019, 2:41pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/11 "2019-05-31T14:41:22Z")

</div>

Hmm that's pretty cool, just tried it and confirmed it does not work on my version of elastic.  
There is no way to apply a pipeline to an index for older versions?

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [May 31, 2019, 3:31pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/12 "2019-05-31T15:31:22Z")

</div>

@nicks1993 in older versions (pre 6.5.0), the ingest pipeline must be supplied when the document is sent to ES. Either in putting it in the URL, or updating whatever automatic systems is pushing the logs to ES (e.g. Fluentd, logstash, beats, etc.)

---

<div class="post-metadata">

**Author:** ![nicks1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicks1993/32/84627_2.png) [@nicks1993](https://discuss.elastic.co/u/nicks1993)\
**Post date:** [May 31, 2019, 4:19pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/13 "2019-05-31T16:19:32Z")

</div>

Perfect, thanks again

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 28, 2019, 4:19pm UTC](https://discuss.elastic.co/t/how-to-use-pipelines-and-processors/183719/14 "2019-06-28T16:19:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.