# Ingest pipeline to extract data

**URL:** <https://discuss.elastic.co/t/ingest-pipeline-to-extract-data/183516>\
**Category:** Elasticsearch\
**Created:** [May 30, 2019, 11:47am UTC](https://discuss.elastic.co/t/ingest-pipeline-to-extract-data/183516 "2019-05-30T11:47:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![nicks1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicks1993/32/84627_2.png) [@nicks1993](https://discuss.elastic.co/u/nicks1993)\
**Post date:** [May 30, 2019, 11:47am UTC](https://discuss.elastic.co/t/ingest-pipeline-to-extract-data/183516/1 "2019-05-30T11:47:38Z")

</div>

if I want to extract the numeric duration value from  
words::words::words::words (duration=432, words)

would this grok pipeline work  
PUT \_ingest/pipeline/parse  
{  
"description" : "parses the duration field",  
"processors" : [  
{  
"grok": {  
"field": "message",  
"patterns": ["%{NUMBER:duration}"]  
}  
}  
]  
}

Would this parse the value and store it in a new field or would I have to create the new field first? Or is there any issues that you notice with it?

Is there anything else I have to do to run this pipeline now that it is created?

---

<div class="post-metadata">

**Author:** ![nicks1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicks1993/32/84627_2.png) [@nicks1993](https://discuss.elastic.co/u/nicks1993)\
**Post date:** [May 30, 2019, 2:40pm UTC](https://discuss.elastic.co/t/ingest-pipeline-to-extract-data/183516/2 "2019-05-30T14:40:48Z")

</div>

I know on the ingest node page it says to run the pipeline like this  
PUT my-index/\_doc/my-id?pipeline=my\_pipeline\_id  
{  
"foo": "bar"  
}

but I don't understand the need of "foo":"bar" my pipeline should just extract duration out of that message field and store it in a new field shouldn't it?

---

<div class="post-metadata">

**Author:** ![gbrown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gbrown/32/34482_2.png) [@gbrown](https://discuss.elastic.co/u/gbrown)\
**Post date:** [May 31, 2019, 4:29pm UTC](https://discuss.elastic.co/t/ingest-pipeline-to-extract-data/183516/3 "2019-05-31T16:29:08Z")

</div>

> [@nicks1993](#):
>
> if I want to extract the numeric duration value from  
> words::words::words::words (duration=432, words)
> 
> [...snip...]
> 
> Would this parse the value and store it in a new field or would I have to create the new field first? Or is there any issues that you notice with it?

It may be helpful to use the [Simulate Pipeline API](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html) to try out your pipeline to see what the result will be, without actually indexing any documents, like so:

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "this is an example pipeline",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": ["%{NUMBER:duration}"]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": "words::words::words::words (duration=432, words)"
      }
    }
  ]
}

```

In this case, this should return a response like:

```auto
{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "duration" : "432",
          "message" : "words::words::words::words (duration=432, words)"
        },
        "_ingest" : {
          "timestamp" : "2019-05-31T16:21:53.147Z"
        }
      }
    }
  ]
}

```

We can see that the number is added to the document as `duration`, because that's the name we used in the grok pattern. However, note that the given grok pattern will just extract the first number in the `message` field, so if you used a document with `"message": "words::words::87::words (duration=432, words)"`, the extracted `duration` would be `87`.

To make this a little more resilient, you can use regular expressions in your grok pattern, like this: `".*duration=%{NUMBER:duration}.*"`. That pattern would extract `432` rather than `87` from the second example message.

> [@nicks1993](#):
>
> Is there anything else I have to do to run this pipeline now that it is created?

To use the pipeline, just index documents as normal, but with the `pipeline=my_pipeline_id` parameter on the request, like in your second post.

> [@nicks1993](#):
>
> but I don't understand the need of "foo":"bar" my pipeline should just extract duration out of that message field and store it in a new field shouldn't it?

The `"foo": "bar"` is just an example document, it's not required or anything - just replace it with the document you want to run through the pipeline and index.

---

<div class="post-metadata">

**Author:** ![nicks1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicks1993/32/84627_2.png) [@nicks1993](https://discuss.elastic.co/u/nicks1993)\
**Post date:** [May 31, 2019, 4:46pm UTC](https://discuss.elastic.co/t/ingest-pipeline-to-extract-data/183516/4 "2019-05-31T16:46:13Z")

</div>

Nice thanks I was actually trying to make a better expression but I was getting parsing errors, guess I need to brush up on my regex.  
As for using the pipeline I believe I'll have to add the pipeline name to fluentd as I don't ingest documents manually here.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 28, 2019, 4:46pm UTC](https://discuss.elastic.co/t/ingest-pipeline-to-extract-data/183516/5 "2019-06-28T16:46:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.