# Is it possible to use the ingestion plugin in a bulk request?

**URL:** <https://discuss.elastic.co/t/is-it-possible-to-use-the-ingestion-plugin-in-a-bulk-request/82481>\
**Category:** Elasticsearch\
**Created:** [April 15, 2017, 6:46pm UTC](https://discuss.elastic.co/t/is-it-possible-to-use-the-ingestion-plugin-in-a-bulk-request/82481 "2017-04-15T18:46:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![apanimesh061](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/apanimesh061/32/927_2.png) [@apanimesh061](https://discuss.elastic.co/u/apanimesh061)\
**Post date:** [April 15, 2017, 6:46pm UTC](https://discuss.elastic.co/t/is-it-possible-to-use-the-ingestion-plugin-in-a-bulk-request/82481/1 "2017-04-15T18:46:16Z")

</div>

I have a total of around 500,000 documents that was earlier inserting using bulk request of 1000 at a time.  
But before I had an ingestion plugin I using a library externally to transform the input and then collect them for a bulk insertion.

Now that I have an ingestion plugin for my purpose, how can I use it to perform bulk document transforms and then send the documents for a bulk insertion?

I am using Elasticsearch `5.2.1`.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 15, 2017, 7:27pm UTC](https://discuss.elastic.co/t/is-it-possible-to-use-the-ingestion-plugin-in-a-bulk-request/82481/2 "2017-04-15T19:27:05Z")

</div>

Just add `?pipeline=your_pipeline` to the bulk request.

---

<div class="post-metadata">

**Author:** ![apanimesh061](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/apanimesh061/32/927_2.png) [@apanimesh061](https://discuss.elastic.co/u/apanimesh061)\
**Post date:** [April 15, 2017, 8:21pm UTC](https://discuss.elastic.co/t/is-it-possible-to-use-the-ingestion-plugin-in-a-bulk-request/82481/3 "2017-04-15T20:21:19Z")

</div>

@dadoonet Thanks for the response.

I used this command to ingest a document using a pipeline I created:

```
curl -X PUT 'http://localhost:9200/test_index/review/1?pipeline=apply-vader-review' -d '{
	"content": "The plot was good, but the characters are uncompelling and the dialog is not great."
}'

```

The ingested document looks like this:

```
{
    "_index":"test_index",
    "_type":"review",
    "_id":"1",
    "_score":1,
    "_source":{
        "content":"The plot was good, but the characters are uncompelling and the dialog is not great.",
        "polarity":{
            "negative":0.327,
            "neutral":0.579,
            "positive":0.094,
            "compound":-0.7042
        }
    }
}

```

I see that I have to give an `_id` for a request. How will I follow this format if I want the document id to be auto-generated during a bulk ingestion?

---

<div class="post-metadata">

**Author:** ![apanimesh061](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/apanimesh061/32/927_2.png) [@apanimesh061](https://discuss.elastic.co/u/apanimesh061)\
**Post date:** [April 15, 2017, 9:05pm UTC](https://discuss.elastic.co/t/is-it-possible-to-use-the-ingestion-plugin-in-a-bulk-request/82481/4 "2017-04-15T21:05:24Z")

</div>

@dadoonet

I figured it out. This is `_bulk` request I tried and was successful.

```
curl -X POST 'http://localhost:9200/test_index/review/_bulk?pipeline=apply-vader-review' -d 
'{ "index" : { "_index" : "test_index", "_type" : "review"} }
{"content": "The plot was good, but the characters are uncompelling and the dialog is not great."}
{ "index" : { "_index" : "test_index", "_type" : "review"} }
{"content": "The plot was good, but the characters are uncompelling and the dialog is not great."}
{ "index" : { "_index" : "test_index", "_type" : "review"} }
{"content": "The plot was good, but the characters are uncompelling and the dialog is not great."}' 

```

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 13, 2017, 9:08pm UTC](https://discuss.elastic.co/t/is-it-possible-to-use-the-ingestion-plugin-in-a-bulk-request/82481/5 "2017-05-13T21:08:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.