# Ingest Pipeline processing issue in for loop

**URL:** <https://discuss.elastic.co/t/ingest-pipeline-processing-issue-in-for-loop/369954>\
**Category:** Elasticsearch\
**Created:** [November 3, 2024, 6:31am UTC](https://discuss.elastic.co/t/ingest-pipeline-processing-issue-in-for-loop/369954 "2024-11-03T06:31:13Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jay\_Timbadia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jay_timbadia/32/86572_2.png) [@Jay\_Timbadia](https://discuss.elastic.co/u/Jay_Timbadia)\
**Post date:** [November 3, 2024, 6:31am UTC](https://discuss.elastic.co/t/ingest-pipeline-processing-issue-in-for-loop/369954/1 "2024-11-03T06:31:13Z")

</div>

I am running ingest pipeline in loop for an array of PDF files.

I am creating new index for each file. But sometimes the pipeline gives 201 and sometimes it gives 200.

In 201, it just creates the index but does not process it.  
Any reason what is the problem here?

Code:

```auto
for idx, f in enumerate(target_files):
    message = ""
    pdf_filename = f.name
    print(pdf_filename)
    response = File.open_binary(ctx, folder_url + pdf_filename) # sharepoint read

    encoded = base64.b64encode(response.content).decode('utf-8')
    payload = {
        "data": encoded
    }

    index_name = re.sub(r'[\W_]+', '_', pdf_filename.lower())

    url = f"http://localhost:9200/{index_name}/_doc/1?pipeline=my-pipeline"

    response = requests.post(url, json=payload)

 
    print(response.status_code) # sometimes it gives 200, sometimes 201.

    if es.indices.exists(index=index_name):
        print("index created by ingest pipeline!")
    
    result = es.search(index=index_name)
    result = result["hits"]["hits"] # when 201, its empty, if 200, its processed.
    print(len(result))

    output_text = result[0]["_source"]["attachment"]["content"]

    es.indices.delete(index=index_name)

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 3, 2024, 6:38am UTC](https://discuss.elastic.co/t/ingest-pipeline-processing-issue-in-for-loop/369954/2 "2024-11-03T06:38:33Z")

</div>

> [@Jay\_Timbadia](#):
>
> I am creating new index for each file.

Why would you do this?? That sounds crazy to me.

---

<div class="post-metadata">

**Author:** ![Jay\_Timbadia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jay_timbadia/32/86572_2.png) [@Jay\_Timbadia](https://discuss.elastic.co/u/Jay_Timbadia)\
**Post date:** [November 3, 2024, 6:49am UTC](https://discuss.elastic.co/t/ingest-pipeline-processing-issue-in-for-loop/369954/3 "2024-11-03T06:49:11Z")

</div>

If I am overwriting in single one, then the results are getting very weird.

That's why I did tried all the approaches.

```auto
flag = False
count = 0

url = f"http://localhost:9200/pdf_index/_doc/1?pipeline=my-pipeline"

final_data = []
for idx, f in enumerate(target_files):
    message = ""
    pdf_filename = f.name
    print(pdf_filename)
    response = File.open_binary(ctx, folder_url + pdf_filename)

    encoded = base64.b64encode(response.content).decode('utf-8')
    payload = {
        "data": encoded
    }

    response = requests.post(url, json=payload)
    
    result = es.search(index=index_name)
    result = result["hits"]["hits"]
    print(len(result))

    output_text = result[0]["_source"]["attachment"]["content"]    
    print(output_text[:100])

```

output:

```auto
test.pdf
1
OIL-INJECTED ROTARY SCREW COMPRESSORS GA 90+-160/GA 110-160 VSD (90-160 kW/125-200 hp)

test1.pdf
1
OIL-INJECTED ROTARY SCREW COMPRESSORS GA 90+-160/GA 110-160 VSD (90-160 kW/125-200 hp)

```

you can see the fine print "print(output\_text[:100])" is same for both the files, which shouldn't be.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 3, 2024, 6:54am UTC](https://discuss.elastic.co/t/ingest-pipeline-processing-issue-in-for-loop/369954/4 "2024-11-03T06:54:52Z")

</div>

> [@Jay\_Timbadia](#):
>
> If I am overwriting in single one, then the results are getting very weird

Indices are like databases so you should not write a single document to each as that will not scale well at all. Instead write all documents to a single index as that is how you are supposed to use Elasticsearch and indices with multiple primary shards can be terabytes in size.

If you are seeing issues when overwriting documents, describe exactlky what those are and try to resolve them.

"the results are getting very weird" is very vague. You need to describe the issues in a lot more detail if someone is going to be able to help.

---

<div class="post-metadata">

**Author:** ![Jay\_Timbadia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jay_timbadia/32/86572_2.png) [@Jay\_Timbadia](https://discuss.elastic.co/u/Jay_Timbadia)\
**Post date:** [November 3, 2024, 7:00am UTC](https://discuss.elastic.co/t/ingest-pipeline-processing-issue-in-for-loop/369954/5 "2024-11-03T07:00:59Z")

</div>

I have pasted a screenshot of the result which to any proper english reader will make sense of weirdness.

The length of the output is 1. And the content is showing same for both the files after processing by ingest pipeline, which should not be the case, since file names are different. (test.pdf & test1.pdf) and so are its content.

The output content

```auto
OIL-INJECTED ROTARY SCREW COMPRESSORS GA 90+-160/GA 110-160 VSD (90-160 kW/125-200 hp)

```

is from test.pdf but test2.pdf has different content, but its showing similar to first file.

To your point of dumping all files to single index, it may or may not work, I have not tried, but the code I have written above is also not incorrect.

It should also work.  
There can be many soln to final output, but all should work, right?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 3, 2024, 7:15am UTC](https://discuss.elastic.co/t/ingest-pipeline-processing-issue-in-for-loop/369954/6 "2024-11-03T07:15:38Z")

</div>

> [@Jay\_Timbadia](#):
>
> I have pasted a screenshot of the result which to any proper english reader will make sense of weirdness.

I find this quite rude. Given that I am volunteering here I do not think I want to spend any more time on this thread. Good luck.

---

<div class="post-metadata">

**Author:** ![Jay\_Timbadia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jay_timbadia/32/86572_2.png) [@Jay\_Timbadia](https://discuss.elastic.co/u/Jay_Timbadia)\
**Post date:** [November 3, 2024, 7:38am UTC](https://discuss.elastic.co/t/ingest-pipeline-processing-issue-in-for-loop/369954/7 "2024-11-03T07:38:13Z")

</div>

Thanks!