# Issue with Pipeline Ordering: Custom Flattening Pipeline Running After ML Inference Pipeline

**URL:** <https://discuss.elastic.co/t/issue-with-pipeline-ordering-custom-flattening-pipeline-running-after-ml-inference-pipeline/374225>\
**Category:** Elastic Search\
**Tags:** painless, connectors\
**Created:** [February 7, 2025, 1:27pm UTC](https://discuss.elastic.co/t/issue-with-pipeline-ordering-custom-flattening-pipeline-running-after-ml-inference-pipeline/374225 "2025-02-07T13:27:05Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![flalar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flalar/32/85777_2.png) [@flalar](https://discuss.elastic.co/u/flalar)\
**Post date:** [February 7, 2025, 1:27pm UTC](https://discuss.elastic.co/t/issue-with-pipeline-ordering-custom-flattening-pipeline-running-after-ml-inference-pipeline/374225/1 "2025-02-07T13:27:05Z")

</div>

Hi everyone,

We are indexing our GitHub repositories to build our AI Assistant Knowledge Base. Our current setup only allows ML inference on text fields located at the document root, yet many key fields—such as review comments and issue comments—are nested within JSON objects.

To address this, we created a custom ingest pipeline called `search-corp-github@custom`. This pipeline uses a Painless script processor to flatten the document by concatenating all relevant fields (with context labels) into a single field.

However, we’re encountering an issue where the ML inference pipeline (`search-corp-github@ml-inference`) appears to execute _before_ our custom pipeline. Consequently, the ML inference processor doesn’t see the flattened field because it hasn’t been created when the inference runs.

Is there a way to control or adjust the execution order between these pipelines via the Search Connector UI? Should I chain these pipelines together using a parent pipeline, or is it acceptable (or even recommended) to modify the managed base pipeline that defines the overall pipeline order—even if that generates a warning?

Alternatively, are there other recommended approaches to ensure that the ML inference processor sees all the necessary information in the document?

Any insights or best practices would be greatly appreciated!

Thanks

---

<div class="post-metadata">

**Author:** ![Jedr\_Blaszyk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jedr_blaszyk/32/127637_2.png) [@Jedr\_Blaszyk](https://discuss.elastic.co/u/Jedr_Blaszyk)\
**Post date:** [February 24, 2025, 5:06pm UTC](https://discuss.elastic.co/t/issue-with-pipeline-ordering-custom-flattening-pipeline-running-after-ml-inference-pipeline/374225/2 "2025-02-24T17:06:18Z")

</div>

Hey @flalar It's expected that the `@ml-inference` sub-pipeline runs before the `@custom` sub-pipeline, because we wanted to let users be able to post-process the outputs of their embeddings.

If you want to do other pre-processing, they can just manually modify the contents of their `@ml-inference` sub-pipeline to add more processors at the front of the list, before the inference processors.

Here is supporting documentation: [Ingest pipelines in Search | Elasticsearch Guide [8.17] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.17/ingest-pipeline-search.html#ingest-pipeline-search-details-specific)

Let me know if you have more questions!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 24, 2025, 5:06pm UTC](https://discuss.elastic.co/t/issue-with-pipeline-ordering-custom-flattening-pipeline-running-after-ml-inference-pipeline/374225/3 "2025-03-24T17:06:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.