# How to replace multiple newlines with two newlines in ingest pipeline gsub

**URL:** <https://discuss.elastic.co/t/how-to-replace-multiple-newlines-with-two-newlines-in-ingest-pipeline-gsub/350570>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [January 8, 2024, 11:04am UTC](https://discuss.elastic.co/t/how-to-replace-multiple-newlines-with-two-newlines-in-ingest-pipeline-gsub/350570 "2024-01-08T11:04:19Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bowfish](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bowfish/32/130394_2.png) [@Bowfish](https://discuss.elastic.co/u/Bowfish)\
**Post date:** [January 8, 2024, 11:04am UTC](https://discuss.elastic.co/t/how-to-replace-multiple-newlines-with-two-newlines-in-ingest-pipeline-gsub/350570/1 "2024-01-08T11:04:19Z")

</div>

I want to replace multiple (more than 3) newlines (\n\n\n) with two newlines (\n\n). If I set "\n\n" as a replacement string the the gsub object it replaces \n\n\n\ with nn.

Here you can find my \_simulate ingest pipeline.

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "gsub": {
          "field": "message",
          "pattern": """Page \d of \d""",
          "replacement": "",
          "ignore_missing": false,
          "description": "Remove Page x of x",
          "on_failure": [
            {
              "append": {
                "description": "Record error information",
                "field": "_ingestion_errors",
                "value": "Processor 'gsub' with tag 'remove_page_numbers' in pipeline '{{ _ingest.on_failure_pipeline }}' failed with message '{{ _ingest.on_failure_message }}'"
              }
            }
          ]
        }
      },
            {
        "gsub": {
          "field": "message",
          "pattern": "\\n\\n",
          "replacement": "\\n",
          "ignore_missing": false,
          "description": "Replace multiple newlines at the beginning",
          "on_failure": [
            {
              "append": {
                "description": "Record error information",
                "field": "_ingestion_errors",
                "value": "Processor 'gsub' with tag 'remove_page_numbers' in pipeline '{{ _ingest.on_failure_pipeline }}' failed with message '{{ _ingest.on_failure_message }}'"
              }
            }
          ]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """

Marketing Intern
May 2009 - August 2009 (4 months)
New York City Metropolitan Area

Jump PR
Public Relations Intern

  Page 2 of 3

   

May 2008 - August 2008 (4 months)
New York City Metropolitan Area

Education
University
Bachelor of Arts - BA, Communication and Media Studies · (August 2010 - May
2014)

Senior High School
 · (September 2006 - June 2010)

  Page 3 of 3"""
      }
    }
  ]
}

```

How does the replacement string have to look like that it that it set \n\n?

Thanks for you help.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 8, 2024, 3:15pm UTC](https://discuss.elastic.co/t/how-to-replace-multiple-newlines-with-two-newlines-in-ingest-pipeline-gsub/350570/2 "2024-01-08T15:15:01Z")

</div>

Hi @Bowfish

I thought we answered this here

Please Do not Open multiple topics with the same question.

Is there something different here

> [@How to replace multiple new lines with one in Ingest Pipeline gsub](https://discuss.elastic.co/t/how-to-replace-multiple-new-lines-with-one-in-ingest-pipeline-gsub/350127/4):
>
> Now it is just a regex exercise I am not a regex expert... but I did a quick google search found [this](https://stackoverflow.com/a/701658) POST \_ingest/pipeline/\_simulate { "pipeline": { "processors": [ { "gsub": { "field": "message", "pattern": "\\\n+", "replacement": "\\\n", "ignore\_missing": false, "description": "Replace multiple newlines", "on\_failure": [ { "append": { "description": "Record error informa…

---

<div class="post-metadata">

**Author:** ![Bowfish](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bowfish/32/130394_2.png) [@Bowfish](https://discuss.elastic.co/u/Bowfish)\
**Post date:** [January 9, 2024, 8:49am UTC](https://discuss.elastic.co/t/how-to-replace-multiple-newlines-with-two-newlines-in-ingest-pipeline-gsub/350570/3 "2024-01-09T08:49:50Z")

</div>

@stephenb There is indeed something different with this post. The question here is how the string of gsub.replacement has to look like that it replaces the the string from the gsub.patter with `\n` .

If i run this ingest pipeline simulate

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "gsub": {
          "description": "Remove unicode no break space",
          "field": "message",
          "ignore_missing": true,
          "pattern": " ",
          "replacement": "",
          "tag": "replace_unicode_bullets",
          "on_failure": [
            {
              "append": {
                "description": "Record error information",
                "field": "_ingestion_errors",
                "value": "Processor 'gsub' with tag 'replace_unicode_bullets' in pipeline '{{ _ingest.on_failure_pipeline }}' failed with message '{{ _ingest.on_failure_message }}'"
              }
            }
          ]
        }
      },
      {
        "gsub": {
          "description": "Remove Page x of y",
          "field": "message",
          "ignore_missing": true,
          "pattern": "Page \\d of \\d",
          "replacement": "",
          "tag": "replace_unicode_bullets",
          "on_failure": [
            {
              "append": {
                "description": "Record error information",
                "field": "_ingestion_errors",
                "value": "Processor 'gsub' with tag 'replace_unicode_bullets' in pipeline '{{ _ingest.on_failure_pipeline }}' failed with message '{{ _ingest.on_failure_message }}'"
              }
            }
          ]
        }
      },
      {
        "gsub": {
          "field": "message",
          "pattern": """(\\n){3,}""",
          "replacement": """\\n\\n""",
          "ignore_missing": false,
          "description": "Replace multiple newlines",
          "on_failure": [
            {
              "append": {
                "description": "Record error information",
                "field": "_ingestion_errors",
                "value": "Processor 'gsub' with tag 'replace_multiple_newlines' in pipeline '{{ _ingest.on_failure_pipeline }}' failed with message '{{ _ingest.on_failure_message }}'"
              }
            }
          ]
        }
      },
      {
        "gsub": {
          "description": "Replace unicode bulltest with dashes",
          "field": "message",
          "ignore_missing": true,
          "pattern": "•",
          "replacement": "-",
          "tag": "replace_unicode_bullets",
          "on_failure": [
            {
              "append": {
                "description": "Record error information",
                "field": "_ingestion_errors",
                "value": "Processor 'gsub' with tag 'replace_unicode_bullets' in pipeline '{{ _ingest.on_failure_pipeline }}' failed with message '{{ _ingest.on_failure_message }}'"
              }
            }
          ]
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": """
•Coordinated Scholarship event at Island\n\nEducation\n\n Page 2 of 3\n\n\n\n \nCollege (MA)\nBachelor's degree, English Studies and Journalism · (2010 - 2013)\n\nNew York University\nContinuing Studies, Graphic Design · (2014 - 2016)\n\nUniversity de Córdoba\nHispanic and Englis Languages, Literatures, and Linguistics,\nGeneral · (2012 - 2012)\n\nMiami College\nAssociate of Arts (A.A.), Mass Communication/Media Studies · (2008 - 2010)\n\nWheaton College (MA)\nBachelor of Arts (B.A.) · (2010 - 2012)\n\n Page 3 of 3"""
      }
    }
  ]
}

```

it replaces `\n\n\n` with `\n`, which is correct.

But if I save the same gsub like this:

```auto
"gsub": {
  "field": "message",
  "pattern": "(\\n){3,}",
  "replacement": "\\n\\n",
  "ignore_missing": false,
  "description": "Replace multiple newlines",
  "on_failure": [
    {
      "append": {
        "description": "Record error information",
        "field": "_ingestion_errors",
        "value": "Processor 'gsub' with tag 'replace_multiple_newlines' in pipeline '{{ _ingest.on_failure_pipeline }}' failed with message '{{ _ingest.on_failure_message }}'"
      }
    }
  ]
}

```

in an ingest pipeline and use this ingest pipline when uploading documents, the pattern is replaced with `n`and not `\n` . How does the replacement string have to look like that `\n` is inserted?

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 9, 2024, 3:11pm UTC](https://discuss.elastic.co/t/how-to-replace-multiple-newlines-with-two-newlines-in-ingest-pipeline-gsub/350570/4 "2024-01-09T15:11:36Z")

</div>

Sorry I'm not sure I am following. Are you saying the simulate works but when you actually `PUT` the pipeline in an ingest documents it does not work?

Is that correct?

Side note, probably could have kept this in the same thread

---

<div class="post-metadata">

**Author:** ![Bowfish](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bowfish/32/130394_2.png) [@Bowfish](https://discuss.elastic.co/u/Bowfish)\
**Post date:** [January 9, 2024, 3:53pm UTC](https://discuss.elastic.co/t/how-to-replace-multiple-newlines-with-two-newlines-in-ingest-pipeline-gsub/350570/5 "2024-01-09T15:53:42Z")

</div>

Yes. That is correct.

---

<div class="post-metadata">

**Author:** ![Bowfish](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bowfish/32/130394_2.png) [@Bowfish](https://discuss.elastic.co/u/Bowfish)\
**Post date:** [January 9, 2024, 3:54pm UTC](https://discuss.elastic.co/t/how-to-replace-multiple-newlines-with-two-newlines-in-ingest-pipeline-gsub/350570/6 "2024-01-09T15:54:20Z")

</div>

> [@stephenb](#):
>
> Side note, probably could have kept this in the same thread

Thanks for the note. Next time I will keep related questions in the same thread.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 9, 2024, 3:56pm UTC](https://discuss.elastic.co/t/how-to-replace-multiple-newlines-with-two-newlines-in-ingest-pipeline-gsub/350570/7 "2024-01-09T15:56:59Z")

</div>

Can you show the exact input and output you desire?

Also I am confused...

You can PUT exactly what you \_simulate with ... why are you changing the syntax?

From this `_simulate`

```auto
          "field": "message",
          "pattern": """(\\n){3,}""",
          "replacement": """\\n\\n""",

```

to this `PUT`

```auto
  "pattern": "(\\n){3,}",
  "replacement": "\\n\\n",
  "ignore_missing": false,

```

Why are you doing that?

When i `PUT` the exact syntax in your simulate I get the exact same output from a simulate as I do when I actually post a document with the pipeline

`_simulate`

```auto
          "message": """
-Coordinated Scholarship event at Island\n\nEducation\n\n \nCollege (MA)\nBachelor's degree, English Studies and Journalism · (2010 - 2013)\n\nNew York University\nContinuing Studies, Graphic Design · (2014 - 2016)\n\nUniversity de Córdoba\nHispanic and Englis Languages, Literatures, and Linguistics,\nGeneral · (2012 - 2012)\n\nMiami College\nAssociate of Arts (A.A.), Mass Communication/Media Studies · (2008 - 2010)\n\nWheaton College (MA)\nBachelor of Arts (B.A.)· (2010 - 2012)\n\n"""
        },

```

`POST` and `_search`

```auto
POST discuss-test/_doc/?pipeline=discuss-test
  {
        "message": """
•Coordinated Scholarship event at Island\n\nEducation\n\n Page 2 of 3\n\n\n\n \nCollege (MA)\nBachelor's degree, English Studies and Journalism · (2010 - 2013)\n\nNew York University\nContinuing Studies, Graphic Design · (2014 - 2016)\n\nUniversity de Córdoba\nHispanic and Englis Languages, Literatures, and Linguistics,\nGeneral · (2012 - 2012)\n\nMiami College\nAssociate of Arts (A.A.), Mass Communication/Media Studies · (2008 - 2010)\n\nWheaton College (MA)\nBachelor of Arts (B.A.) · (2010 - 2012)\n\n Page 3 of 3"""
      }

GET discuss-test/_search

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "discuss-test",
        "_id": "3Mbt7owByr6kya5vpOvZ",
        "_score": 1,
        "_ignored": [
          "message.keyword"
        ],
        "_source": {
          "message": """
-Coordinated Scholarship event at Island\n\nEducation\n\n \nCollege (MA)\nBachelor's degree, English Studies and Journalism · (2010 - 2013)\n\nNew York University\nContinuing Studies, Graphic Design · (2014 - 2016)\n\nUniversity de Córdoba\nHispanic and Englis Languages, Literatures, and Linguistics,\nGeneral · (2012 - 2012)\n\nMiami College\nAssociate of Arts (A.A.), Mass Communication/Media Studies · (2008 - 2010)\n\nWheaton College (MA)\nBachelor of Arts (B.A.)· (2010 - 2012)\n\n"""
        }
      }
    ]
  }
}

```

Here they are side by side

```auto
_simlutate

-Coordinated Scholarship event at Island\n\nEducation\n\n \nCollege (MA)\nBachelor's degree, English Studies and Journalism · (2010 - 2013)\n\nNew York University\nContinuing Studies, Graphic Design · (2014 - 2016)\n\nUniversity de Córdoba\nHispanic and Englis Languages, Literatures, and Linguistics,\nGeneral · (2012 - 2012)\n\nMiami College\nAssociate of Arts (A.A.), Mass Communication/Media Studies · (2008 - 2010)\n\nWheaton College (MA)\nBachelor of Arts (B.A.)· (2010 - 2012)\n\n"""

POST and _search

-Coordinated Scholarship event at Island\n\nEducation\n\n \nCollege (MA)\nBachelor's degree, English Studies and Journalism · (2010 - 2013)\n\nNew York University\nContinuing Studies, Graphic Design · (2014 - 2016)\n\nUniversity de Córdoba\nHispanic and Englis Languages, Literatures, and Linguistics,\nGeneral · (2012 - 2012)\n\nMiami College\nAssociate of Arts (A.A.), Mass Communication/Media Studies · (2008 - 2010)\n\nWheaton College (MA)\nBachelor of Arts (B.A.)· (2010 - 2012)\n\n

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 6, 2024, 3:58pm UTC](https://discuss.elastic.co/t/how-to-replace-multiple-newlines-with-two-newlines-in-ingest-pipeline-gsub/350570/8 "2024-02-06T15:58:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.