# Pipeline simulation

**URL:** <https://discuss.elastic.co/t/pipeline-simulation/304391>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [May 10, 2022, 7:27pm UTC](https://discuss.elastic.co/t/pipeline-simulation/304391 "2022-05-10T19:27:29Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![hebph001](https://avatars.discourse-cdn.com/v4/letter/h/d07c76/32.png) [@hebph001](https://discuss.elastic.co/u/hebph001)\
**Post date:** [May 10, 2022, 7:27pm UTC](https://discuss.elastic.co/t/pipeline-simulation/304391/1 "2022-05-10T19:27:29Z")

</div>

Hi Elastic community,

I'm having a hard time figuring out what is wrong with my pipeline simulation. I'm trying to simulate this way

```auto
GET /_ingest/pipeline/test-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "message": {
          "customer_id" : "e55d52c8-b28e-4611-bcc9-d45d3095d158"
        }
      }
    }
  ]
}

```

And I still get this error. No quotes are missing so why is ES is complaining about it?

```auto
{
  "docs" : [
    {
      "error" : {
        "root_cause" : [
          {
            "type" : "illegal_argument_exception",
            "reason" : """com.fasterxml.jackson.core.JsonParseException: Unexpected character ('c' (code 99)): was expecting double-quote to start field name
 at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 3]"""
          }
        ],
        "type" : "illegal_argument_exception",
        "reason" : """com.fasterxml.jackson.core.JsonParseException: Unexpected character ('c' (code 99)): was expecting double-quote to start field name
 at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 3]""",
        "caused_by" : {
          "type" : "json_parse_exception",
          "reason" : """Unexpected character ('c' (code 99)): was expecting double-quote to start field name
 at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 3]"""
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![RabBit\_BR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rabbit_br/32/82261_2.png) [@RabBit\_BR](https://discuss.elastic.co/u/RabBit_BR)\
**Post date:** [May 10, 2022, 8:04pm UTC](https://discuss.elastic.co/t/pipeline-simulation/304391/2 "2022-05-10T20:04:46Z")

</div>

Hi!  
is it possible to send us your pipeline?

---

<div class="post-metadata">

**Author:** ![hebph001](https://avatars.discourse-cdn.com/v4/letter/h/d07c76/32.png) [@hebph001](https://discuss.elastic.co/u/hebph001)\
**Post date:** [May 10, 2022, 8:19pm UTC](https://discuss.elastic.co/t/pipeline-simulation/304391/3 "2022-05-10T20:19:39Z")

</div>

FYI, I simplified the given example when using \_simulate endpoint. There are more field but even if I add them under "message" object, the same error is showing up.

```auto
[
  {
    "json": {
      "field": "message"
    }
  },
  {
    "date": {
      "field": "message.end",
      "target_field": "@timestamp",
      "formats": [
        "UNIX_MS"
      ],
      "timezone": "UTC",
      "ignore_failure": true
    }
  },
  {
    "remove": {
      "field": "message.customer_id",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.source_info.ip",
      "target_field": "source.ip",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.severity",
      "target_field": "sophos.event.severity",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.endpoint_id",
      "target_field": "sophos.event.endpoint_id",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.endpoint_type",
      "target_field": "sophos.event.endpoint_type",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.type",
      "target_field": "sophos.event.type",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.id",
      "target_field": "sophos.event.id",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.group",
      "target_field": "sophos.event.category",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.name",
      "target_field": "sophos.event.description",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.datastream",
      "target_field": "sophos.event.identifier",
      "ignore_missing": true
    }
  },
  {
    "date": {
      "field": "message.rt",
      "target_field": "sophos.event.created_at",
      "formats": [
        ""
      ],
      "timezone": "UTC",
      "ignore_failure": true
    }
  },
  {
    "rename": {
      "field": "message.suser",
      "target_field": "sophos.event.logged_user",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.dhost",
      "target_field": "source.domain",
      "ignore_missing": true
    }
  }
]

```

Thank you!

---

<div class="post-metadata">

**Author:** ![hebph001](https://avatars.discourse-cdn.com/v4/letter/h/d07c76/32.png) [@hebph001](https://discuss.elastic.co/u/hebph001)\
**Post date:** [May 10, 2022, 8:21pm UTC](https://discuss.elastic.co/t/pipeline-simulation/304391/4 "2022-05-10T20:21:04Z")

</div>

The source logs look like this:

```auto
{"endpoint_id": "524335a2-3937-4ae1-957e-15fa81d0c3a4", "endpoint_type": "server", "source_info": {"ip": "192.168.165.67"}, "customer_id": "e55d52c8-b28e-4611-bcc9-d45d3095d158", "severity": "low", "type": "Event::Endpoint::UpdateSuccess", "id": "d0f721ff-0dd9-4bc6-9714-d1c1be8f0996", "group": "UPDATING", "name": "Update succeeded", "datastream": "event", "end": "2022-04-13T14:50:27.635Z", "rt": "2022-04-13T14:50:27.649Z", "suser": "n/a", "dhost": "cfzsso01"}

```

---

<div class="post-metadata">

**Author:** ![RabBit\_BR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rabbit_br/32/82261_2.png) [@RabBit\_BR](https://discuss.elastic.co/u/RabBit_BR)\
**Post date:** [May 10, 2022, 10:31pm UTC](https://discuss.elastic.co/t/pipeline-simulation/304391/5 "2022-05-10T22:31:00Z")

</div>

The error occur here:x

> [@hebph001](#):
>
> ```auto
> {
> "json": {
> "field": "message"
> }
> }
> 
> ```

Docs:

> **[JSON processor | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/json-processor.html)**

Example:

```auto
GET /_ingest/pipeline/my-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "message": {
          "customer_id": """{"foo": 2000}"""
        }
      }
    }
  ]
}

PUT _ingest/pipeline/my-pipeline
{
  "description": "My optional pipeline description",
  "processors": [
    {
      "json": {
        "field": "message.customer_id"
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![hebph001](https://avatars.discourse-cdn.com/v4/letter/h/d07c76/32.png) [@hebph001](https://discuss.elastic.co/u/hebph001)\
**Post date:** [May 11, 2022, 12:33pm UTC](https://discuss.elastic.co/t/pipeline-simulation/304391/6 "2022-05-11T12:33:15Z")

</div>

Thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 8, 2022, 12:34pm UTC](https://discuss.elastic.co/t/pipeline-simulation/304391/7 "2022-06-08T12:34:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.