Pipeline simulation

Elastic Stack Elasticsearch
1

Hi Elastic community,

I'm having a hard time figuring out what is wrong with my pipeline simulation. I'm trying to simulate this way

GET /_ingest/pipeline/test-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "message": {
          "customer_id" : "e55d52c8-b28e-4611-bcc9-d45d3095d158"
        }
      }
    }
  ]
}

And I still get this error. No quotes are missing so why is ES is complaining about it?

{
  "docs" : [
    {
      "error" : {
        "root_cause" : [
          {
            "type" : "illegal_argument_exception",
            "reason" : """com.fasterxml.jackson.core.JsonParseException: Unexpected character ('c' (code 99)): was expecting double-quote to start field name
 at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 3]"""
          }
        ],
        "type" : "illegal_argument_exception",
        "reason" : """com.fasterxml.jackson.core.JsonParseException: Unexpected character ('c' (code 99)): was expecting double-quote to start field name
 at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 3]""",
        "caused_by" : {
          "type" : "json_parse_exception",
          "reason" : """Unexpected character ('c' (code 99)): was expecting double-quote to start field name
 at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 3]"""
        }
      }
    }
  ]
}
2

Hi!
is it possible to send us your pipeline?

3

FYI, I simplified the given example when using _simulate endpoint. There are more field but even if I add them under "message" object, the same error is showing up.

[
  {
    "json": {
      "field": "message"
    }
  },
  {
    "date": {
      "field": "message.end",
      "target_field": "@timestamp",
      "formats": [
        "UNIX_MS"
      ],
      "timezone": "UTC",
      "ignore_failure": true
    }
  },
  {
    "remove": {
      "field": "message.customer_id",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.source_info.ip",
      "target_field": "source.ip",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.severity",
      "target_field": "sophos.event.severity",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.endpoint_id",
      "target_field": "sophos.event.endpoint_id",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.endpoint_type",
      "target_field": "sophos.event.endpoint_type",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.type",
      "target_field": "sophos.event.type",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.id",
      "target_field": "sophos.event.id",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.group",
      "target_field": "sophos.event.category",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.name",
      "target_field": "sophos.event.description",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.datastream",
      "target_field": "sophos.event.identifier",
      "ignore_missing": true
    }
  },
  {
    "date": {
      "field": "message.rt",
      "target_field": "sophos.event.created_at",
      "formats": [
        ""
      ],
      "timezone": "UTC",
      "ignore_failure": true
    }
  },
  {
    "rename": {
      "field": "message.suser",
      "target_field": "sophos.event.logged_user",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "message.dhost",
      "target_field": "source.domain",
      "ignore_missing": true
    }
  }
]

Thank you!

4

The source logs look like this:

{"endpoint_id": "524335a2-3937-4ae1-957e-15fa81d0c3a4", "endpoint_type": "server", "source_info": {"ip": "192.168.165.67"}, "customer_id": "e55d52c8-b28e-4611-bcc9-d45d3095d158", "severity": "low", "type": "Event::Endpoint::UpdateSuccess", "id": "d0f721ff-0dd9-4bc6-9714-d1c1be8f0996", "group": "UPDATING", "name": "Update succeeded", "datastream": "event", "end": "2022-04-13T14:50:27.635Z", "rt": "2022-04-13T14:50:27.649Z", "suser": "n/a", "dhost": "cfzsso01"}
5

The error occur here:x

Docs:

Example:

GET /_ingest/pipeline/my-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "message": {
          "customer_id": """{"foo": 2000}"""
        }
      }
    }
  ]
}

PUT _ingest/pipeline/my-pipeline
{
  "description": "My optional pipeline description",
  "processors": [
    {
      "json": {
        "field": "message.customer_id"
      }
    }
  ]
}
6

Thank you!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.