Parse ingest pipeline _simulate

Elastic Stack Elasticsearch
1

I'm fighting against an ingest pipeline which might not work.
The solution must be simple but I lost track.

The following simulation don't work. The Dev Tools in Kibana throws a "bad string" error. But why?

POST _ingest/pipeline/lasernet/_simulate
{
  "docs": [
    {
      "_source": {SAG-DFO-015;2019-04-10 13:18:45.846;Pass-Through - Destination selection;10;6368;1052522;SAG_DFO_015_DESTINATION_SELECTION_E6823FFA_3E0B_4E26_98C9_A62B4F11B6F4;Applying pre-processing module jobinfos to SAG_DFO_015_DESTINATION_SELECTION_E6823FFA_3E0B_4E26_98C9_A62B4F11B6F4}
    }
  ]
}

When I add a " character in "_source":{" the bad string error disappear but get the below error...

POST _ingest/pipeline/lasernet/_simulate
{
  "docs": [
    {
      "_source": {"SAG-DFO-015;2019-04-10 13:18:45.846;Pass-Through - Destination selection;10;6368;1052522;SAG_DFO_015_DESTINATION_SELECTION_E6823FFA_3E0B_4E26_98C9_A62B4F11B6F4;Applying pre-processing module jobinfos to SAG_DFO_015_DESTINATION_SELECTION_E6823FFA_3E0B_4E26_98C9_A62B4F11B6F4}
    }
  ]
}

{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "Failed to parse content to map"
      }
    ],
    "type": "parse_exception",
    "reason": "Failed to parse content to map",
    "caused_by": {
      "type": "json_parse_exception",
      "reason": "Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@a36d6b2; line: 4, column: 295]"
    }
  },
  "status": 400
}
2

The pipeline ingest pattern looks as follows:

"lasernet" : {
    "description" : "Convert Lasernet logfile",
    "processors" : [
      {
        "grok" : {
          "field" : "message",
          "patterns" : [
            "%{GREEDYDATA:Server};%{TIMESTAMP_ISO8601:Time};%{GREEDYDATA:Service};%{INT:value1};%{INT:value2};%{GREEDYDATA:value3};%{GREEDYDATA:Hash};%{GREEDYDATA:Status}"
          ]
        }
      }
    ]
  }
3

This is not a pipeline issue. The JSON you supplied is not valid, because it is not a proper JSON object. The _source field resembles what you sent to Elasticsearch, which usually is an object. So your _source needs to be something like

"_source" :  { "field" : "SAG-DFO-015;2019-04-1 ..." }

In your fixed example the _source is just a string field, which is something you would never sent to ES as it cannot be properly indexed.

--Alex

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.