Elasticsearch Ingest node gsub processor replace character

Elastic Stack Elasticsearch
1

I am trying to use the gsub processor to replace characters such as " to \"

curl -XPOST 'localhost:9200/_ingest/pipeline/_simulate?pretty' -H 'Content-Type: application/json' -d'
{
  "pipeline": {
  "description" : "parse multiple patterns",
  "processors": [
    {
	  "gsub": {
        "field": "message",
        "pattern": """,
        "replacement": "\""
      }
    }
  ]
},
"docs":[
  {
    "_source": {
      "message": "I have a "pen", you know!"
    }
  }
  ]
}
'

but has error

 "error" : {
        "reason" : "Failed to parse content to map"
      }
    ],
    "type" : "parse_exception",
    "reason" : "Failed to parse content to map",
    "caused_by" : {
      "type" : "json_parse_exception",
      "reason" : "Unexpected character ('\"' (code 34)): was expecting comma to separate Object entries\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@7b179814; line: 9, column: 23]"
    }
  },
  "status" : 400
}
2

The problem is that what you give in the source above is not valid JSON. You're supposed to encode the double quotes before it reaches the pipeline.

3

my log is shipped by filebeat, the original log is just like this192.168.1.2 - - [10/Jul/2015:15:51:09 +0800] "GET /ubuntu.iso HTTP/1.0", i try to parse it in someways with grok processor, but cannot process double quotes,

curl -XPOST 'localhost:9200/_ingest/pipeline/_simulate?pretty' -H 'Content-Type: application/json' -d'
{
  "pipeline": {
  "description" : "parse multiple patterns",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IP:ip}\\s-\\s-\\s\\[%{TIME:time}\\s\\S+\\]\\s\"%{REQUEST:request}\"\\s\\d+"],
		"pattern_definitions" : {
          "IP" : "(\\S+)",
		  "TIME": "(\\S+)",
		  "REQUEST": "\\w+"
        }
      }
    }
  ]
},
"docs":[
  {
    "_source": {
      "message": "192.168.1.2 - - [10/Jul/2015:15:51:09 +0800] "GET" 168"
    }
  }
  ]
}
'

do i need to encode the message in json format with filbeat first,and then send to es?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.