How to convert string to JSON?


(Stepan) #1

My logstash configuration is:
input {
tcp{
codec => json_lines {charset => "CP1251"}
...
}
}
output {
elasticsearch{...}}

But there is a problem that I can recieve string to already mapped as object field.
In that case in elasticsearch occures an error and I lose this line in the results:

org.elasticsearch.index.mapper.MapperParsingException: object mapping for [detail.data.response.body] tried to parse field [body] as object, but found a concrete value

I tried mutate/replace or
filter
{
if [detail][data][response][body]
{
if [detail][data][response][body] == "Unauthorized"
{
json_encode
{
source => "[detail][data][response][body]"
}
}
}
}

but had no success. Can you help me, please?


(Magnus Bäck) #2

The mapping of detail.data.response.body is inconsistent with the data you're attempting to index. What kind of value(s) should detail.data.response.body contain? An object or a value? What are you trying to index? Comment out the elasticsearch output and add a stdout { codec => rubydebug } output.


(Stepan) #3

As usual it comes as json:
"body":{"values":[], etc}
But sometimes I recieve
"body":"Unauthorized"
or
"body":"Not Found"

I would like to replace this kinds of strings with
"body":{"thesebeautifulpeoplegavemeastring":"Unauthorized"}
or
"body":{"thesebeautifulpeoplegavemeastring":"Not Found"}

[2017-10-05T17:16:21,702][DEBUG][o.e.a.b.TransportShardBulkAction] [test_node_name] [logstash-2017.10.05][1] failed to execute bulk item (index) BulkShardRequest [[logstash-2017.10.05][1]] containing [10] requests
org.elasticsearch.index.mapper.MapperParsingException: object mapping for [detail.data.response.body] tried to parse field [body] as object, but found a concrete value


(Magnus Bäck) #4

I think you'll have to use a ruby filter to check whether the value is a string and, if so, convert it to a hash. Something like

if event.get('body').is_a? String
  event.set('body', {"thesebeautifulpeoplegavemeastring" => "Unauthorized"})
end

might work.


(Stepan) #5

looks like what I wanted, but when i added filter

filter 
{
	if [detail][data][response][body]
	{
		ruby
		{
			code => "
			if event.get('detail.data.response.body').is_a? String
				event.set('detail.data.response.body', {'givenstring' => event.get('detail.data.response.body')})
			end"
		}
	}
}

the logstash began to restart all the time and became unresponsive


(Guy Boertje) #6

try event.get('[detail][data][response][body]')


(Stepan) #7

made

filter 
{
	if [detail][data][response][body]
	{
		ruby
		{
			code => "
			if event.get('[detail][data][response][body]').is_a? String
				event.set('[detail][data][response][body]', {'[detail][data][response][body][givenstring]' => event.get('[detail][data][response][body]')})
			end"
		}
	}
}

didnt work, again infinite restart


(Magnus Bäck) #8

What's the error message?

Also,

{'[detail][data][response][body][givenstring]' => event.get('[detail][data][response][body]')}

should be:

{'givenstring' => event.get('[detail][data][response][body]')}

(Guy Boertje) #9

A) Post some lines from the console output to show the infinite restart - usually, there is some details that we can use to troubleshoot.

B) Which version of Logstash are you using?

C) try:

filter 
{
  if [detail][data][response][body]
  {
    ruby
    { init => '
        BODY_PATH = "[detail][data][response][body]".freeze
        BODY_STRING = "[givenstring]".freeze
      '
      code => '
        body_val = event.get(BODY_PATH)
        if body_val.is_a?(String)
          event.set(BODY_PATH, {BODY_STRING => body_val})
        end
      '
    }
  }
}

About the Ruby code.

  1. Create strings once as Ruby Constants in the plugin initialization. This prevents unnecessary string creation for every filter invocation - the event.set will deconstruct the path reference into new string parts internally anyway so a new string each time is a waste.
  2. Freeze the Constants, this will prevent other code from changing them.
  3. Extract the value to a local variable once and reuse.

(Stepan) #10

That works perfectly! Thank you very much! Especially for explanation!


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.