How to convert string to JSON?

Elastic Stack Logstash
1

My logstash configuration is:
input {
tcp{
codec => json_lines {charset => "CP1251"}
...
}
}
output {
elasticsearch{...}}

But there is a problem that I can recieve string to already mapped as object field.
In that case in elasticsearch occures an error and I lose this line in the results:

org.elasticsearch.index.mapper.MapperParsingException: object mapping for [detail.data.response.body] tried to parse field [body] as object, but found a concrete value

I tried mutate/replace or
filter
{
if [detail][data][response][body]
{
if [detail][data][response][body] == "Unauthorized"
{
json_encode
{
source => "[detail][data][response][body]"
}
}
}
}

but had no success. Can you help me, please?

2

The mapping of detail.data.response.body is inconsistent with the data you're attempting to index. What kind of value(s) should detail.data.response.body contain? An object or a value? What are you trying to index? Comment out the elasticsearch output and add a stdout { codec => rubydebug } output.

3

As usual it comes as json:
"body":{"values":[], etc}
But sometimes I recieve
"body":"Unauthorized"
or
"body":"Not Found"

I would like to replace this kinds of strings with
"body":{"thesebeautifulpeoplegavemeastring":"Unauthorized"}
or
"body":{"thesebeautifulpeoplegavemeastring":"Not Found"}

[2017-10-05T17:16:21,702][DEBUG][o.e.a.b.TransportShardBulkAction] [test_node_name] [logstash-2017.10.05][1] failed to execute bulk item (index) BulkShardRequest [[logstash-2017.10.05][1]] containing [10] requests
org.elasticsearch.index.mapper.MapperParsingException: object mapping for [detail.data.response.body] tried to parse field [body] as object, but found a concrete value

4

I think you'll have to use a ruby filter to check whether the value is a string and, if so, convert it to a hash. Something like

if event.get('body').is_a? String
  event.set('body', {"thesebeautifulpeoplegavemeastring" => "Unauthorized"})
end

might work.

5

looks like what I wanted, but when i added filter

filter 
{
	if [detail][data][response][body]
	{
		ruby
		{
			code => "
			if event.get('detail.data.response.body').is_a? String
				event.set('detail.data.response.body', {'givenstring' => event.get('detail.data.response.body')})
			end"
		}
	}
}

the logstash began to restart all the time and became unresponsive

6

try event.get('[detail][data][response][body]')

7

made

filter 
{
	if [detail][data][response][body]
	{
		ruby
		{
			code => "
			if event.get('[detail][data][response][body]').is_a? String
				event.set('[detail][data][response][body]', {'[detail][data][response][body][givenstring]' => event.get('[detail][data][response][body]')})
			end"
		}
	}
}

didnt work, again infinite restart

8

What's the error message?

Also,

{'[detail][data][response][body][givenstring]' => event.get('[detail][data][response][body]')}

should be:

{'givenstring' => event.get('[detail][data][response][body]')}
1 Like
9

A) Post some lines from the console output to show the infinite restart - usually, there is some details that we can use to troubleshoot.

B) Which version of Logstash are you using?

C) try:

filter 
{
  if [detail][data][response][body]
  {
    ruby
    { init => '
        BODY_PATH = "[detail][data][response][body]".freeze
        BODY_STRING = "[givenstring]".freeze
      '
      code => '
        body_val = event.get(BODY_PATH)
        if body_val.is_a?(String)
          event.set(BODY_PATH, {BODY_STRING => body_val})
        end
      '
    }
  }
}

About the Ruby code.

  1. Create strings once as Ruby Constants in the plugin initialization. This prevents unnecessary string creation for every filter invocation - the event.set will deconstruct the path reference into new string parts internally anyway so a new string each time is a waste.
  2. Freeze the Constants, this will prevent other code from changing them.
  3. Extract the value to a local variable once and reuse.
10

That works perfectly! Thank you very much! Especially for explanation!

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.