Extracting the JSON fields from the message

[Raed (Raed Shari)]

I'm getting the below JSON as a message:

    {
		"requestUrl": "http://localhost:8080/frameworks/298",
		"requestUri": "/frameworks/298",
		"requestMethod": "PUT",
		"requestHeaders": {
			"authorization": "***",
			"accept-language": "en",
			"content-type": "application/json"
		},
		"requestBody": {
			"name": "NodeJs"
		}
	}

Using the below filter my whole JSON gets printed in a field called message in Kibana which is fine. but when I try to extract a specific attribute in the JSON It didn't work.

    filter {
	    json { 
	        source => "message"
	    }
	    mutate {
		    add_field => { 
			    "requestUrl" => "%{[message][requestUrl]}"
			    "requestUri" => "%{[message][requestUri]}"
			    "requestMethod" => "%{[message][requestMethod]}"
			    "requestHeaders" => "%{[message][requestHeaders]}"
			    "requestBody" => "%{[message][requestBody]}"
		    }
	    }
    }

All I want is to add a field in Kibana called "requestUrl" and it should contain the requestUrl value in the JSON.

[Raed (Raed Shari)]

Also tried without using mutate and it didn't work.

    filter {
	    json { 
	        source => "message"
		    add_field => { 
			    "requestUrl" => "%{[message][requestUrl]}"
			    "requestUri" => "%{[message][requestUri]}"
			    "requestMethod" => "%{[message][requestMethod]}"
			    "requestHeaders" => "%{[message][requestHeaders]}"
			    "requestBody" => "%{[message][requestBody]}"
		    }
	    }
    }

[Badger]

@Raed

"requestUrl" => "%{[message][requestUrl]}"

If the json filter works you will have a field called [requestUrl], not [message][requestUrl]. Likewise for the others.

[Raed (Raed Shari)]

Thanks @Badger

I have tried that and many more ways but without any luck.
Only the full JSON I can print as a new field, but I can't access any attribute inside it. You can see that only the last field is working.

"requestUrl" => '%{[requestUrl]}'
"requestUri" => '%{requestUri}'
"requestMethod" => "%{[message][requestMethod]}"
"requestHeaders" => "%{[message][requestHeaders]}"
"full-json-message" => "%{message}"

Also, the Logstash console doesn't show me any parsing error. Actually, it doesn't show any error at all.

[Luca_Belluccini]

Hello @Raed

The json filter should be enough and you should end up with all the fields already.
There is no need for the mutate filter:

filter {
  json {
    source => "message"
  }
}

In addition, if you want, you might discard the original message:

filter {
  json {
    source => "message"
    remove_field => "message"
  }
}

For more information about the JSON filter, check the documentation.

[Raed (Raed Shari)]

Hi @Luca_Belluccini

That was my initial filter and what I'm seeing everywhere, but it is not working with me.
I have updated my config again, but the fields still not showing up.

input {
	tcp {
		port => 5000
	}
}

filter {
  json {
    source => "message"
  }
}

output {
	elasticsearch {
		hosts => "elasticsearch:9200"
		index => "logstash-local"
	}
}

And here is the full log that Logstash receives maybe someone can catch my mistake.

[Luca_Belluccini]

Oh I get it!

You have a JSON coming in the TCP input, but once decoded, you have another JSON inside the message field.
E.g.

{ "abc": 1, "message": "{\"zzz\": { \"www\": 312 } }" }

So you have a JSON message, and then a JSON message within the field message.

To parse it, just use the json filter twice:

json {
  source => "message"
}
json {
  source => "message"
  remove_field => [ "message" ]
}

I've tested it with the following pipeline:

input {
    java_generator {
        lines => [ '{ "abc": 1, "message": "{\"zzz\": { \"www\": 312 } }" }' ]
        eps => 0.1
      }
}

filter { 
    json {
		source => "message"
	}
	json {
		source => "message"
		remove_field => ["message"]
	}
}
output {
	 stdout { codec => rubydebug }
}

Result:

{
       "@timestamp" => 2020-04-05T23:45:54.819Z,
              "zzz" => {
        "www" => 312
    },
              "abc" => 1
}

[Raed (Raed Shari)]

WOW. Thanks @Luca_Belluccini for that finding. It is working now.
I don't want to tell how much time I spent to solve this.

Anyway I'm getting parsing error now in Logstash Console:

"reason"=>"failed to parse field [requestHeaders]

Full Error:

[2020-04-05T23:52:30,087][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-local", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x63c8f9c1>], :response=>{"index"=>{"_index"=>"logstash-local", "_type"=>"_doc", "_id"=>"zpbCTHEBMaOFCSGqikp9", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [requestHeaders] of type [text] in document with id 'zpbCTHEBMaOFCSGqikp9'. Preview of field's value: '{authorization=Basic cmFlZDpwYXNz, content-length=262, accept-language=ar, cookie=JSESSIONID=3E1879B8967DA08AEA7F45B218D88444, postman-token=27f891dd-a2de-42bf-a7d5-3f1bd8e0b959, host=localhost:8080, content-type=application/json, connection=keep-alive, cache-control=no-cache, accept-encoding=gzip, deflate, br, accept=*/*, user-agent=PostmanRuntime/7.24.0}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:125"}}}}}

But it is working when I remove the nested JSONs inside the message JSON.

filter {
	json {
		source => "message"
	}
	json {
		source => "message"
		remove_field => ["requestHeaders","requestBody"]
	}
}

Any Idea how to parse the nested JSONs also? or don't worry, I will try to find the solution somewhere.

Thanks again @Luca_Belluccini you don't know how much this helped me.

[Luca_Belluccini]

Hello @Raed

You're getting a mapping conflict:

failed to parse field [requestHeaders] of type [text] in document with id

This happens because requestHeaders is usually a Map, but due to the initial attempts you've made, requestHeaders has been detected by Elasticsearch as a text field.

Mappings (which tell Elasticsearch the type of the fields) cannot be changed once the index has been created.
During your first attempts, requestHeaders contained %{[message][requestHeaders]} and Elasticsearch detected a text field.
Now, requestHeaders contains a map such as {"authorization": "Basic....", "content-length": "262" ... }.

I suggest to follow those steps:

1. Stop Logstash

2. Delete the destination index (logstash-local)

3. [OPTIONAL] The best approach would be to set up an Index Template.

   For example, if you're using Elasticsearch 7.3 or more recent, use the flattened data type for requestHeaders:

   PUT _template/logstash_template
   {
     "index_patterns": ["logstash*"],
     "settings": {
     },
     "mappings": {
       "properties": {
         "requestHeaders": {
           "type": "flattened"
         }
       }
     }
   }
   

4. Start Logstash

[Raed (Raed Shari)]

Thanks again @Luca_Belluccini for such a great support.
I've deleted the index as you suggested, and it is WORKING now.