UserAgent Transformation in Logstash

Elastic Stack Logstash
1

I have the below useragent captured as a string in our application logs, we are publishing this logs to elastic search from logstash and currently storing it as string, We would like to convert this string to json object and store it as json objects in elastic search index, so that it will be easy for us to query it in kibana.

input :

"clientUserAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.67 Safari/537.36"

Expected output in ES:

"clientUserAgent": {
"os": {
"major": null,
"minor": null,
"patch": null,
"family": "Linux",
"patch_minor": null
},
"device": {
"brand": null,
"model": null,
"family": "Other"
},
"user_agent": {
"major": "75",
"minor": "0",
"patch": "3770",
"family": "Chrome"
},
"string": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.67 Safari/537.36"
}

can I able to achieve this using logstash filters and is there any working example ?.

2

There is a useragent filter that parses these strings.

3

I tried the useragent filter and can see its filtered all the output into separate objects ,Ideally i want to group everything in clientuseragent json object as in the expected output

{
"minor" => "0",
"os_minor" => "13",
"clientCorrelationId" => "",
"os_major" => "10",
"principalId" => "",
"patch" => "3538",
"file" => "ServiceEntryInterceptor.java:31",
"major" => "70",
"payload" => {
"duration" => nil,
"exception" => nil,
"queryParameters" => "",
"httpStatus" => nil,
"message" => "Starting request",
"httpMethod" => "GET",
"httpStatusCode" => nil
},
"requestId" => "d1fb11ef-b51a-11e9-ac5c-c9c6a729005b",
"@version" => "1",
"host" => "a54d67dd9c10",
"context" => "default",
"customerId" => "",
"correlationId" => "d1fb11ef-b51a-11e9-ac5c-c9c6a729005b",
"timestamp" => "2019-08-02T12:44:00.838+01",
"clientUserAgent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36",
"os" => "Mac OS X",
"level" => "INFO",
"thread" => "http-nio-8080-exec-10",
"message" => "Starting request",
"tags" => [
[0] "_grokparsefailure"
],
"@timestamp" => 2019-08-02T14:32:03.135Z,
"breadcrumb" => "reference:1",
"application" => "reference",
"principalPermissions" => "",
"build" => "",
"name" => "Chrome",
"os_name" => "Mac OS X",

4

So specify the target option on the filter. If you don't like the resulting arrangement of fields then use mutate to move them around.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.