How to be able to pull data from a field and then create a new field for the extracted data?

#1

Hello,

I am working on a way to be able to pull an area of data which is shown in a field. I would like to be able to run a scripted field to be able to pull out the intended data and to create a new field with it. But still keeping the original field as is.

for example with the data that I have been using to create this template is below (e.g. GET request):

GET      /example/of/request/to/get/XXXXXXX-xxxx-XXXXX/to/create/new/field

So, from this, the data that I am trying to retrieve and create a new field is from the "Request" - the original field. To pull out the ID - "XXXXXXX-xxxx-XXXXX" and to great a new field called "ID" - new field.

Would this be better using scripted fields or would this be better with a grok or sort of filter in a Logstash conf?

Thank you!

#2

In general, working with data is a little bit simpler if its directly in the original document. I'd start with a logstash grok filter.

You could also use a kibana scripted field - its possible that it would work better for you.
Scripted fields are documented here - https://www.elastic.co/guide/en/kibana/current/scripted-fields.html

#3

Hey @mattkime,
Thanks for your help and for the link!

So far in my .conf file for Logstash, it takes the file that is in question with the data in it (the GET request) and uses filtering to breakdown the structure of the file. This is already in place to be able to put the data into different fields.

The data from the file example -

2019-10-14 23:59:35 192.168.1.1 GET /example/of/request/to/get/XXXXXXX-xxxx-XXXXX/to/create/new/field - 443 - 192.168.1.23 - - 200 0 0 312

The filter grok pattern -

filter {
  grok {
match => { "message" => "%{DATE:Date} %{TIME:Time} %{IPV4:IP1} %{WORD:Type} %{PATH:Request} %{NOTSPACE:Ignore} %{INT:Port} %{NOTSPACE:IGnore} %{IPV4:IP2} %{NOTSPACE:IGNore} %{NOTSPACE:IGNOre} %{INT:Code} %{INT:Value1} %{INT:Value2} %{INT:TimeTaken}"}
  }
}

Following this and with the data being in Kibana, would there be a way, with using scripted fields, to be able to select the "Request" field and to great another field with the data acquired? So to pull the "XXXXXXX-xxxx-XXXXX" data (From the "Request" field) and put it into a new field.

Thank you