Kibana runtime fields in elastic 8.4.0

[Ajmal_Khalil]

Hi,

I'm trying to create a runtime field but getting errors with my syntax also after creating the field not getting that under discover in fields.
I hope you can help me in this.
Actuall i have logs like this

"No transactions left to log execution times" and "Connection for Client Code [100] borrowed in 1 milliseconds" etc in the message field but i want to create separate multiple runtime fields for them and want to create a dashboard based on these values. These logs are in message form like "Connection for Client Code [100] borrowed in 1 milliseconds" .
My runtime field script is like this and i copied this from one of your previous post and edit my values. please see below

String Connection for Client Code [101] borrowed in 1 milliseconds=grok('%{GREEDYDATA:leading_data}/CN=%{DATA:Connection for Client Code [101] borrowed in 1 milliseconds}'').extract(params._source.message)?.Connection for Client Code [101] borrowed in 1 milliseconds;
if (Connection for Client Code [101] borrowed in 1 milliseconds != null) emit(Connection for Client Code [101] borrowed in 1 milliseconds);

Can you please help me in writing a script so that i can get custom values from the logs and create fields for each different logs value. thanks

[stephenb (Stephen Brown)]

Hi @Ajmal_Khalil I tried to format your post above but it is still very hard to read

To Format Code / Docs please select and use this button

Can you Provide a sample JSON document source.

Can you Format the Code into Multi-line

Can you provide what the result is that you want

This will help us help you

Example

Source doc.. this is not really valid see all the quotes, please copy actual sample doc.

{
   "_source": {
        "message": ""No transactions left to log execution times" and "Connection for Client Code [100] borrowed in 1 milliseconds" etc in the message field but i want to create separate multiple runtime fields for them and want to create a dashboard based on these values. These logs are in message form like "Connection for Client Code [100] borrowed in 1 milliseconds""
        }
    }
}

Example Code:... Can you format this into something Readable?

String Connection for Client Code [101] borrowed in 1 milliseconds=grok('%{GREEDYDATA:leading_data}/CN=%{DATA:Connection for Client Code [101] borrowed in 1 milliseconds}'').extract(params._source.message)?.Connection for Client Code [101] borrowed in 1 milliseconds;
if (Connection for Client Code [101] borrowed in 1 milliseconds != null) emit(Connection for Client Code [101] borrowed in 1 milliseconds);

What is the expected result you want? You never actual tell us..

I would love to help but apologies I do not have enough understanding / details to help

Here is a nice long post on runtime fields ... notice the level of detail we get into plus the details / formats etc... This is how you get a good answer!

Extract value from message field ( runtime fields)

You are mixing / not quite getting the concepts. You are adding a runtime field to an empty index, then you are adding that empty index to an index pattern, and expecting the runtime field that is ONLY on the empty index to work on all the other indices you did not add the runtime field too ... it will not work. So now I will be prescriptive... as you are much closer to option b so lets do that to start. First delete the empty index you created... DELETE openvpn-dashboard Then delete your…

Also @Ajmal_Khalil have you already done the initial parsing of your logs?

How are you ingesting the Data with Filebeat?
What do the logs now look like in Elastic as a Document?
Did you solve the Multi-Line Issues?
Do you have an understanding the difference between and Ingest Pipeline (parsing before the document is written)
vs
Runtime Field : Parsing at runtime / after the document is written.
Typically you want to do as much parsing on Ingest you can then do runtime for specific use cases...
When I look at your case .. my initial feeling is that you may want to use an ingest pipeline because they are more flexible but if that is not the case we cant try runtime.

Perhaps we should back up... tell use these details and show us sample docs and then show us what you want as a result.