Extract information from raw messages in KIBANA

[tofmonaute (Christophe Larcher)]

  Hi,

I've been struggling for weeks to create a vizualization with dynamic labels : extracted from a a search request.
( we dont want to create a new field in logstash for some errors that occurs with a very low frequency ...we want to extract information at Vizualization creation )
I tried to use Timelion but unsuccessfully....
Is there any way to fullfill this ? It seems impossible.

Kibana currently does not provide a way to extract values from raw messages ?

The feature exists in splunk ... and is very easy to use.

Some one can help ?

[jbudz (Jon Budzenski)]

Elasticsearch supports runtime scripts. In Kibana if you go to the index patterns page -> scripted fields -> add you can extract text into a new field. These will behave as if the field was indexed in Kibana.

There's a runtime performance hit for every script, it has to be calculated for every document. Performance hit may be negligible if the search size is small.

[kristoflarcher]

Thanks for response Jon

I 've already tried to test scripted fields as follow with a simple regex working in any regex tester
to extract information from string field msg :

test field:
def m = /Detected server\s+(\w+)started)/.matcher(doc["msg"].value);
if ( m.matches() ) {
return m.group(1)
} else {
return "no match"
}

and When i did a search in Discover, i had no mach at all :
Time test msg
Dec 23, 2019 @ 14:28:53.437 no match Detected server BL2_500c_MBR2 started on node RefNode_REF_500B_EB
Dec 23, 2019 @ 14:28:53.436 no match Detected server BL2_500c_MBR2 started on node RefNode_REF_500B_EB
Dec 23, 2019 @ 14:28:53.431 no match Detected server BL2_500c_MBR2 started on node RefNode_REF_500B_EB
Dec 23, 2019 @ 14:28:53.429 no match Detected server BL2_500c_MBR2 started on node RefNode_REF_500B_EB

So I enlarged the regex to include the whole log line as follow :

test field:
def m = /^(.*)$/.matcher(doc["msg"].value);
if ( m.matches() ) {
   return m.group(1)
} else {
   return "no match"
} 


and  When i did a search in Discover, i am matching just a word...; i really dont understand....
   Time 	                                                   test   	         msg   
	Dec 23, 2019 @ 14:28:53.437	BL2_500c_MBR2 	Detected server BL2_500c_MBR2 started on node RefNode_REF_500B_EB
	Dec 23, 2019 @ 14:28:53.436	BL2_500c_MBR2 	Detected server BL2_500c_MBR2 started on node RefNode_REF_500B_EB
	Dec 23, 2019 @ 14:28:53.431	BL2_500c_MBR2 	Detected server BL2_500c_MBR2 started on node RefNode_REF_500B_EB
	Dec 23, 2019 @ 14:28:53.429	BL2_500c_MBR2 	Detected server BL2_500c_MBR2 started on node RefNode_REF_500B_EB

I just remind that we just wanted to use a dynamic label in a 1 vizualization : a script field is a new field....that we dont want...