How can I filter certain information from the logs?

[hta]

We work with ELK Stack and I have the task of creating meaningful visualizations from the log entries. I have logs in the following format:

{
  "@timestamp": [
    "2023-08-08T00:00:11.2123"
  ],
  "xxxxx": [
    "yyyyy"
  ],
  "message": [
    "some text some informations; name{name='marc', school='dsds', moreinformation='more'} more information"
  ],
  "abcdfg": [
    "some text some informations"
  ]
}

How could I, for example, filter out and count certain information from the message field?

I need the field for the school and it would be good if I count how many students a school has (based on the students' names).

I have it with the grok processors at the pipeline. Unfortunately I didn't come to a solution. Is there a better way? what else can I do?

[cperzrt10 (Cristian)]

Hi @hta , try the following grok pattern

filter {
  grok {
    id => "name school grok filter"
    match => { 'message' => '^.*name=\'%{WORD:name}\'.*school=\'%{WORD:school}\''}
  }
}

it gets the name and the school from the message.

[hta]

thank you very much for your help. That worked. I'm hoping the newly extracted fields will show up tomorrow when new logs come in.
Where is the best place to enter the pattern? I added this under Kibana in the pipeline as a processor without "filter" and "grok" only '^.*name=\'%{WORD:name}\'.*school=\'%{WORD:school}\'
It worked in the debugger. Is that ok?

[cperzrt10 (Cristian)]

Finally it worked?

[hta]

I can not say it. we are having technical difficulties. There are currently no new logs

[hta]

yes, it worked. Thx!!!

[hta]

Can you maybe tell me what I can do? I changed the above expression slightly. For example, with WORD:student.name I would like to create a field "student" and this field contains the information such as name. In Kibana "discover" I can see the logs, but it shows me that "student.name" is not mapped (Unmapped fields). How do I get that? I described the field under Index Management -> Index Templates -> Settings (from template). Still it doesn't work...

[cperzrt10 (Cristian)]

Hi. maibe somethin like this

filter {
  grok {
    id => "name school grok filter"
    match => { 'message' => '^.*name=\'%{WORD:name}\'.*school=\'%{WORD:school}\''}
  }
 mutate {
   # Renames the 'HOSTORIP' field to 'client_ip'
   rename => { "student" => "student.name" }
  }
}

https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-rename