Remove Parent fields in logstash filter

Elastic Stack Logstash
1

I have large log json message which I have to parse to visualize at kibana.
I have used json filter first to parse message but there are generated lots of parent and dynamic fields. Due to dynamic fields in each log message number of fields are increasing and kibana is slow. I want to remove to get unique fields for each log. how to resolve this issue. For the below example like I want to remove parent fields like nodes.processes.dynamicvalue
some part of message after applying json filter
example:
nodes.processes.2A2807E813FE36C5.fullPid.startTime.millisecondsSinceEpoch
1704797811445
nodes.processes.2A2807E813FE36C5.group.key.value
C55706E813FE36C5
nodes.processes.2A2807E813FE36C5.integrityLevel
LOW
nodes.processes.2A2807E813FE36C5.name
Microsoft Edge
nodes.processes.2A2807E813FE36C5.node.key.value
2A2807E813FE36C5
nodes.processes.2A2807E813FE36C5.parent.key.value
5A2607E813FE36C5
nodes.processes.2A2807E813FE36C5.sessionId
1
nodes.processes.2A2807E813FE36C5.subsystem
SYS_WIN32
nodes.processes.2A2807E813FE36C5.user.name
SANJAY-AIO\sgupta
nodes.processes.2A2807E813FE36C5.user.sid
S-1-5-21-3758888377-4075476628-3217939268-1002
nodes.processes.3D0B07E813FE36C5.activeContent.signed
E_FALSE
nodes.processes.3D0B07E813FE36C5.activeContent.type
AC_FILE
nodes.processes.3D0B07E813FE36C5.commandLine
"C:\Program Files\Mozilla Firefox\firefox.exe"
nodes.processes.3D0B07E813FE36C5.completenessHintsBitmask
263168

2

You want to remove the node.processes top level field?

If so, the following filter may work.

filter {
    mutate {
        remove_field => ["[nodes][processes]"]
    }
}
3

I also want to remove parent3 fileld which is dynamic

4

Then you will need to use a ruby filter and write some code to do that, I do not have much experience in ruby but there are a couple of questions already answered in the forum about this.

Also, after you remove all the dynamic fields after nodes.processes, will it have anything left? If not then it is easier to just remove nodes.processes.

5

Yes there are more fields which left after deleting nodes.processes.dynamicvalue.
I have used the code following for the above : this is my filter file. here json filter is working but parent fields are not deleting. I have already tested online regex pattern tester tool for check regex pattern, it is correct. but in logstash this code is not working
filter {
json {
source => "message"
}

ruby {
code => "
event.to_hash.keys.each do |key|
if key.start_with?('nodes.files.')
new_key = key.gsub(/^nodes.files.([^.]+)./, '')
event.set(new_key, event.get(key))
event.remove(key)
end
end
"
}
}

6

How to fix it

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.