Convert Packetbeat Field Contents to Lowercase

Elastic Stack Logstash
1

I am sending DNS queries from a windows DNS server to Logstash via Packetbeat.
When I view the contents of the field dns.question.name in Kibana, some entries contain capitalizations.

I have tried the following in a Logstash conf file but it does not work.

filter {
mutate {
lowercase => [ "dns.question.name" ]
}
}
I'm pretty sure that the error has to do with the periods in the field name, but I am not sure how to handle this.
Thank you.

2

In the Discover pane of kibana, if you expand an event and go to the JSON tab, what does the dns.question.name field look like? It may be an array, or it may be a nested field which would be called [dns][question][name] in logstash.

3

if it is in the format of a nested field, [dns][question][name], then would my filter need to be formatted like this?

filter {
mutate {
lowercase => [ [dns][question][name] ]
}
}
I am also not familiar with how to expand an event. I will google that and reply.
The expanded event shows
"dns.question.name": [

  "skypedataprdcolneu01.cloudapp.net"

]
4

You can expand the event just by clicking on that row in the Discover pane.

    mutate { lowercase => [ "[dns][question][name]" ] }

Is what I meant. If it is an array then it could be

    mutate { lowercase => [ "[dns][question][name][0]" ] }

or

    mutate { lowercase => [ "[dns.question.name][0]" ] }
5

When I use the format of :

mutate { lowercase => [ "[dns][question][name]" ] }

The Logstash service restarts after being up for about 15 seconds or so.
root@s-v-elk01:/etc/logstash/conf.d# systemctl status logstash
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-04-16 17:21:46 UTC; 11s ago
Main PID: 420542 (java)
Tasks: 19 (limit: 4618)
Memory: 635.8M
CGroup: /system.slice/logstash.service
└─420542 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=>

Apr 16 17:21:46 s-v-elk01 systemd[1]: Started logstash.
Apr 16 17:21:46 s-v-elk01 logstash[420542]: Using bundled JDK: /usr/share/logstash/jdk
Apr 16 17:21:46 s-v-elk01 logstash[420542]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Apr 16 17:21:57 s-v-elk01 logstash[420542]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Apr 16 17:21:57 s-v-elk01 logstash[420542]: [2021-04-16T17:21:57,990][INFO ][logstash.runner ] Log4j configuration path used is: /etc/logstash/log4j2.properties
Apr 16 17:21:57 s-v-elk01 logstash[420542]: [2021-04-16T17:21:57,997][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.12.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a>
root@s-v-elk01:/etc/logstash/conf.d# systemctl status logstash
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-04-16 17:21:59 UTC; 1s ago
Main PID: 420647 (java)
Tasks: 15 (limit: 4618)
Memory: 197.1M
CGroup: /system.slice/logstash.service
└─420647 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=>

Apr 16 17:21:59 s-v-elk01 systemd[1]: Started logstash.
Apr 16 17:21:59 s-v-elk01 logstash[420647]: Using bundled JDK: /usr/share/logstash/jdk
Apr 16 17:21:59 s-v-elk01 logstash[420647]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

6

You will need to see what is in the logstash log, which is probably under /var/log/logstash

7

This is what i am seeing there
[2021-04-16T19:44:20,058][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "if", [A-Za-z0-9_-], '"', "'", "}" at line 5, column 1 (byte 69) after filter {\n mutate {\n lowercase => [ "[dns][question][name]" ]\n}\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in block in converge_state'"]}
[2021-04-16T19:44:20,147][INFO ][logstash.runner ] Logstash shut down.

8

OK so it is objecting to whatever comes after

 filter {
   mutate {
   lowercase => [ "[dns][question][name]" ]
   }

I would expect there to be a } next to close the filter section. What does your configuration have there?

9

There was a curly brace "}" but I just deleted everything there and recreated the filter statement.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.