Extract field from message field with in logs

Elastic Stack Logstash
1

im working with an elk stack that I did not deploy. I have an "index" for dns logs. it gives me a few default fields. The message field is given, but I noticed that the "client" and "query" with in it are things I would like to get a separate field for because they contain pertinent info. how would I add this ?
what file would I edit to not he Logstash server to add those two "fields" /etc/logstash/conf.d/* ?

the client field has the # sign added to it can I separate that ?
Im an elk noob sorry

message:<30>Feb 11 09:48:11 x.x.x.x named[22083]: client x.x.x.x#56007 (z03resources.renlearnrp.com): query: z03resources.renlearnrp.com IN A +ED (x.x.x.x)

2

You'll need to inform whatever tool you're using to ingest that data how to break out those values. You mentioned Logstash, which is perfectly capable of doing this... but I don't know that tool so I can't really give you any assistance there. The best I can do is point you at the log parsing docs. You probably just need to update the grok pattern you're using.

3

sorry moved it to Logstash forum instead of Kibana. but that thanks for the input

4

I would start by picking apart the syslog header.

    grok { match => [ "message", "^<%{NUMBER:level}>%{SYSLOGTIMESTAMP:ts} %{IPV4:ip1} %{WORD:program}\[%{NUMBER:pid}\]: %{GREEDYDATA:restOfLine}" ] }

I'm not sure what fields you want from that message, but this should get you started.

    grok { match => [ "restOfLine", "^%{WORD:something1} %{IPV4:ip2}#%{NUMBER:port} \(%{HOSTNAME:host1}\): (?<query>[^(]+)\(%{IPV4:ip3}\)" ] }

That gets me

   "program" => "named",
     "query" => "query: z03resources.renlearnrp.com IN A +ED ",
       "ip1" => "1.2.3.4",
        "ts" => "Feb 11 09:48:11",
       "ip2" => "5.6.7.8",
       "pid" => "22083",
     "host1" => "z03resources.renlearnrp.com",
     "level" => "30",
      "port" => "56007",
"something1" => "client"
       "ip3" => "8.1.2.4"
5

Thank you for the info much appreciated. This is deff something for me to go through . This would go in the /etc/logstash/conf.f/*.filter.conf file in logstash correct ?

6

Typically you would create a configuration file in /etc/logstash/conf.d and point logstash to the file using -f

7

I normally just edit the files in there and sysctl restart logstash. is that not normal ?
do I have to start logstash with a specific conf file every time ?

8

You can point -f at a specific file, or at a directory, in which case it will concatenate all the files in the directory to create a configuration. Whatever works for you.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.