Creating a Logstash filter help

Elastic Stack Logstash
1

Hi,

I am in a process of checking if kibana can be used to help me with my log analysis. I am new to the logstash and elasticsearch technology and will be happy to get help to jump start my tests.

I am using syslog to forward my events to the Kibana server.

for the following log format, can someone help me with:

  1. How do I logstash to receive the syslog events which are not coming from a logstash forwarder?

  2. what is the logstash filter you recommend to use

  3. How do I set the filter in logstash

    01 2015-10-22T06:34:58.455Z 172.21.1.100 PaServer 0.08.002.02 INFO Changed logger configuration. username=nyotronsupport action_category=Logger Conf

    01 2015-10-22T06:34:29.532Z 172.21.1.100 PaServer 0.08.002.02 INFO logged in from IP: 172.21.1.1 username=nyotronsupport action_category=Login

    01 2015-10-22T06:29:42.232Z 172.21.1.100 PaServer 0.08.002.02 INFO Agent VLADIMIR-PC (172.21.1.125) updated BPM to version 507.349

    01 2015-10-22T06:29:29.134Z 172.21.1.100 PaServer 0.08.002.02 INFO Agent FREDDY-PC (172.21.1.187) updated BPM Policy

    01 2015-10-22T06:26:20.400Z 172.21.1.100 PaServer 0.08.002.02 INFO Agent 172.21.1.100 configured with slots: OSP, WEB, ADP, MESSANGER

    01 2015-10-21T21:56:35.406Z 172.21.1.100 PaServer 0.08.002.02 EVENT endpoint=NIRSH-LPT(172.21.1.165), type=File Access, caller=SYSTEM\SYSTEM, callee='C:\Windows\CSC\v2.0.6\namespace\nt-dc-1\Home\nirsh\Sales\Office One Note','My_Notebook-2015-10-21.zip','C:\Windows\CSC\v2.0.6\temp','temp-{c45eb58a-77f2-11e5-bae3-005056c00008}.dat', MD5=66666666666666666666666666666666, description=''

    01 2015-10-21T21:56:35.390Z 172.21.1.100 PaServer 0.08.002.02 EVENT endpoint=NIRSH-LPT(172.21.1.165), type=File Access, caller=SYSTEM\SYSTEM, callee='C:\Windows\CSC\v2.0.6\namespace\nt-dc-1\Home\nirsh\Sales\Office One Note','My_Notebook-2015-10-21.zip','C:\Windows\CSC\v2.0.6\temp','temp-{c45eb589-77f2-11e5-bae3-005056c00008}.dat', MD5=66666666666666666666666666666666, description=''

    01 2015-10-21T21:56:23.780Z 172.21.1.100 PaServer 0.08.002.02 EVENT endpoint=NIRSH-LPT(172.21.1.165), type=File Access, caller=SYSTEM\SYSTEM, callee='C:\Windows\CSC\v2.0.6\namespace\nt-dc-1\Home\nirsh\Sales\Office One Note','My_Notebook-2015-10-21.zip','C:\Windows\CSC\v2.0.6\temp','temp-{c45eb57d-77f2-11e5-bae3-005056c00008}.dat', MD5=66666666666666666666666666666666, description=''

    01 2015-10-21T18:36:10.625Z 172.21.1.100 PaServer 0.08.002.02 INFO Updated Agent NIRSH-LPT (172.21.1.165) to version 3.4

    01 2015-10-21T21:09:21.624Z 172.21.1.100 PaServer 0.08.002.02 EVENT endpoint=MAROM-LPT(172.21.1.109), type=Network Access, caller=C:\PROGRAM FILES (X86)\FORTINET\FORTICLIENT\UPDATE_TASK.EXE, callee='96.45.33.99','80','','', MD5=1DC7AFB00A67CCCDCC7D5D59FA0A4491, description=''

    01 2015-10-21T21:09:21.609Z 172.21.1.100 PaServer 0.08.002.02 EVENT endpoint=MAROM-LPT(172.21.1.109), type=Network Access, caller=C:\PROGRAM FILES (X86)\FORTINET\FORTICLIENT\UPDATE_TASK.EXE, callee='96.45.33.102','80','','', MD5=1DC7AFB00A67CCCDCC7D5D59FA0A4491, description=''

    01 2015-10-21T21:09:21.609Z 172.21.1.100 PaServer 0.08.002.02 EVENT endpoint=MAROM-LPT(172.21.1.109), type=Network Access, caller=C:\PROGRAM FILES (X86)\FORTINET\FORTICLIENT\UPDATE_TASK.EXE, callee='96.45.33.101','80','','', MD5=1DC7AFB00A67CCCDCC7D5D59FA0A4491, description='

Thanks for the help
Yoram

2

I would use either tcp or udp input and then a grok, something like:

input {
  tcp {
    port => 5514
    type => syslog
  }
}
filter {
  if [type] == "syslog" {
    grok {
      match => {
        message => '%{POSINT:syslog_pri} %{TIMESTAMP_ISO8601:syslog_timestamp} (%{IPORHOST:host} )?%{DATA:process}(?:[\[ ]?%{POSINT:pid}[\] ]?:?)? %{GREEDYDATA:syslog_message}'
      }
    }
    date {
      match => [ 'syslog_timestamp', 'ISO8601' ]
      remove_field => [ 'syslog_timestamp' ]
    }
    if [syslog_pri] {
      syslog_pri { }
    }
  }
}

Some details to fill in yourself since there is no standard format for syslog messages. Hope it helps.

3

Hi,

Thanks for the answer. I am flying back from a business travel so I will check it on Sunday and let you know if it worked for me.

two questions about the filter.
1.I saw you selected the port 5514 which is not a standard syslog port. Any reason to do so?
2. If I use UDP I am guessing that I need to change the TCP in the filter to UDP? right

Thanks
Yoram

4

Ok. Don't forget to update that grok line to match your format. https://grokdebug.herokuapp.com is your friend.

  1. The port is arbitrary - use what you prefer.
  2. Correct.

Good luck.

5

Thanks.

Grok is another thing to learn... Well I have the weekend

Thanks again

6

I saw you selected the port 5514 which is not a standard syslog port. Any reason to do so?

Unless you run Logstash as root (which isn't recommended) or employ one of the possible workarounds Logstash won't be able to listen on any port <1024.