What is the best way to extract Syslog Fields with Beats Inputs

Elastic Stack Beats
1

Hi,

I setup ELK and Logstash with a Beats input:

input {
  beats {
    port => 5044
    ssl => false
    # ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
    # ssl_key => "/etc/pki/tls/private/logstash-beats.key"
    # codec => syslog
  }
}

I experimented with assigning a codec, but this doesn't work. Also the SSL stuff is inactive because this is a test environment.

Now I have filebeat, and I forward logs:

output:
  logstash:
    enabled: true
    hosts:
      - elk:5044
    # tls:
    # certificate_authorities:
    #    - /etc/pki/tls/certs/logstash-beats.crt
    timeout: 15

filebeat:
  prospectors:
    -
      paths:
        - /var/log/syslog
        - /var/log/auth.log
      document_type: syslog

I would like to get fields extracted in Kibana from messages like this:

Mar 15 15:56:30 mjo dnsmasq-dhcp[4818]: DHCPREQUEST(virbr0) 192.168.100.225 52:54:00:d7:47:49

In Kibana the application field (dnsmasq-dhcp) does not get extracted:

Pasted image1052×130 16.1 KB

Is there an easy way to perform the field extractions to benefit from the visualizations? I would like to use the Beats agent to forward Syslog as well as files, because the integration is central and easy to manage. Of course I can use Rsyslog and forward into Logstash directly, but I'd like to do all the Forwarding with Filebeat here.

Thanks ;),
Marius

2

Hi @marius, you don't need a codec here. You need grok and date filters. See the example filter in this Logstash configuration example.

If you need to debug grok patterns try this handy site: https://grokdebug.herokuapp.com/

3

Ok, I get that.

  • The type of the received syslog entry via Beats is "log".

So I would apply Grok filters from the Syslog example on the type "log"?

The problem with Beats is that this is a very generic type. It would be better if we could set types like "app log" "sys log" or "foo log" and match Grok patterns specifically per prospector this way.

5

The problem with Beats is that this is a very generic type. It would be better if we could set types like "app log" "sys log" or "foo log" and match Grok patterns specifically per prospector this way.

Your document_type: syslog should take care of that.

The configuration you posted isn't consistent with the Kibana screenshot; the configuration reads syslog and auth.log but the event in Kibana is from daemon.log.