Grok stopping output to elasticsearch from logstash

Elastic Stack Logstash
1

I am having an issue with grok on my syslog content.

I am getting logs but when I try to use grok on it everything stops.

Sample data;

<85>Jun 14 16:38:36--5:00 10.128.107.37 Action="drop" UUid="{0x0,0x0,0x0,0x0}" inzone="External"

There is more information after inzone, but I am trying to figure out the issue so right now I am just trying to match the Date IP of the host then GreedyData for the rest.
I had thought that removing greedydata might help but it does not.

grok

 <%{NUMBER}>%{SYSLOGTIMESTAMP:syslog_timestamp}\S+ %{IP:syslog_host} %{GREEDYDATA:syslog_message}

I get no errors, I just stop getting data in Kibana to see. if I comment out the grok in my config it starts coming back in again.

Logstash is starting after I uncomment the grok in the config file I just get no data. Then I remove it and the data comes back.

A full line of data with that grok works fine in the debugger.

Config file.
=-=-=-=-=-=-=-=-=-=-

input {
        tcp {
        type => "syslog"
        port => 5140
        }
}

input {
        udp {
        type => "syslog"
        port => 5140
        }
}


filter {
               grok {
                       match => { "message" => ["<%{NUMBER}>%{SYSLOGTIMESTAMP:syslog_timestamp}\S+ %{IP:syslog_hostname} %{GREEDYDATA:test}"
                               ]}
               }

        date {
                match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
                timezone => "UTC"
        }
}

output {
if [type] == "syslog" and "_grokparsefailure" in [tags] {
        file { path => "/var/log/failed_syslog_events" }
}
  elasticsearch {
        hosts => ["localhost:9200"]
  }
}

[EDIT: added code fences (~~~) around each code block to improve readability -- @yaauie]

2

Hi,

Have a very welcome at the Elastic community.
Thank you for your post.

At first I had a short look on your message format.
What is weird on this message for me, is the timestamp. Would you mind to describe the source of this message?
In the defined rfc definitions especially this string especially is not possible --5:00
https://www.ietf.org/rfc/rfc3164.txt rfc3164 bsd syslog protocol 4.1.2
https://www.ietf.org/rfc/rfc5424.txt rfc5424 syslog protocol 6.2.3.1. examples
https://www.ietf.org/rfc/rfc3339.txt rfc3339 internet timestamps
If you have a look to the examples there, there is always without a timezone information or if timezone, then like this +00:00 or -00:00 and without space after the seconds.

Ok if you have this grok active what happens in your logstash log? Would you mind posting the logfile and the logstash configuration file. Because I had a short test and apart, that I would change some things on the grok itself which we could discuss later on, the logstash configuration started very normally. What logstash version you are using?

3

The source is a checkpoint firewall;
Here is anything in logstash.yml that does not have a # in front.

 cat logstash.yml | grep -v \#
pipeline.batch.size: 225
path.data: /var/lib/logstash
pipeline.workers: 8
path.logs: /var/log/logstash

I think I have found where all my events are going to;

if [type] == "syslog" and "_grokparsefailure" in [tags] {
    file { path => "/var/log/failed_syslog_events" }
}

It looks like all the events are going in to that. So its failing grok parsing. I am just not sure why.

4

Which version are you running? With 7.1.1 the following parses just fine

input { generator { count => 1 lines => [ '<85>Jun 14 16:38:36--5:00 10.128.107.37 Action="drop" UUid="{0x0,0x0,0x0,0x0}" inzone="External"' ] } }
filter {
    grok {
        match => {
            "message" => [ "<%{NUMBER}>%{SYSLOGTIMESTAMP:syslog_timestamp}\S+ %{IP:syslog_hostname} %{GREEDYDATA:test}" ]
        }
    }
}
output { stdout { codec => rubydebug { metadata => false } } }

It took me a minute to understand that you are using \S+ to discard the timezone :slight_smile:

5

I am using 7.1.1.
I just tried your exact parser, which makes sense to me, and it it tests fine in grok debugger.
But its now tossing everything into my failed_syslog_events file.
Even stuff that is working in the debugger.

It seems to be something with the date.

This does not work;
<%{NUMBER}>%{SYSLOGTIMESTAMP:syslog_timestamp}\S+ %{IP:syslog_hostname} %{GREEDYDATA:test}

Nor this;
<%{NUMBER}>%{SYSLOGTIMESTAMP:syslog_timestamp}\S+ %{GREEDYDATA:test}

Or this;
<%{NUMBER}>%{SYSLOGTIMESTAMP:syslog_timestamp}%{GREEDYDATA:test}
But this does
<%{NUMBER}>%{GREEDYDATA:test}

6

I would suggest running with '--log.level debug --config.debug --config.test_and_exit' to make sure you really do just have the one grok filter that you think you have, and you are not picking up another version from a some.conf.bak file in the path.config directory.

7

Yes, they go to this file /var/log/failed_syslog_events.

I have seen this \S+ but this is what I wanted to write about after knowing what it.
With the date filter you transport these dates into your timestampe and at the moment you are assuming, that this timestamp is UTC time. But without the timezone information, this is - at least what I assume - just a wrong information. Would be good, if you have a look into file /var/log/failed_syslog_events, and look to the timestamps, if they really match your time, or if this time zone information is needed. Apart from that, yes, this line you posted is rendering.
Because these syslogs are not normated like a normal syslog format should be.
There are as well forum entries regarding Checkpoint Firewalls. Could you please check, what format you have configured in your Checkpoint firewall?


https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk122323
Do you maybe use this tool CPLogToSyslog?
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk115392

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk87560&partition=Advanced&product=Security
https://help.deepsecurity.trendmicro.com/Events-Alerts/syslog-parsing.html

Apart from the fact, that you can do the config.test_and_exit and if this is delivering OK, you could still disable the inputs with beat and output with elasticsearch and replace them by stdin {} and stdout{}.
Then start logstash by hand with -e parameter. This enables you to just pasting your logline to the logstash wating for input and pressing enter when logstash started?
What is the output you get back?
Otherwise, it still would make sense to use a configuration with a bit more debugging information if needed I could send one.

And like Badger told about already, be secure, that depending on your version, there are no additional configuration files. If you are not sure, please post a ls -lR /etc/logstash.

8

I just wrote my on definition for the date section and its fine now.

Thanks for helping.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.