Grok is not parsing GREEDYDATA field

Elastic Stack Logstash
1

Hi Everybody!

I have been facing a problem using Grok with OpenLDAP log, where it ignores a GREEDYDATA value in my match rule.
Here's a working rule example:

Source log line:
May 9 18:53:01 openldap-master slapd[4566]: conn=227215 op=0 BIND dn="cn=admin,dc=example,dc=com" method=128

Grok match rule:
match => [ "message", "%{SYSLOGBASE} conn=%{INT:ConnNumber} op=(?:[0-9]+) %{WORD:OpType} dn=%{GREEDYDATA:BindDN} method=128" ]

ElasticSearch output:
{ "_index": "logstash-2016.05.09", "_type": "Test", "_id": "AVSXq6-xMOq1MYWLr-cM", "_version": 1, "_score": 1, "_source": { "message": "May 9 18:53:01 openldap-master slapd[4566]: conn=227215 op=0 BIND dn="cn=admin,dc=example,dc=com" method=128", "@version": "1", "@timestamp": "2016-05-09T21:53:01.000Z", "path": "/tmp/openldap_log.txt", "host": "logstash-server", "type": "Test", "timestamp": "May 9 18:53:01", "logsource": "openldap-master", "program": "slapd", "pid": "4566", "ConnNumber": "227215", "OpType": "BIND", "BindDN": "cn=admin,dc=example,dc=com" } }

And here's the part that is not working:

Source log line:
May 9 18:56:55 openldap-master slapd[4566]: conn=226965 op=50 MOD dn="cn=user,ou=users,dc=example,dc=com"

Grok match rule:
match => [ "message", "%{SYSLOGBASE} conn=%{INT:ConnNumber} op=(?:[0-9]+) %{WORD:OpType} dn=%{GREEDYDATA:ModDN}" ]

ElasticSearch output:
{ "_index": "logstash-2016.05.09", "_type": "Test", "_id": "AVSXq6-xMOq1MYWLr-cL", "_version": 1, "_score": 1, "_source": { "message": "May 9 18:56:55 openldap-server slapd[4566]: conn=226965 op=50 MOD dn="cn=user,ou=users,dc=example,dc=com"", "@version": "1", "@timestamp": "2016-05-09T21:56:55.000Z", "path": "/tmp/openldap_log.txt", "host": "logstash-server", "type": "Test", "timestamp": "May 9 18:56:55", "logsource": "openldap-server", "program": "slapd", "pid": "4566", "ConnNumber": "226965", "OpType": "MOD" } }

As you can see, in the first example, Grok recognizes the GREEDYDATA field and send it correctly to ES, but in the second example, it doesn't recognize the ModDN field.
Does anybody know what could be happening here?!

Thanks in advance!

2

Try using a NOTSPACE instead of the greedy data.

3

And if your DN could contain spaces, you can use QUOTEDSTRING grok pattern.

1 Like
4

Thanks warkolm!

But as fbaligand said, many of my DNs contains spaces, so I think that QUOTEDSTRING would fit better.

5

I found out that the problem wasn't in the pattern used, but in the order that my "match" rules where placed in my logstash conf file.
Some of my log lines were being partially matched by a rule that was declared before the rule that fully matches those log lines.
For this reason, these lines were "grokked" partially too, with missing fields, generating the error explained in the post.
What I still can't understand is why logstash consider a partial match as a successfull match...

Thanks warkolm and fbaligand for the help!

6

You're welcome :slight_smile: