# OpenLdap Log parsing? (strange errors)

**URL:** <https://discuss.elastic.co/t/openldap-log-parsing-strange-errors/204701>\
**Category:** Logstash\
**Created:** [October 22, 2019, 5:35pm UTC](https://discuss.elastic.co/t/openldap-log-parsing-strange-errors/204701 "2019-10-22T17:35:45Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![iukea](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iukea/32/49083_2.png) [@iukea](https://discuss.elastic.co/u/iukea)\
**Post date:** [October 22, 2019, 5:35pm UTC](https://discuss.elastic.co/t/openldap-log-parsing-strange-errors/204701/1 "2019-10-22T17:35:45Z")

</div>

I am having a crazy hard time trying to parse logs coming from my LDAP servers.

- Sending data Vida filebeat to logstash.

And found a nice example openldap.conf file someone was working on (it works okay but i have made some changes)

- [https://github.com/ltb-project/openldap-elk/blob/master/logstash/openldap.conf](https://github.com/ltb-project/openldap-elk/blob/master/logstash/openldap.conf)

and now i am hitting the same type of parsing errors as this user (post linked Below)

> [@Help needed with OpenLDAP Logstash filter](https://discuss.elastic.co/t/help-needed-with-openldap-logstash-filter/104605):
>
> I have the following logstash.conf file that I have taken from github input { file { path =\> "/var/log/openldap.log" start\_position =\> beginning type =\> "openldap\_log\_file" } } filter { grok { match =\> [ "message", "%{SYSLOGBASE} (?:(?:\<= (?:b|m)db\_%{DATA:index\_error\_filter\_type}\_candidates: \(%{WORD:index\_error\_attribute\_name}\) not indexed)|(?:ppolicy\_%{DATA:ppolicy\_op}: %{DATA:ppolicy\_data})|(?:connection\_input: conn=%{INT:connection} deferring operation: %{DATA:d…

My issue is I am still getting large amounts of log data just shoved into the message field and it is not getting parsed correctly by my grok patters.

Was curious if anyone in the Elk community has built a logstash filter for OpenLdap logs?

---

<div class="post-metadata">

**Author:** ![iukea](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iukea/32/49083_2.png) [@iukea](https://discuss.elastic.co/u/iukea)\
**Post date:** [October 22, 2019, 6:05pm UTC](https://discuss.elastic.co/t/openldap-log-parsing-strange-errors/204701/2 "2019-10-22T18:05:16Z")

</div>

Attached is an example log i am having issues with

```auto
{
  "_index": "ldap-0.1-2019.10.22",
  "_type": "_doc",
  "_id": "6odc9G0B6D8KZPEhuaYL",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-10-22T16:46:19.718Z",
    "type": "log",
    "source": "/var/log/misclog",
    "beat": {
      "version": "5.6.12",
      "name": "myserver",
      "hostname": "myhost"
    },
    "tags": [
      "beats_input_codec_plain_applied",
      "_grokparsefailure"
    ],
    "@version": "1",
    "input_type": "log",
    "message": "Oct 22 12:46:19 admin02.dal.sync.lan slapd[28575]: do_syncrep2: rid=005 LDAP_RES_INTERMEDIATE - NEW_COOKIE",
    "fields": {
      "index_name": "ldap_log_ship"
    },
    "offset": 90126486
  },
  "fields": {
    "@timestamp": [
      "2019-10-22T16:46:19.718Z"
    ]
  },
  "sort": [
    1571762779718
  ]
}{
  "_index": "ldap-0.1-2019.10.22",
  "_type": "_doc",
  "_id": "6odc9G0B6D8KZPEhuaYL",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-10-22T16:46:19.718Z",
    "type": "log",
    "source": "/var/log/misclog",
    "beat": {
      "version": "5.6.12",
      "name": "hello.dal.sync.lan",
      "hostname": "hello.dal.sync.lan"
    },
    "tags": [
      "beats_input_codec_plain_applied",
      "_grokparsefailure"
    ],
    "@version": "1",
    "input_type": "log",
    "message": "Oct 22 12:46:19 hello.dal.sync.lan slapd[28575]: do_syncrep2: rid=005 LDAP_RES_INTERMEDIATE - NEW_COOKIE",
    "fields": {
      "index_name": "ldap_log_ship"
    },
    "offset": 90126486
  },
  "fields": {
    "@timestamp": [
      "2019-10-22T16:46:19.718Z"
    ]
  },
  "sort": [
    1571762779718
  ]
}

```

and my logstash .conf file

```
filter {
  grok {
    match => ["message", "%{SYSLOGBASE} (?:(?:<= (?:b|m)db_%{DATA:index_error_filter_type}_candidates: \(%{WORD:index_error_attribute_name}\) not indexed)|(?:ppolicy_%{DATA:ppolicy_op}: %{DATA:ppolicy_data})|(?:connection_input: conn=%{INT:connection} deferring operation: %{DATA:deferring_op})|(?:connection_read\(%{INT:fd_number}\): no connection!)|(?:conn=%{INT:connection} (?:(?:fd=%{INT:fd_number} (?:(?:closed(?: \(connection lost\)|))|(?:ACCEPT from IP=%{IP:src_ip}\:%{INT:src_port} \(IP=%{IP:dst_ip}\:%{INT:dst_port}\))|(?:TLS established tls_ssf=%{INT:tls_ssf} ssf=%{INT:ssf})))|(?:op=%{INT:operation_number} (?:(?:(?:(?:SEARCH )|(?:))RESULT (?:tag=%{INT:tag}|oid=(?:%{DATA:oid}(?:))) err=%{INT:error_code}(?:(?: nentries=%{INT:nentries})|(?:)) text=(?:(?:%{DATA:error_text})|(?:)))|(?:%{WORD:operation_name}(?:(?: %{DATA:data})|(?:))))))))%{SPACE}$"]
  }
  date {
    locale => "en"
    match => ["timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]
    target => "@timestamp"
  }
  if [operation_name] == "BIND" {
    grok {
      match => ["data", "(?:(?:(?<bind_dn>anonymous))|(?:dn=\"%{DATA:bind_dn}\")) (?:(?:method=%{WORD:bind_method})|(?:mech=%{WORD:bind_mech} ssf=%{INT:bind_ssf}))%{SPACE}$"]
      remove_field => ["data"]
    }
  }
  if [operation_name] == "SRCH" {
    grok {
      match => ["data", "(?:(?:base=\"%{DATA:search_base}\" scope=%{INT:search_scope} deref=%{INT:search_deref} filter=\"%{DATA:search_filter}\")|(?:attr=%{DATA:search_attr}))%{SPACE}$"]
      remove_field => ["data"]
    }
  }
  if [operation_name] == "MOD" {
    grok {
      match => ["data", "(?:(?:dn=\"%{DATA:mod_dn}\")|(?:attr=%{DATA:mod_attr}))%{SPACE}$"]
      remove_field => ["data"]
    }
  }
  if [operation_name] == "MODRDN" {
    grok {
      match => ["data", "dn=\"%{DATA:modrdn_dn}\"%{SPACE}$"]
      remove_field => ["data"]
    }
  }
  if [operation_name] == "ADD" {
    grok {
      match => ["data", "dn=\"%{DATA:add_dn}\"%{SPACE}$"]
      remove_field => ["data"]
    }
  }
  if [operation_name] == "DEL" {
    grok {
      match => ["data", "dn=\"%{DATA:del_dn}\"%{SPACE}$"]
      remove_field => ["data"]
    }
  }
  if [operation_name] == "CMP" {
    grok {
      match => ["data", "dn=\"%{DATA:cmp_dn}\" attr=\"%{DATA:cmp_attr}\"%{SPACE}$"]
      remove_field => ["data"]
    }
  }
  if [operation_name] == "EXT" {
    grok {
      match => ["data", "oid=%{DATA:ext_oid}%{SPACE}$"]
      remove_field => ["data"]
    }
  }
  if [ppolicy_op] == "bind" {
    grok {
      match => ["ppolicy_data", "(?:(?:Entry %{DATA:ppolicy_bind_dn} has an expired password: %{INT:ppolicy_grace} grace logins)|(?:Setting warning for password expiry for %{DATA:ppolicy_bind_dn} = %{INT:ppolicy_expiration} seconds))%{SPACE}$"]
      remove_field => ["ppolicy_data"]
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 19, 2019, 6:05pm UTC](https://discuss.elastic.co/t/openldap-log-parsing-strange-errors/204701/3 "2019-11-19T18:05:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.