# Shibboleth Auth Logs \_grokparsefailure

**URL:** <https://discuss.elastic.co/t/shibboleth-auth-logs-grokparsefailure/151280>\
**Category:** Logstash\
**Created:** [October 5, 2018, 8:32pm UTC](https://discuss.elastic.co/t/shibboleth-auth-logs-grokparsefailure/151280 "2018-10-05T20:32:00Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dr\_Natas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dr_natas/32/36046_2.png) [@Dr\_Natas](https://discuss.elastic.co/u/Dr_Natas)\
**Post date:** [October 5, 2018, 8:32pm UTC](https://discuss.elastic.co/t/shibboleth-auth-logs-grokparsefailure/151280/1 "2018-10-05T20:32:00Z")

</div>

How filebeat sends my shibboleth logs.  
/etc/filebeat/filebeat.yml  
Note: Code is truncated.

```
#--------------------------- Prospectors Path ---------------------------------
#filebeat.config.inputs:
# enabled: true
# path: /etc/filebeat/conf.d/*.yml
#
filebeat.config.prospectors:
enabled: true
path: /etc/filebeat/conf.d/*.yml

```

This is my filebeat config for shibboleth log shipping.  
/etc/filebeat/conf.d/shibb.yml

```
====================================SHIBBOLETH========== =========================================
- type: log 
  paths:
  - /var/log/shibboleth/idp-process.log

exclude_lines: ['^DBG']
tags: ["shibboleth"]

# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']

# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
exclude_files: ['.gz$']

# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
fields:
  level: debug
  review: 1

```

This is my shiv-dev code.

```
input {
generator {
	count => 1 
	message => "2018-09-27 21:23:54,633 - INFO [org.ldaptive.auth.Authenticator:282] - Authentication succeeded for dn: CN=jane.doe192,OU=Students,DC=three,DC=fifty,DC=onion"

}
}

filter {

grok { 
	match => { 
		"message" => "%{TIMESTAMP_ISO8601:logdate} - %{LOGLEVEL:log_level} (?<ldap-notes>\[[a-zA-Z0-9.:]*\]) - Authentication %{DATA:Success} for dn: %{GREEDYDATA:dubs}"
	}
}

kv {
	source => "dubs"
	field_split => ","
}

mutate {
   gsub => ["[dubs]", "DC|OU|CN|=", ""]
   split => { "[dubs]" => "," } # creates an array of strings in the [message] field]
   add_field => {
		"Common Name" => "%{[dubs][2]}.%{[dubs][3]}.%{[dubs][4]}"
   }
   remove_field => ["DC"]

   remove_tag => ["othertags"]
   }

mutate {
    # Original message has been fully parsed, so remove it.
    remove_field => ["message"]
	# Remove kv_pairs to avoid duplicate logs
    remove_field => ["dubs"]
}

}

output { stdout { codec => rubydebug } }

```

Sample output of dev code.

```
{
 "@timestamp" => 2018-10-05T18:49:18.663Z,
    "logdate" => "2018-09-27 21:23:54,633",
 "ldap-notes" => "[org.ldaptive.auth.Authenticator:282]",
   "@version" => "1",
    "Success" => "succeeded",
       "host" => "localhost",
   "sequence" => 0,
  "log_level" => "INFO",
         "CN" => "jane.doe192",
"Common Name" => "three.fifty.onion",
         "OU" => "Students"
}

```

I'm getting a \_grokparsefailure and for the life of me can't figure out why I'm getting this error.

 ![grokparsefailure](https://us1.discourse-cdn.com/elastic/original/3X/7/5/7509d4e08a405b62c75817b30d5fd384d9807017.png)

I was able to turn on debbuging.

```
[2018-10-05T14:39:23,556][DEBUG][logstash.pipeline] filter received {"event"=>{"beat"=>{"name"=>"logserver", "hostname"=>"local.host.name", "version"=>"6.3.2"}, "prospector"=>{"type"=>"log"}, "message"=>"2018-10-05 21:27:41,822 - INFO [org.ldaptive.auth.Authenticator:282] - Authentication failed for dn: CN=jane.doe057,OU=Staff,DC=three,DC=fifty,DC=onion", "tags"=>["shibboleth", "beats_input_codec_plain_applied"], "offset"=>27724776, "source"=>"/var/log/shibboleth/idp-process.log", "@timestamp"=>2018-10-05T21:39:21.163Z, "host"=>{"name"=>"serverSendingLogs"}, "fields"=>{"level"=>"debug", "env"=>"production", "review"=>1}, "input"=>{"type"=>"log"}, "@version"=>"1"}}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 2, 2018, 8:32pm UTC](https://discuss.elastic.co/t/shibboleth-auth-logs-grokparsefailure/151280/2 "2018-11-02T20:32:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.