Shibboleth - Log Improvements, looking for feedback|suggestions

[Dr_Natas]

Looking for some feedback and/or improvement notes. The only way for me to get my shib logs is to do a rsync of my shib logs. This caused an issue that would cause my logs to be re-read causing duplicate entries for 24 hours. I found a custom ruby code that seems to be working for me, but if anyone has any suggestions on improving this code; I will be all ears. Thanks!

filter {
if "shibboleth" in [tags] {
	grok { 
		match => { 
			"message" => "%{TIMESTAMP_ISO8601:logdate} - %{LOGLEVEL:log_level} (?<ldap-notes>\[[a-zA-Z0-9.:]*\]) - %{IP:client} - Authentication %{DATA:Success} for dn: %{GREEDYDATA:dubs}"
		}
	}
       
	# Create custom hashes
    fingerprint {
 	    source              => "message"
    	target              => "[@metadata][fingerprint]"
	    concatenate_sources => "true"
    	method              => "SHA512"
		key                 => "logstash"
	}

    # Custom Code Found that deletes duplicate messages
    ruby {
      code => "
        s = ''
        h = event.to_hash
        h.each { |k, v|
          if k != '@timestamp'
            s += ',' + k.to_s + ':' + v.to_s
          end
        }
        event.set('message', s)
      "
    }

	#Manipulate the timestamp for the logs read
	date {
		match => [ "logdate", "YYYY-MM-dd HH:mm:ss,SSS" ]
		target => "@timestamp"
	}	

	geoip {
		source => "client"
		target => "geoip"
        add_tag => [ "geoip" ]
		add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
		add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
 	}

	mutate {
		convert => [ "[geoip][coordinates]", "float"]
	}	
	
	#Key Value Index Split
	kv {
		source => "dubs"
		field_split => ","
	}

	# Got lucky here, all logs are consistent thus magic 
	mutate {
   		gsub         => ["[dubs]", "DC|OU|CN|=", ""]
   		split        => { "[dubs]" => "," } # creates an array of strings in the [message] field]
   		add_field    => {"Common Name" => "%{[dubs][2]}.%{[dubs][3]}.%{[dubs][4]}"	}
		remove_field => [ "DC" ]
    	#remove_tag   => [ "aruba" ]
	}	
	
	# Clean up job
	mutate {
    	# Original message has been fully parsed, so remove it.
    	remove_field => [ "message" ]
		# Remove kv_pairs to avoid duplicate logs
    	remove_field => [ "dubs" ]
	}
	
	# Only digging for gold here | Dump all trash
	if "_grokparsefailure" in [tags]{
	     drop {}
     }
}

}

[Badger]

Did you intend the ruby filter to come before the fingerprint? I ask because the ruby filter sets the [message] field and the mutate deletes it, so the ruby is really a no-op.

[Dr_Natas]

To be honest this is my first hello world introduction to logstash. The way it sounded in my head was that once the message got hashed then the ruby code would apply and seeing that my main log file was rsync it would see the sames logs over and over again then delete them. Thus saving me duplicates. I may be wrong, and not even know it.