Ah...ok...that helps then..thank you...I was worried 
Our use case for where "dots" may appear in a field name is after the kv {} filter runs. We don't always know the field names that log sources are sending us. The Ruby code works for us, but an "official" solution would be nice go have.
@bblank There has been some internal discussion on how to better handle dotted fields (in the Elasticsearch team itself, not the Logstash team), but the dust has not yet settled. For the foreseeable future, the official solution is to use the aforementioned de_dot filter.
This ruby solution works "most" of the time for us, but I just found some fields which have "[ ]" in the name and the "dots" are not being replaced. Any ruby coders out there willing to help? e.g. field name = ad.key[12]="some text value"
Do you need the brackets? I would think those would be undesirable. Look into the mutate filter's gsub option. It will allow you to strip square braces.
I used this ruby filter instead of de_dot because my dotted fields are nested under params and I don't know what they are in advance:
filter {
ruby {
code => "
params = event['params'] && event['params'].to_hash
params.keys.each { |k| params[ k.gsub('.','_') ] = params.delete(k) if k.include?'.' } unless params.nil?
"
}
}
I am new to ELK so I may be wrong, but gsub only works on the field contents, not the name of the field. Right?
@bblank you're correct. I wasn't looking too close when I wrote that.
This is a really tricky situation as that's likely to be an undesirable field name anyway. If you know what it is, you can do a remove_field after copying the contents to a new field. If you don't know what the field name is, that makes things much harder.
I think I found our best solution to our problem (which BTW is because we are using the kv { } filter and have NO control over what kv pairs we are sent...
We have "dots", "[ ]", and some key names start with "_" which are all no-no's to send to elasticsearch. This is simple and efficient.
kv {
trimkey => "\.\[\]"
prefix => "cef_extension_"
source => "@message"
}That's a great use of existing tools!
Hi Gary
I am using Logstash to parse logs from Bro IDS. This ruby filter works on most of the conf files except a file called "weird.conf"
Can you please help.
//below is the config file
input {
file {
type => "bro-weird_log"
start_position => "end"
sincedb_path => "/var/tmp/.bro_weird_sincedb"
#Edit the following path to reflect the location of your log files. You can also change the extension if you use something else
path => "/usr/local/bro/logs/current/weird.log"
}
}
filter {
#Let's get rid of those header lines; they begin with a hash
if [message] =~ /^#/ {
drop { }
}
#get rid of "."
ruby {
code => "
event.to_hash.keys.each { |k| event[ k.sub('.','_') ] = event.remove(k) if k.include?'.' }
"
}
#Now, using the csv filter, we can define the Bro log fields
if [type] == "bro-weird_log" {
csv {
#weird.log:#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p name addl notice peer
columns => ["ts","uid","id.orig_h","id.orig_p","id.resp_h","id.resp_p","name","addl","notice","peer"]
#If you use a custom delimiter, change the following value in between the quotes to your delimiter. Otherwise, leave the next line alone.
separator => " "
}
#Let's convert our timestamp into the 'ts' field, so we can use Kibana features natively
date {
match => [ "ts", "UNIX" ]
}
# add geoip attributes
geoip {
source => "id.orig_h"
target => "orig_geoip"
}
geoip {
source => "id.resp_h"
target => "resp_geoip"
}
mutate {
convert => [ "id.orig_p", "integer" ]
convert => [ "id.resp_p", "integer" ]
}
}
}
output {
stdout { codec => rubydebug }
elasticsearch { hosts => localhost }
}
Hmmm - a lot of the fields generated by Topbeat contain "." in them and I'm struggling to rename of remove them in logstash. Recommendation here?
Anybody know if there is a way I can get Topbeat to name the fields differently before sending?
@LeeSyd, I've asked the Beats dev team, and the lead developer says that Topbeat has not used dots in field names since before v1.0. Which version are you using?
Can this be revisited for the new Elastic Stack release? We're stuck on 1.9 until you revert these breaking changes.
Pinging this from here: https://github.com/elastic/logstash/issues/4752 - would be great to see where this discussion went.
+1 to allowing dots again in subsequent versions. We also do not plan to move forward with ElasticSearch until this is resolved..
1 Like
we would also like to see dots allowed in subsequent versions. stuck on 1.7 until this happens
Hi,
since every filter only works on toplevel I wrote a ruby filter to replace every dots with underscore in keys recursively:
filter {
ruby {
init => "
def remove_dots hash
new = Hash.new
hash.each { |k,v|
if v.is_a? Hash
v = remove_dots(v)
end
new[ k.gsub('.','_') ] = v
if v.is_a? Array
v.each { |elem|
if elem.is_a? Hash
elem = remove_dots(elem)
end
new[ k.gsub('.','_') ] = elem
} unless v.nil?
end
} unless hash.nil?
return new
end
"
code => "
event.instance_variable_set(:@data,remove_dots(event.to_hash))
"
}
}cheers
3 Likes
Hi @loren
I have the same problem as you .
I have a dotted field under params and i have to delete the . characters.
I tried your solution but it doen't work.
I'm not so familar with ruby, is there any specification in your code ?
My Json looks like:
> { > "ip" : "1.1.1.1" > "params"{ > ".title" : "title", > "reference" : "ref" > }
B.R
That does not look like valid JSON to me. Could that be the problem? Either way, no, there's nothing special to add or include for that Ruby script.