# Field name cannot contain '.' **URL:** \ **Category:** Logstash\ **Created:** [October 29, 2015, 2:05pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251 "2015-10-29T14:05:04Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![DigiAngel](https://avatars.discourse-cdn.com/v4/letter/d/977dab/32.png) [@DigiAngel](https://discuss.elastic.co/u/DigiAngel)\ **Post date:** [November 12, 2015, 6:51pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/22 "2015-11-12T18:51:13Z")
Ah...ok...that helps then..thank you...I was worried 🙂 ---
**Author:** ![bblank](https://avatars.discourse-cdn.com/v4/letter/b/f14d63/32.png) [@bblank](https://discuss.elastic.co/u/bblank)\ **Post date:** [January 25, 2016, 7:48pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/23 "2016-01-25T19:48:12Z")
Our use case for where "dots" may appear in a field name is after the kv {} filter runs. We don't always know the field names that log sources are sending us. The Ruby code works for us, but an "official" solution would be nice go have. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [January 25, 2016, 8:04pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/24 "2016-01-25T20:04:16Z")
@bblank There has been some internal discussion on how to better handle dotted fields (in the Elasticsearch team itself, not the Logstash team), but the dust has not yet settled. For the foreseeable future, the official solution is to use the aforementioned [de\_dot](https://www.elastic.co/blog/introducing-the-de_dot-filter) filter. ---
**Author:** ![bblank](https://avatars.discourse-cdn.com/v4/letter/b/f14d63/32.png) [@bblank](https://discuss.elastic.co/u/bblank)\ **Post date:** [January 26, 2016, 10:24pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/25 "2016-01-26T22:24:25Z")
This ruby solution works "most" of the time for us, but I just found some fields which have "[]" in the name and the "dots" are not being replaced. Any ruby coders out there willing to help? e.g. field name = ad.key[12]="some text value" ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [January 27, 2016, 12:01am UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/26 "2016-01-27T00:01:14Z")
Do you need the brackets? I would think those would be undesirable. Look into the mutate filter's gsub option. It will allow you to strip square braces. ---
**Author:** ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)\ **Post date:** [January 27, 2016, 12:04am UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/27 "2016-01-27T00:04:35Z")
I used this ruby filter instead of `de_dot` because my dotted fields are nested under `params` and I don't know what they are in advance: ```auto filter { ruby { code => " params = event['params'] && event['params'].to_hash params.keys.each { |k| params[k.gsub('.','_')] = params.delete(k) if k.include?'.' } unless params.nil? " } } ``` ---
**Author:** ![bblank](https://avatars.discourse-cdn.com/v4/letter/b/f14d63/32.png) [@bblank](https://discuss.elastic.co/u/bblank)\ **Post date:** [January 27, 2016, 2:06pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/28 "2016-01-27T14:06:36Z")
I am new to ELK so I may be wrong, but gsub only works on the field contents, not the name of the field. Right? ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [January 27, 2016, 2:48pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/29 "2016-01-27T14:48:08Z")
@bblank you're correct. I wasn't looking too close when I wrote that. This is a really tricky situation as that's likely to be an undesirable field name anyway. If you know what it is, you can do a `remove_field` _after_ copying the contents to a new field. If you don't know what the field name is, that makes things much harder. ---
**Author:** ![bblank](https://avatars.discourse-cdn.com/v4/letter/b/f14d63/32.png) [@bblank](https://discuss.elastic.co/u/bblank)\ **Post date:** [January 27, 2016, 3:46pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/30 "2016-01-27T15:46:55Z")
I think I found our best solution to our problem (which BTW is because we are using the kv { } filter and have NO control over what kv pairs we are sent... We have "dots", "[]", and some key names start with "\_" which are all no-no's to send to elasticsearch. This is simple and efficient. ``` kv { trimkey => "\.\[\]" prefix => "cef_extension_" source => "@message" } ``` ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [January 27, 2016, 4:20pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/31 "2016-01-27T16:20:08Z")
That's a great use of existing tools! ---
**Author:** ![mainak-s](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@mainak-s](https://discuss.elastic.co/u/mainak-s)\ **Post date:** [February 1, 2016, 11:44am UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/32 "2016-02-01T11:44:18Z")
Hi Gary I am using Logstash to parse logs from Bro IDS. This ruby filter works on most of the conf files except a file called "weird.conf" Can you please help. ![](https://us1.discourse-cdn.com/elastic/original/2X/2/2f9660ba5c67a53ce134a665b4578211a3845dfb.JPG) //below is the config file input { file { type =\> "bro-weird\_log" start\_position =\> "end" sincedb\_path =\> "/var/tmp/.bro\_weird\_sincedb" ``` #Edit the following path to reflect the location of your log files. You can also change the extension if you use something else path => "/usr/local/bro/logs/current/weird.log" ``` } } filter { #Let's get rid of those header lines; they begin with a hash if [message] =~ /^#/ { drop { } } #get rid of "." ruby { code =\> " event.to\_hash.keys.each { |k| event[k.sub('.','\_')] = event.remove(k) if k.include?'.' } " } #Now, using the csv filter, we can define the Bro log fields if [type] == "bro-weird\_log" { csv { ``` #weird.log:#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p name addl notice peer columns => ["ts","uid","id.orig_h","id.orig_p","id.resp_h","id.resp_p","name","addl","notice","peer"] #If you use a custom delimiter, change the following value in between the quotes to your delimiter. Otherwise, leave the next line alone. separator => " " } #Let's convert our timestamp into the 'ts' field, so we can use Kibana features natively date { match => ["ts", "UNIX"] } # add geoip attributes geoip { source => "id.orig_h" target => "orig_geoip" } geoip { source => "id.resp_h" target => "resp_geoip" } mutate { convert => ["id.orig_p", "integer"] convert => ["id.resp_p", "integer"] } ``` } } output { # stdout { codec =\> rubydebug } elasticsearch { hosts =\> localhost } } ---
**Author:** ![LeeSyd](https://avatars.discourse-cdn.com/v4/letter/l/8dc957/32.png) [@LeeSyd](https://discuss.elastic.co/u/LeeSyd)\ **Post date:** [February 8, 2016, 5:24pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/33 "2016-02-08T17:24:18Z")
Hmmm - a lot of the fields generated by Topbeat contain "." in them and I'm struggling to rename of remove them in logstash. Recommendation here? Anybody know if there is a way I can get Topbeat to name the fields differently before sending? ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [February 18, 2016, 7:55pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/34 "2016-02-18T19:55:17Z")
@LeeSyd, I've asked the Beats dev team, and the lead developer says that Topbeat has not used dots in field names since before v1.0. Which version are you using? ---
**Author:** ![ryanmaclean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryanmaclean/32/8230_2.png) [@ryanmaclean](https://discuss.elastic.co/u/ryanmaclean)\ **Post date:** [March 2, 2016, 9:37pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/35 "2016-03-02T21:37:10Z")
Can this be revisited for the new Elastic Stack release? We're stuck on 1.9 until you revert these breaking changes. ---
**Author:** ![ryanmaclean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryanmaclean/32/8230_2.png) [@ryanmaclean](https://discuss.elastic.co/u/ryanmaclean)\ **Post date:** [March 2, 2016, 9:49pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/36 "2016-03-02T21:49:30Z")
Pinging this from here: [https://github.com/elastic/logstash/issues/4752](https://github.com/elastic/logstash/issues/4752) - would be great to see where this discussion went. ---
**Author:** ![or4cle](https://avatars.discourse-cdn.com/v4/letter/o/e19b73/32.png) [@or4cle](https://discuss.elastic.co/u/or4cle)\ **Post date:** [March 18, 2016, 12:47am UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/37 "2016-03-18T00:47:20Z")
+1 to allowing dots again in subsequent versions. We also do not plan to move forward with ElasticSearch until this is resolved.. ---
**Author:** ![brain](https://avatars.discourse-cdn.com/v4/letter/b/439d5e/32.png) [@brain](https://discuss.elastic.co/u/brain)\ **Post date:** [March 28, 2016, 10:02am UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/38 "2016-03-28T10:02:13Z")
we would also like to see dots allowed in subsequent versions. stuck on 1.7 until this happens ---
**Author:** ![hanzmeier1234](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@hanzmeier1234](https://discuss.elastic.co/u/hanzmeier1234)\ **Post date:** [April 18, 2016, 2:53pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/39 "2016-04-18T14:53:39Z")
Hi, since every filter only works on toplevel I wrote a ruby filter to replace every dots with underscore in keys recursively: ```auto filter { ruby { init => " def remove_dots hash new = Hash.new hash.each { |k,v| if v.is_a? Hash v = remove_dots(v) end new[k.gsub('.','_')] = v if v.is_a? Array v.each { |elem| if elem.is_a? Hash elem = remove_dots(elem) end new[k.gsub('.','_')] = elem } unless v.nil? end } unless hash.nil? return new end " code => " event.instance_variable_set(:@data,remove_dots(event.to_hash)) " } } ``` cheers ---
**Author:** ![Nizar\_Khalifa](https://avatars.discourse-cdn.com/v4/letter/n/ed655f/32.png) [@Nizar\_Khalifa](https://discuss.elastic.co/u/Nizar_Khalifa)\ **Post date:** [June 22, 2016, 7:47am UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/40 "2016-06-22T07:47:11Z")
Hi @loren I have the same problem as you . I have a dotted field under `params` and i have to delete the `.` characters. I tried your solution but it doen't work. I'm not so familar with ruby, is there any specification in your code ? My Json looks like: > ``` > > { > > "ip" : "1.1.1.1" > > "params"{ > > ".title" : "title", > > "reference" : "ref" > > } > > ``` B.R ---
**Author:** ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)\ **Post date:** [June 24, 2016, 2:06pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/41 "2016-06-24T14:06:30Z")
That does not look like valid JSON to me. Could that be the problem? Either way, no, there's nothing special to add or include for that Ruby script. [Previous page](https://discuss.elastic.co/t/field-name-cannot-contain/33251.md?page=1) [Next page](https://discuss.elastic.co/t/field-name-cannot-contain/33251.md?page=3)