Field name cannot contain '.'

radu.stefanache · October 29, 2015, 2:05pm

Hello everyone,

today I've upgraded to ELK 2.0 and did some tests .
I am currently parsing a line that looks like this :

key=value&key=value&key=value

Obviously I am using the kv filter . There is one issue though, I get sometimes keys like this one :

key[0].ip
key[1].ip

In 1.7 this was not a problem but now Elasticsearch will not store this .

Can I do anything from logstash or I have to create the mapping in advance on Elasticsearch and that should/will be fixed ?

"key[0]" : {
    "properties" : {
      "ip" : {
        "type" : "ip",
        "store": "yes"
        }
      }
}

Thank you in advance!

radu.stefanache · October 29, 2015, 3:00pm

I've been looking in the logs more and I see that any field that contains a dot in the name is not parsed :

MapperParsingException[Field name [elapsed.time] cannot contain '.']

Is there anything I can do from logstash or this is up to the mapping of the index ?

theuntergeek · October 29, 2015, 3:00pm

Field names cannot contain the . character in Elasticsearch 2.0.

I apologize for the inconvenience this will be, but you'll have to change all field names to not have a period in them.

radu.stefanache · October 29, 2015, 3:02pm

Hi Aaron,

thanks for the reply . As you can see in my previous comment, even the elapsed filter will not work .
Does it mean that some of the community filters will need to be modified ?

theuntergeek · October 29, 2015, 3:25pm

The problem isn't in the elapsed filter. You can use Logstash with dotted fields with no problems. You just can't send an event to Elasticsearch where any field names contain dots.

You'll need to reformat your data, whether with mutate filters (adding and removing fields as necessary), or at the source.

We've bandied about a "de-dot" filter or codec to cover instances such as these. Perhaps it's more pressing than we thought.

radu.stefanache · October 29, 2015, 3:35pm

Hi Aaron,

I will try to figure out a fix . The thing is that I am currently using the elapsed plugin and this plugin adds the following fields to the event :

elapsed.time
elapsed.timestamp_start

Elasticsearch doesn't like this and the events will not be indexed . I presume I am not the only person using this filter
As for the other fields, I will use mutate and ruby to create an object out of the field .

theuntergeek · October 29, 2015, 3:38pm

Indeed. We can push a change to the elapsed filter to replace the . with a _

This will be a breaking change for many people, however. We are trying to tread lightly with all such changes.

jcandalino · October 29, 2015, 6:15pm

I'm facing this issue as well. I can't even get my test cluster to start now.

Likely root cause: MapperParsingException[Field name [listing.requestAmount] cannot contain '.']

theuntergeek · October 29, 2015, 6:39pm

I'm going to start work on an addition to the mutate filter right now to "de-dot" fields, but those fields will have to be named.

A de-dot, "shotgun-approach" filter will come afterwards. This will iterate through all fields in the event to catch and change fields. This one will likely be a very expensive operation as it iterates through all, but I expect there will be some who don't know all of the fields which might have dots. This solution will be for them.

theuntergeek · October 29, 2015, 6:54pm

@radu.stefanache We just published version 3.0.0 of logstash-filter-elapsed. This is a breaking change.

All dots in field names and tags have been replaced by _ (an underscore)
This means you may need to change some conditionals in your Logstash configuration
Some of your Kibana dashboards and/or queries may require change
Any other outputs used will have to be adapted to use the new field names and tags.

You can upgrade to this version by doing:

bin/plugin update logstash-filter-elapsed

from your Logstash directory.

juerkan · October 29, 2015, 8:47pm

Hi Aaron,
We have also the problem with dots in our fields, and unfortunately we can not change the source.
Do you still think about a shotgun approach (maybe in the mutate filter)? This would be very helpful to us.

many thanks

Gary_Hodgson · November 1, 2015, 12:13am

I just came across this problem too as some dynamic fields are being added with a dot in the fieldname. After attempting to use the mutate filter with no luck I ended up using the ruby filter. I'll paste it below in case it's of use to others.

filter {
  ruby {
        code => "
          event.to_hash.keys.each { |k| event[ k.sub('.','_') ] = event.remove(k) if k.include?'.' }
        "
    }
}

juerkan · November 4, 2015, 7:19pm

Hi,
Thank you very much, it works.
Just a little issue with more then one dot in a field -> the ruby code replace just the first dot in a fieldname.
But that is not really a problem, because i insert the ruby filter twice.

many thanks
juergen

Gary_Hodgson · November 4, 2015, 7:39pm

Hi,

Ah yes, sorry I don't really know ruby that well. The following will replace all dots:

ruby {
        code => "
          event.to_hash.keys.each { |k| event[ k.gsub('.','_') ] = event.remove(k) if k.include?'.' }
        "
    }

Best regards,
Gary

Christopher_Blasnik · November 10, 2015, 10:03am

Hi,

I just came across the same problem

"type":"mapper_parsing_exception","reason":"Field name [data.0.count] cannot contain '.'"}

However, this is a HUGE show-stopper for us, since we are basically "flattening" json data before inserting them into Elasticsearch.

e.g.:
{
  "foo": {
    "bar": "something" 
  }
}

becomes

{ "foo.bar": "something" }

This is VALID Json. You are essentially breaking VALID JSON input.

~~I am really upset about this, what the heck is the reasoning behind this?~~
ok, I just found the responsible commit for this:

github.com/elastic/elasticsearch

Mappings: Enforce field names do not contain dot

Field names containing dots can cause problems. For example, @jpountz
made this recreation which cause no error, but can result in a
serialization exception if the type already exists:
https://gist.github.com/jpountz/8c66817e00a322b81f85

But this is not just a potential conflict. It also has larger problems,
since only the leaf mapper is created. The intermediate "foo" object
field would not exist if only "foo.bar" was in the mappings.

This change forbids the use of dots in field names. It also
fixes an issue with passing through the update_all_types setting,
which was always set to true whenever a type already existed (!).

I do not think we should worry about backwards compatibility here. This
should be a hard break (and added to the migration plugin).

by rjernst on 01:22AM - 07 Jul 15

changed 3 files with 28 additions and 10 deletions.

So gathering from this information: this change won't affect existing indices / field names?

theuntergeek · November 10, 2015, 3:02pm

The new de_dot filter can turn dotted fields into nested fields.

I'm not sure how it affects existing field names, but you would not be able to re-index that data without changing the field names.

Christopher_Blasnik · November 11, 2015, 2:53pm

Thanks! However, it does look like this filter might have a performance impact (doing what it does).

I reckon we will be reindexing (with field name changes) after all. Unfortunately this also affects another cluster which writes about 10 GiB of data every day. ouch. (might keep a "vintage" mode cluster / parallel software for that, though).

However, the question still remains: why this sort of breaking change when it might have been enough to state that mappings as posted in this gist https://gist.github.com/jpountz/8c66817e00a322b81f85 cannot be mixed?

Would it not have been better to try and fix the underlying cause? (I cannot judge the feasibility of that though!)

theuntergeek · November 12, 2015, 4:37am

@Christopher_Blasnik, because the ability was removed in Elasticsearch 2.0.

See the breaking changes section of the Elasticsearch documentation, under the header, Field names may not contain dots.

DigiAngel · November 12, 2015, 6:35pm

I'm assuming this impacts all the geoip items?

theuntergeek · November 12, 2015, 6:43pm

Kibana 3.x represents object structures (sub-fields) with dotted notation, but they are still object structures within Elasticsearch.

The GeoIP filter (as in 1.5.x and 2.x) sends an object, not dotted fields.