Field name cannot contain '.'

Elastic Stack Logstash
1

Hello everyone,

today I've upgraded to ELK 2.0 and did some tests .
I am currently parsing a line that looks like this :

key=value&key=value&key=value

Obviously I am using the kv filter . There is one issue though, I get sometimes keys like this one :

key[0].ip
key[1].ip

In 1.7 this was not a problem but now Elasticsearch will not store this .

Can I do anything from logstash or I have to create the mapping in advance on Elasticsearch and that should/will be fixed ?

"key[0]" : {
    "properties" : {
      "ip" : {
        "type" : "ip",
        "store": "yes"
        }
      }
}

Thank you in advance!

2

I've been looking in the logs more and I see that any field that contains a dot in the name is not parsed :

MapperParsingException[Field name [elapsed.time] cannot contain '.']

Is there anything I can do from logstash or this is up to the mapping of the index ?

3

Field names cannot contain the . character in Elasticsearch 2.0.

I apologize for the inconvenience this will be, but you'll have to change all field names to not have a period in them.

4

Hi Aaron,

thanks for the reply . As you can see in my previous comment, even the elapsed filter will not work .
Does it mean that some of the community filters will need to be modified ?

5

The problem isn't in the elapsed filter. You can use Logstash with dotted fields with no problems. You just can't send an event to Elasticsearch where any field names contain dots.

You'll need to reformat your data, whether with mutate filters (adding and removing fields as necessary), or at the source.

We've bandied about a "de-dot" filter or codec to cover instances such as these. Perhaps it's more pressing than we thought.

6

Hi Aaron,

I will try to figure out a fix . The thing is that I am currently using the elapsed plugin and this plugin adds the following fields to the event :

elapsed.time
elapsed.timestamp_start

Elasticsearch doesn't like this and the events will not be indexed . I presume I am not the only person using this filter :smile:
As for the other fields, I will use mutate and ruby to create an object out of the field .

7

Indeed. We can push a change to the elapsed filter to replace the . with a _

This will be a breaking change for many people, however. We are trying to tread lightly with all such changes.

1 Like
8

I'm facing this issue as well. I can't even get my test cluster to start now.

Likely root cause: MapperParsingException[Field name [listing.requestAmount] cannot contain '.']
9

I'm going to start work on an addition to the mutate filter right now to "de-dot" fields, but those fields will have to be named.

A de-dot, "shotgun-approach" filter will come afterwards. This will iterate through all fields in the event to catch and change fields. This one will likely be a very expensive operation as it iterates through all, but I expect there will be some who don't know all of the fields which might have dots. This solution will be for them.

10

@radu.stefanache We just published version 3.0.0 of logstash-filter-elapsed. This is a breaking change.

  • All dots in field names and tags have been replaced by _ (an underscore)
  • This means you may need to change some conditionals in your Logstash configuration
  • Some of your Kibana dashboards and/or queries may require change
  • Any other outputs used will have to be adapted to use the new field names and tags.

You can upgrade to this version by doing:

bin/plugin update logstash-filter-elapsed

from your Logstash directory.

1 Like
11

Hi Aaron,
We have also the problem with dots in our fields, and unfortunately we can not change the source.
Do you still think about a shotgun approach (maybe in the mutate filter)? This would be very helpful to us.

many thanks

12

I just came across this problem too as some dynamic fields are being added with a dot in the fieldname. After attempting to use the mutate filter with no luck I ended up using the ruby filter. I'll paste it below in case it's of use to others.

filter {
  ruby {
        code => "
          event.to_hash.keys.each { |k| event[ k.sub('.','_') ] = event.remove(k) if k.include?'.' }
        "
    }
}
13

Hi,
Thank you very much, it works.
Just a little issue with more then one dot in a field -> the ruby code replace just the first dot in a fieldname.
But that is not really a problem, because i insert the ruby filter twice.

many thanks
juergen

14

Hi,

Ah yes, sorry I don't really know ruby that well. The following will replace all dots:

ruby {
        code => "
          event.to_hash.keys.each { |k| event[ k.gsub('.','_') ] = event.remove(k) if k.include?'.' }
        "
    }

Best regards,
Gary

2 Likes
15

Hi,

I just came across the same problem

"type":"mapper_parsing_exception","reason":"Field name [data.0.count] cannot contain '.'"}

However, this is a HUGE show-stopper for us, since we are basically "flattening" json data before inserting them into Elasticsearch.

e.g.:
{
  "foo": {
    "bar": "something" 
  }
}

becomes

{ "foo.bar": "something" }

This is VALID Json. You are essentially breaking VALID JSON input.

I am really upset about this, what the heck is the reasoning behind this?
ok, I just found the responsible commit for this:

So gathering from this information: this change won't affect existing indices / field names?

1 Like
16

The new de_dot filter can turn dotted fields into nested fields.

I'm not sure how it affects existing field names, but you would not be able to re-index that data without changing the field names.

17

Thanks! However, it does look like this filter might have a performance impact (doing what it does).

I reckon we will be reindexing (with field name changes) after all. Unfortunately this also affects another cluster which writes about 10 GiB of data every day. ouch. :wink: (might keep a "vintage" mode cluster / parallel software for that, though).

However, the question still remains: why this sort of breaking change when it might have been enough to state that mappings as posted in this gist https://gist.github.com/jpountz/8c66817e00a322b81f85 cannot be mixed?

Would it not have been better to try and fix the underlying cause? :smile: (I cannot judge the feasibility of that though!)

18

@Christopher_Blasnik, because the ability was removed in Elasticsearch 2.0.

See the breaking changes section of the Elasticsearch documentation, under the header, Field names may not contain dots.

20

I'm assuming this impacts all the geoip items?

21

Kibana 3.x represents object structures (sub-fields) with dotted notation, but they are still object structures within Elasticsearch.

The GeoIP filter (as in 1.5.x and 2.x) sends an object, not dotted fields.