# Field name cannot contain '.'

**URL:** <https://discuss.elastic.co/t/field-name-cannot-contain/33251>\
**Category:** Logstash\
**Created:** [October 29, 2015, 2:05pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251 "2015-10-29T14:05:04Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![radu.stefanache](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@radu.stefanache](https://discuss.elastic.co/u/radu.stefanache)\
**Post date:** [October 29, 2015, 2:05pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/1 "2015-10-29T14:05:04Z")

</div>

Hello everyone,

today I've upgraded to ELK 2.0 and did some tests .  
I am currently parsing a line that looks like this :

```
key=value&key=value&key=value

```

Obviously I am using the kv filter . There is one issue though, I get sometimes keys like this one :

```
key[0].ip
key[1].ip

```

In 1.7 this was not a problem but now Elasticsearch will not store this .

Can I do anything from logstash or I have to create the mapping in advance on Elasticsearch and that should/will be fixed ?

```
"key[0]" : {
    "properties" : {
      "ip" : {
        "type" : "ip",
        "store": "yes"
        }
      }
}

```

Thank you in advance!

---

<div class="post-metadata">

**Author:** ![radu.stefanache](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@radu.stefanache](https://discuss.elastic.co/u/radu.stefanache)\
**Post date:** [October 29, 2015, 3:00pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/2 "2015-10-29T15:00:24Z")

</div>

I've been looking in the logs more and I see that any field that contains a dot in the name is not parsed :

```
MapperParsingException[Field name [elapsed.time] cannot contain '.']

```

Is there anything I can do from logstash or this is up to the mapping of the index ?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [October 29, 2015, 3:00pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/3 "2015-10-29T15:00:56Z")

</div>

Field names cannot contain the `.` character in Elasticsearch 2.0.

I apologize for the inconvenience this will be, but you'll have to change all field names to not have a period in them.

---

<div class="post-metadata">

**Author:** ![radu.stefanache](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@radu.stefanache](https://discuss.elastic.co/u/radu.stefanache)\
**Post date:** [October 29, 2015, 3:02pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/4 "2015-10-29T15:02:50Z")

</div>

Hi Aaron,

thanks for the reply . As you can see in my previous comment, even the elapsed filter will not work .  
Does it mean that some of the community filters will need to be modified ?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [October 29, 2015, 3:25pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/5 "2015-10-29T15:25:11Z")

</div>

The problem isn't in the elapsed filter. You can use Logstash with dotted fields with no problems. You just can't send an event to Elasticsearch where any field names contain dots.

You'll need to reformat your data, whether with mutate filters (adding and removing fields as necessary), or at the source.

We've bandied about a "de-dot" filter or codec to cover instances such as these. Perhaps it's more pressing than we thought.

---

<div class="post-metadata">

**Author:** ![radu.stefanache](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@radu.stefanache](https://discuss.elastic.co/u/radu.stefanache)\
**Post date:** [October 29, 2015, 3:35pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/6 "2015-10-29T15:35:13Z")

</div>

Hi Aaron,

I will try to figure out a fix . The thing is that I am currently using the elapsed plugin and this plugin adds the following fields to the event :

```
elapsed.time
elapsed.timestamp_start

```

Elasticsearch doesn't like this and the events will not be indexed . I presume I am not the only person using this filter 😄  
As for the other fields, I will use mutate and ruby to create an object out of the field .

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [October 29, 2015, 3:38pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/7 "2015-10-29T15:38:01Z")

</div>

Indeed. We can push a change to the elapsed filter to replace the . with a \_

This will be a breaking change for many people, however. We are trying to tread lightly with all such changes.

---

<div class="post-metadata">

**Author:** ![jcandalino](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jcandalino/32/5577_2.png) [@jcandalino](https://discuss.elastic.co/u/jcandalino)\
**Post date:** [October 29, 2015, 6:15pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/8 "2015-10-29T18:15:01Z")

</div>

I'm facing this issue as well. I can't even get my test cluster to start now.

```
Likely root cause: MapperParsingException[Field name [listing.requestAmount] cannot contain '.']
```

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [October 29, 2015, 6:39pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/9 "2015-10-29T18:39:33Z")

</div>

I'm going to start work on an addition to the mutate filter right now to "de-dot" fields, but those fields will have to be named.

A de-dot, "shotgun-approach" filter will come afterwards. This will iterate through all fields in the event to catch and change fields. This one will likely be a very expensive operation as it iterates through all, but I expect there will be some who don't know all of the fields which might have dots. This solution will be for them.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [October 29, 2015, 6:54pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/10 "2015-10-29T18:54:59Z")

</div>

@radu.stefanache We just published version 3.0.0 of logstash-filter-elapsed. This is a breaking change.

- All dots in field names _and_ tags have been replaced by \_ (an underscore)
- This means you _may_ need to change some conditionals in your Logstash configuration
- Some of your Kibana dashboards and/or queries may require change
- Any other outputs used will have to be adapted to use the new field names and tags.

You can upgrade to this version by doing:

```
bin/plugin update logstash-filter-elapsed

```

from your Logstash directory.

---

<div class="post-metadata">

**Author:** ![juerkan](https://avatars.discourse-cdn.com/v4/letter/j/4491bb/32.png) [@juerkan](https://discuss.elastic.co/u/juerkan)\
**Post date:** [October 29, 2015, 8:47pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/11 "2015-10-29T20:47:17Z")

</div>

Hi Aaron,  
We have also the problem with dots in our fields, and unfortunately we can not change the source.  
Do you still think about a shotgun approach (maybe in the mutate filter)? This would be very helpful to us.

many thanks

---

<div class="post-metadata">

**Author:** ![Gary\_Hodgson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gary_hodgson/32/5619_2.png) [@Gary\_Hodgson](https://discuss.elastic.co/u/Gary_Hodgson)\
**Post date:** [November 1, 2015, 12:13am UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/12 "2015-11-01T00:13:32Z")

</div>

I just came across this problem too as some dynamic fields are being added with a dot in the fieldname. After attempting to use the mutate filter with no luck I ended up using the ruby filter. I'll paste it below in case it's of use to others.

```
filter {
  ruby {
        code => "
          event.to_hash.keys.each { |k| event[k.sub('.','_')] = event.remove(k) if k.include?'.' }
        "
    }
}
```

---

<div class="post-metadata">

**Author:** ![juerkan](https://avatars.discourse-cdn.com/v4/letter/j/4491bb/32.png) [@juerkan](https://discuss.elastic.co/u/juerkan)\
**Post date:** [November 4, 2015, 7:19pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/13 "2015-11-04T19:19:47Z")

</div>

Hi,  
Thank you very much, it works.  
Just a little issue with more then one dot in a field -\> the ruby code replace just the first dot in a fieldname.  
But that is not really a problem, because i insert the ruby filter twice.

many thanks  
juergen

---

<div class="post-metadata">

**Author:** ![Gary\_Hodgson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gary_hodgson/32/5619_2.png) [@Gary\_Hodgson](https://discuss.elastic.co/u/Gary_Hodgson)\
**Post date:** [November 4, 2015, 7:39pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/14 "2015-11-04T19:39:59Z")

</div>

Hi,

Ah yes, sorry I don't really know ruby that well. The following will replace all dots:

```
ruby {
        code => "
          event.to_hash.keys.each { |k| event[k.gsub('.','_')] = event.remove(k) if k.include?'.' }
        "
    }

```

Best regards,  
Gary

---

<div class="post-metadata">

**Author:** ![Christopher\_Blasnik](https://avatars.discourse-cdn.com/v4/letter/c/9f8e36/32.png) [@Christopher\_Blasnik](https://discuss.elastic.co/u/Christopher_Blasnik)\
**Post date:** [November 10, 2015, 10:03am UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/15 "2015-11-10T10:03:40Z")

</div>

Hi,

I just came across the same problem

```auto
"type":"mapper_parsing_exception","reason":"Field name [data.0.count] cannot contain '.'"}

```

However, this is a HUGE show-stopper for us, since we are basically "flattening" json data before inserting them into Elasticsearch.

```auto
e.g.:
{
  "foo": {
    "bar": "something" 
  }
}

```

becomes

```auto
{ "foo.bar": "something" }

```

This is **VALID** Json. You are essentially breaking VALID JSON input.

~~I am really upset about this, what the heck is the reasoning behind this?~~  
ok, I just found the responsible commit for this:

> <https://github.com/elastic/elasticsearch/commit/aed1f68e494c65ad50b98a3e0a7a2b6a794b2965>

So gathering from this information: this change won't affect existing indices / field names?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [November 10, 2015, 3:02pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/16 "2015-11-10T15:02:52Z")

</div>

The new [`de_dot`](https://www.elastic.co/blog/introducing-the-de_dot-filter) filter can turn dotted fields into nested fields.

> [@Christopher\_Blasnik](#):
>
> So gathering from this information: this change won't affect existing indices / field names?

I'm not sure how it affects existing field names, but you would not be able to re-index that data without changing the field names.

---

<div class="post-metadata">

**Author:** ![Christopher\_Blasnik](https://avatars.discourse-cdn.com/v4/letter/c/9f8e36/32.png) [@Christopher\_Blasnik](https://discuss.elastic.co/u/Christopher_Blasnik)\
**Post date:** [November 11, 2015, 2:53pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/17 "2015-11-11T14:53:44Z")

</div>

Thanks! However, it does look like this filter might have a performance impact (doing what it does).

I reckon we will be reindexing (with field name changes) after all. Unfortunately this also affects another cluster which writes about 10 GiB of data every day. ouch. 😉 (might keep a "vintage" mode cluster / parallel software for that, though).

However, the question still remains: why this sort of breaking change when it might have been enough to state that mappings as posted in this gist [https://gist.github.com/jpountz/8c66817e00a322b81f85](https://gist.github.com/jpountz/8c66817e00a322b81f85) cannot be mixed?

Would it not have been better to try and _fix the underlying cause?_ 😄 (I cannot judge the feasibility of that though!)

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [November 12, 2015, 4:37am UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/18 "2015-11-12T04:37:41Z")

</div>

@Christopher_Blasnik, because the ability was removed in Elasticsearch 2.0.

See the breaking changes section of the Elasticsearch documentation, under the header, [Field names may not contain dots](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking_20_mapping_changes.html#_field_names_may_not_contain_dots).

---

<div class="post-metadata">

**Author:** ![DigiAngel](https://avatars.discourse-cdn.com/v4/letter/d/977dab/32.png) [@DigiAngel](https://discuss.elastic.co/u/DigiAngel)\
**Post date:** [November 12, 2015, 6:35pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/20 "2015-11-12T18:35:27Z")

</div>

I'm assuming this impacts all the geoip items?

![](https://us1.discourse-cdn.com/elastic/original/2X/4/4426f9c7172d8db63cc630def918fefdc80fa152.png) ![](https://us1.discourse-cdn.com/elastic/original/2X/3/3278ca9341cbad6a340fdbee5afac03e44d33d1f.png)

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [November 12, 2015, 6:43pm UTC](https://discuss.elastic.co/t/field-name-cannot-contain/33251/21 "2015-11-12T18:43:39Z")

</div>

Kibana 3.x represents object structures (sub-fields) with dotted notation, but they are still object structures within Elasticsearch.

The GeoIP filter (as in 1.5.x and 2.x) sends an object, not dotted fields.

[Next page](https://discuss.elastic.co/t/field-name-cannot-contain/33251.md?page=2)