# Logstash 2.1.0 stop creating raw fields

**URL:** <https://discuss.elastic.co/t/logstash-2-1-0-stop-creating-raw-fields/36598>\
**Category:** Logstash\
**Created:** [December 8, 2015, 3:06am UTC](https://discuss.elastic.co/t/logstash-2-1-0-stop-creating-raw-fields/36598 "2015-12-08T03:06:41Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![danilo\_gimenez\_ramos](https://avatars.discourse-cdn.com/v4/letter/d/c6cbf5/32.png) [@danilo\_gimenez\_ramos](https://discuss.elastic.co/u/danilo_gimenez_ramos)\
**Post date:** [December 8, 2015, 3:06am UTC](https://discuss.elastic.co/t/logstash-2-1-0-stop-creating-raw-fields/36598/1 "2015-12-08T03:06:41Z")

</div>

Hello,

I've upgrade all my client servers with Logstash 2.1.0 and then I wipe out all data in /var/lib/elasticsearch/ on Elasticsearch server in order to start form scratch with indices. After that I can't see the raw fields anymore in Kibana.  
When I run curl -XGET '[http://esserver](http://esserver):port/logstash-myindex-2015.12.08/\_mapping?pretty=true' the raw fields doesn't show.

How can I get back the raw fields?

Thanks,  
Danilo

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [December 8, 2015, 4:06am UTC](https://discuss.elastic.co/t/logstash-2-1-0-stop-creating-raw-fields/36598/2 "2015-12-08T04:06:31Z")

</div>

> [@danilo\_gimenez\_ramos](#):
>
> I wipe out all data in /var/lib/elasticsearch/ on Elasticsearch server in order to start form scratch with indices

Don't do that, you should **always** use the APIs to delete indices.

Did you change the mapping to include the `.raw` fields? If not then you need to!

---

<div class="post-metadata">

**Author:** ![danilo\_gimenez\_ramos](https://avatars.discourse-cdn.com/v4/letter/d/c6cbf5/32.png) [@danilo\_gimenez\_ramos](https://discuss.elastic.co/u/danilo_gimenez_ramos)\
**Post date:** [December 8, 2015, 11:11am UTC](https://discuss.elastic.co/t/logstash-2-1-0-stop-creating-raw-fields/36598/3 "2015-12-08T11:11:58Z")

</div>

Hello Warkolm, how can I change the mapping to include the raw fields?

I have this "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.4-java/lib/logstash/outputs/elasticsearch/elasticsearch-template.json" on server with Logstash installed but I am not using the elasticsearch java output plugin but the embeded elasticsearch plugin.

Thanks

---

<div class="post-metadata">

**Author:** ![danilo\_gimenez\_ramos](https://avatars.discourse-cdn.com/v4/letter/d/c6cbf5/32.png) [@danilo\_gimenez\_ramos](https://discuss.elastic.co/u/danilo_gimenez_ramos)\
**Post date:** [December 8, 2015, 11:35am UTC](https://discuss.elastic.co/t/logstash-2-1-0-stop-creating-raw-fields/36598/4 "2015-12-08T11:35:22Z")

</div>

Running curl -XGET elasticsearch-server:port/\_template?pretty=true  
I got this results:

{  
"logstash" : {  
"order" : 0,  
"template" : "logstash-_",  
"settings" : {  
"index" : {  
"refresh\_interval" : "5s"  
}  
},  
"mappings" : {  
"default" : {  
"dynamic\_templates" : [ {  
"message\_field" : {  
"mapping" : {  
"fielddata" : {  
"format" : "disabled"  
},  
"index" : "analyzed",  
"omit\_norms" : true,  
"type" : "string"  
},  
"match\_mapping\_type" : "string",  
"match" : "message"  
}  
}, {  
"string\_fields" : {  
"mapping" : {  
"fielddata" : {  
"format" : "disabled"  
},  
"index" : "analyzed",  
"omit\_norms" : true,  
"type" : "string",  
"fields" : {  
"raw" : {  
"ignore\_above" : 256,  
"index" : "not\_analyzed",  
"type" : "string",  
"doc\_values" : true  
}  
}  
},  
"match\_mapping\_type" : "string",  
"match" : "_"  
}  
}, {  
"float\_fields" : {  
"mapping" : {  
"type" : "float",  
"doc\_values" : true  
},  
"match\_mapping\_type" : "float",  
"match" : "_"  
}  
}, {  
"double\_fields" : {  
"mapping" : {  
"type" : "double",  
"doc\_values" : true  
},  
"match\_mapping\_type" : "double",  
"match" : "_"  
}  
}, {  
"byte\_fields" : {  
"mapping" : {  
"type" : "byte",  
"doc\_values" : true  
},  
"match\_mapping\_type" : "byte",  
"match" : "_"  
}  
}, {  
"short\_fields" : {  
"mapping" : {  
"type" : "short",  
"doc\_values" : true  
},  
"match\_mapping\_type" : "short",  
"match" : "_"  
}  
}, {  
"integer\_fields" : {  
"mapping" : {  
"type" : "integer",  
"doc\_values" : true  
},  
"match\_mapping\_type" : "integer",  
"match" : "_"  
}  
}, {  
"long\_fields" : {  
"mapping" : {  
"type" : "long",  
"doc\_values" : true  
},  
"match\_mapping\_type" : "long",  
"match" : "_"  
}  
}, {  
"date\_fields" : {  
"mapping" : {  
"type" : "date",  
"doc\_values" : true  
},  
"match\_mapping\_type" : "date",  
"match" : "_"  
}  
}, {  
"geo\_point\_fields" : {  
"mapping" : {  
"type" : "geo\_point",  
"doc\_values" : true  
},  
"match\_mapping\_type" : "geo\_point",  
"match" : "_"  
}  
} ],  
"\_all" : {  
"omit\_norms" : true,  
"enabled" : true  
},  
"properties" : {  
"@timestamp" : {  
"type" : "date",  
"doc\_values" : true  
},  
"geoip" : {  
"dynamic" : true,  
"type" : "object",  
"properties" : {  
"ip" : {  
"type" : "ip",  
"doc\_values" : true  
},  
"latitude" : {  
"type" : "float",  
"doc\_values" : true  
},  
"location" : {  
"type" : "geo\_point",  
"doc\_values" : true  
},  
"longitude" : {  
"type" : "float",  
"doc\_values" : true  
}  
}  
},  
"@version" : {  
"index" : "not\_analyzed",  
"type" : "string",  
"doc\_values" : true  
}  
}  
}  
},  
"aliases" : { }  
}  
}

I my template the raw fields, teorically, should be created, right?

---

<div class="post-metadata">

**Author:** ![danilo\_gimenez\_ramos](https://avatars.discourse-cdn.com/v4/letter/d/c6cbf5/32.png) [@danilo\_gimenez\_ramos](https://discuss.elastic.co/u/danilo_gimenez_ramos)\
**Post date:** [December 8, 2015, 12:00pm UTC](https://discuss.elastic.co/t/logstash-2-1-0-stop-creating-raw-fields/36598/5 "2015-12-08T12:00:02Z")

</div>

I am a complete noob.

In my Lostash conf I have some servers with:  
output {  
elasticsearch {  
hosts =\> "es-server:port"  
index =\> "logstash-pool-ldapspo-cons-%{+YYYY.MM.dd}"  
}  
}

and others with:  
output {  
if "127.0.0.1" not in [message] {  
elasticsearch {  
hosts =\> "es-server:port"  
index =\> "logstash-haproxy-spo-prov-%{+YYYY.MM.dd}"  
}  
}  
}

Ok, so with these configuration I don't get the raw fields on Kibana, but if I do this:

curl -XPUT [http://es-server](http://es-server):port/\_template/logstash -d '  
{  
"template" : "logstash-_",  
"settings" : {  
"index.refresh\_interval" : "5s"  
},  
"mappings" : {  
"default" : {  
"\_all" : {"enabled" : true, "omit\_norms" : true},  
"dynamic\_templates" : [ {  
"message\_field" : {  
"match" : "message",  
"match\_mapping\_type" : "string",  
"mapping" : {  
"type" : "string", "index" : "analyzed", "omit\_norms" : true,  
"fields" : {  
"raw" : {"type": "string", "index" : "not\_analyzed", "ignore\_above" : 256}  
}  
}  
}  
}, {  
"string\_fields" : {  
"match" : "_",  
"match\_mapping\_type" : "string",  
"mapping" : {  
"type" : "string", "index" : "analyzed", "omit\_norms" : true,  
"fields" : {  
"raw" : {"type": "string", "index" : "not\_analyzed", "ignore\_above" : 256}  
}  
}  
}  
} ],  
"properties" : {  
"@version": { "type": "string", "index": "not\_analyzed" },  
"geoip" : {  
"type" : "object",  
"dynamic": true,  
"properties" : {  
"location" : { "type" : "geo\_point" }  
}  
}  
}  
}  
}  
}  
'

And if go to Kibana and create a new index called "logstash-_" then a I have the raw fields. But if I create and index called "logstash-haproxy-spo-prov-_" I don't get it.

What am I missing here?

---

<div class="post-metadata">

**Author:** ![danilo\_gimenez\_ramos](https://avatars.discourse-cdn.com/v4/letter/d/c6cbf5/32.png) [@danilo\_gimenez\_ramos](https://discuss.elastic.co/u/danilo_gimenez_ramos)\
**Post date:** [December 9, 2015, 7:34pm UTC](https://discuss.elastic.co/t/logstash-2-1-0-stop-creating-raw-fields/36598/6 "2015-12-09T19:34:14Z")

</div>

I don't know what is happened with this environment which the raw field was not being created, so I do a fresh install on all Logstash servers and the Elasticsearch server and now the raw field are being created normally.

Danilo

---

<div class="post-metadata">

**Author:** ![vtst2412](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vtst2412/32/6228_2.png) [@vtst2412](https://discuss.elastic.co/u/vtst2412)\
**Post date:** [December 9, 2015, 8:11pm UTC](https://discuss.elastic.co/t/logstash-2-1-0-stop-creating-raw-fields/36598/7 "2015-12-09T20:11:35Z")

</div>

That's probably because the dynamic template that comes default with logstash (matches index "logstash-\*") was being overruled by another template with higher order whose pattern matches "logstash-haproxy-spo-prov-". In your old environment, did you create any new template?

---

<div class="post-metadata">

**Author:** ![danilo\_gimenez\_ramos](https://avatars.discourse-cdn.com/v4/letter/d/c6cbf5/32.png) [@danilo\_gimenez\_ramos](https://discuss.elastic.co/u/danilo_gimenez_ramos)\
**Post date:** [December 15, 2015, 10:43pm UTC](https://discuss.elastic.co/t/logstash-2-1-0-stop-creating-raw-fields/36598/8 "2015-12-15T22:43:16Z")

</div>

No, I don't. I just reinstalled everything.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:18am UTC](https://discuss.elastic.co/t/logstash-2-1-0-stop-creating-raw-fields/36598/9 "2017-07-06T05:18:17Z")

</div>