# Missing raw fields with default index
**URL:** https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484
**Category:** Elasticsearch
**Created:** [March 15, 2016, 7:37pm UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484 "2016-03-15T19:37:13Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@Kirrilian](https://discuss.elastic.co/u/Kirrilian)
#### Post date: [March 15, 2016, 7:37pm UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484/1 "2016-03-15T19:37:13Z")
I've setup an ELK server for production use after doing a proof of concept but I'm missing the raw fields on the production server like I have on the POC server. Everything else is working but I need the raw fields because some of the data has dashes so I can't use the regular fields.
Versions:
OS: CentOS 7
logstash-2.2.2-1.noarch
elasticsearch-2.2.0-1.noarch
kibana-4.4.1-1.x86\_64
Here is my current logstash.conf:
input {
syslog {
port =\> 5514
type =\> "syslog"
tags =\> ["syslog"]
}
}
filter {
if [type] == "syslog" {
grok {
break\_on\_match =\> true
# multiple filters included because all logs are dumped into one syslog and transferred over
# the log type for the parse is at the end of the line in a comment
match =\> [
"message", "^%{GREEDYDATA:event}[%{WORD:loglevel}] %{NUMBER:event\_id} %{GREEDYDATA:parsed\_msg}", #windows
"message", "^%{GREEDYDATA:event}[%{WORD:loglevel}] message repeated %{NUMBER} times: [ %{NUMBER:event\_id}", #windows
"message", "^%{HOSTNAME:event} %{NUMBER:event\_id} %{GREEDYDATA:parsed\_msg}", #windows
"message", "^%{HOSTNAME:event}[%{WORD} %{NUMBER:event\_id} %{GREEDYDATA:parsed\_msg}", #windows
"message", "^ %{NUMBER:event\_id} %{HOSTNAME:logsource} events %{HOSTNAME:event} %{GREEDYDATA:parsed\_msg}", #firewall
"message", "^ %{NUMBER:event\_id} %{HOSTNAME:logsource} events %{WORD:event} %{GREEDYDATA:parsed\_msg}", #firewall
"message", "^ %{NUMBER:event\_id} %{HOSTNAME:logsource} events type=%{WORD:event} %{GREEDYDATA:parsed\_msg}", #firewall
"message", "^ %{NUMBER:event\_id} %{HOSTNAME:logsource} events %{WORD:event}: %{GREEDYDATA:parsed\_msg}", #firewall
"message", "^ %{NUMBER:event\_id} %{HOSTNAME:logsource} %{HOSTNAME:event} %{GREEDYDATA:parsed\_msg}", #firewall
"message", "^%{IP:logsource}-1 %{GREEDYDATA:event}[%{NUMBER:event\_id}]: %{GREEDYDATA:parsed\_msg}", #switches
"message", "^%{IP:logsource} %{WORD:event} %{GREEDYDATA:parsed\_msg}", #switches
"message", "^syslog %{WORD:event} %{GREEDYDATA:parsed\_msg}", #switches
"message", "^ %{NUMBER} %{WORD} events type=%{WORD:event} %{GREEDYDATA:parsed\_msg}", #waps
"message", "^%{SYSLOG5424PRI}%{CISCOTIMESTAMP} %{HOSTNAME:logsource} [%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HAPROXYTIME} %{HOSTNAME:program} +%{WORD:loglevel} +%{GREEDYDATA:parsed\_msg}", #vcenter
"message", "^%{SYSLOG5424PRI}%{CISCOTIMESTAMP} %{HOSTNAME:logsource}.\*Z [%{WORD} %{WORD:loglevel} '%{WORD:program}' +%{GREEDYDATA:parsed\_msg}", #vcenter
"message", "^%{SYSLOG5424PRI}%{CISCOTIMESTAMP} %{HOSTNAME:logsource}.\*Z [%{WORD} %{WORD:loglevel} '[%{WORD:program}]' +%{GREEDYDATA:parsed\_msg}", #vcenter
"message", "^%{SYSLOG5424PRI}%{CISCOTIMESTAMP} %{HOSTNAME:logsource}.\*Z [%{WORD} %{WORD:loglevel} '%{WORD:program}'] +%{GREEDYDATA:parsed\_msg}", #vcenter
"message", "^%{SYSLOG5424PRI}%{CISCOTIMESTAMP} %{HOSTNAME:logsource} %{GREEDYDATA:parsed\_msg}", #vcenter
"message", "^%{SYSLOG5424PRI}%{SYSLOGLINE} %{GREEDYDATA:parsed\_msg}"] #vcenter
}
}
}
output {
elasticsearch {
hosts =\> ["localhost:9200"]
sniffing =\> true
manage\_template =\> false
index =\> "logstash-%{+YYYY.MM.dd}"
}
}
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 15, 2016, 8:08pm UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484/2 "2016-03-15T20:08:31Z")
> manage\_template =\> false
This tells Logstash that you're managing the index templates yourself. Are you actually doing that?
---
### Author:  [@Kirrilian](https://discuss.elastic.co/u/Kirrilian)
#### Post date: [March 15, 2016, 8:22pm UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484/3 "2016-03-15T20:22:54Z")
No I am not, so I can remove that, restart logstash and the raw fields will show back up?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 15, 2016, 8:24pm UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484/4 "2016-03-15T20:24:28Z")
Yes, but only for newly created indexes.
---
### Author:  [@Kirrilian](https://discuss.elastic.co/u/Kirrilian)
#### Post date: [March 15, 2016, 8:26pm UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484/5 "2016-03-15T20:26:31Z")
So that means those fields should show up tomorrow when it creates the daily index, awesome! Thank you so much, I didn't even think to check on that parameter, sorry about that.
---
### Author:  [@Kirrilian](https://discuss.elastic.co/u/Kirrilian)
#### Post date: [March 16, 2016, 3:00pm UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484/6 "2016-03-16T15:00:22Z")
Unfortunately that didn't work ☹ The new index was created but I still don't have the raw fields.
I'm stuck on this one and can't move forward with my dashboards until I have the raw fields.
Any ideas?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 16, 2016, 7:39pm UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484/7 "2016-03-16T19:39:09Z")
Check the installed mapping template(s). Do they contain what you expect (i.e. list .raw subfields for string fields)? If the index template looks correct, what happens if you create a new index (e.g. on a day some time in the future), does _that_ index get the correct mappings?
---
### Author:  [@Kirrilian](https://discuss.elastic.co/u/Kirrilian)
#### Post date: [March 16, 2016, 7:46pm UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484/8 "2016-03-16T19:46:45Z")
That's a good suggestion, I'll check on both of those. I do want to say that they showed up after I restarted elasticsearch (I only restarted logstash) but only for certain fields. Totally weird.
It should create raw entries for any custom fields I have too right? That's how it was setup before and all my stuff uses the raw fields when the data contains dashes.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 17, 2016, 6:27am UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484/9 "2016-03-17T06:27:42Z")
There should be .raw subfields for string fields that don't have an explicit configuration to _not_ have a .raw subfield.
---
### Author:  [@Kirrilian](https://discuss.elastic.co/u/Kirrilian)
#### Post date: [March 21, 2016, 3:46pm UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484/10 "2016-03-21T15:46:38Z")
I found the problem, I took some mutates out of my logstash config because I didn't think I needed them. I deleted all the indices and added the mutate statements back in. Everything is working fine now, thanks!
---
### Author:  [@Kirrilian](https://discuss.elastic.co/u/Kirrilian)
#### Post date: [March 21, 2016, 3:47pm UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484/11 "2016-03-21T15:47:31Z")
#convert our datatypes so we can report on them
mutate {
convert =\> { "event\_id" =\> "string" }
}
mutate {
convert =\> { "event" =\> "string" }
}
mutate {
convert =\> { "logsource" =\> "string" }
}
mutate {
convert =\> { "program" =\> "string" }
}
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:06pm UTC](https://discuss.elastic.co/t/missing-raw-fields-with-default-index/44484/12 "2017-07-05T23:06:32Z")