Missing raw fields with default index

Elastic Stack Elasticsearch
1

I've setup an ELK server for production use after doing a proof of concept but I'm missing the raw fields on the production server like I have on the POC server. Everything else is working but I need the raw fields because some of the data has dashes so I can't use the regular fields.

Versions:
OS: CentOS 7
logstash-2.2.2-1.noarch
elasticsearch-2.2.0-1.noarch
kibana-4.4.1-1.x86_64

Here is my current logstash.conf:

input {
syslog {
port => 5514
type => "syslog"
tags => ["syslog"]
}
}

filter {
if [type] == "syslog" {
grok {
break_on_match => true
# multiple filters included because all logs are dumped into one syslog and transferred over
# the log type for the parse is at the end of the line in a comment
match => [
"message", "^%{GREEDYDATA:event}[%{WORD:loglevel}] %{NUMBER:event_id} %{GREEDYDATA:parsed_msg}", #windows
"message", "^%{GREEDYDATA:event}[%{WORD:loglevel}] message repeated %{NUMBER} times: [ %{NUMBER:event_id}", #windows
"message", "^%{HOSTNAME:event} %{NUMBER:event_id} %{GREEDYDATA:parsed_msg}", #windows
"message", "^%{HOSTNAME:event}[%{WORD} %{NUMBER:event_id} %{GREEDYDATA:parsed_msg}", #windows
"message", "^ %{NUMBER:event_id} %{HOSTNAME:logsource} events %{HOSTNAME:event} %{GREEDYDATA:parsed_msg}", #firewall
"message", "^ %{NUMBER:event_id} %{HOSTNAME:logsource} events %{WORD:event} %{GREEDYDATA:parsed_msg}", #firewall
"message", "^ %{NUMBER:event_id} %{HOSTNAME:logsource} events type=%{WORD:event} %{GREEDYDATA:parsed_msg}", #firewall
"message", "^ %{NUMBER:event_id} %{HOSTNAME:logsource} events %{WORD:event}: %{GREEDYDATA:parsed_msg}", #firewall
"message", "^ %{NUMBER:event_id} %{HOSTNAME:logsource} %{HOSTNAME:event} %{GREEDYDATA:parsed_msg}", #firewall
"message", "^%{IP:logsource}-1 %{GREEDYDATA:event}[%{NUMBER:event_id}]: %{GREEDYDATA:parsed_msg}", #switches
"message", "^%{IP:logsource} %{WORD:event} %{GREEDYDATA:parsed_msg}", #switches
"message", "^syslog %{WORD:event} %{GREEDYDATA:parsed_msg}", #switches
"message", "^ %{NUMBER} %{WORD} events type=%{WORD:event} %{GREEDYDATA:parsed_msg}", #waps
"message", "^%{SYSLOG5424PRI}%{CISCOTIMESTAMP} %{HOSTNAME:logsource} [%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HAPROXYTIME} %{HOSTNAME:program} +%{WORD:loglevel} +%{GREEDYDATA:parsed_msg}", #vcenter
"message", "^%{SYSLOG5424PRI}%{CISCOTIMESTAMP} %{HOSTNAME:logsource}.*Z [%{WORD} %{WORD:loglevel} '%{WORD:program}' +%{GREEDYDATA:parsed_msg}", #vcenter
"message", "^%{SYSLOG5424PRI}%{CISCOTIMESTAMP} %{HOSTNAME:logsource}.*Z [%{WORD} %{WORD:loglevel} '[%{WORD:program}]' +%{GREEDYDATA:parsed_msg}", #vcenter
"message", "^%{SYSLOG5424PRI}%{CISCOTIMESTAMP} %{HOSTNAME:logsource}.*Z [%{WORD} %{WORD:loglevel} '%{WORD:program}'] +%{GREEDYDATA:parsed_msg}", #vcenter
"message", "^%{SYSLOG5424PRI}%{CISCOTIMESTAMP} %{HOSTNAME:logsource} %{GREEDYDATA:parsed_msg}", #vcenter
"message", "^%{SYSLOG5424PRI}%{SYSLOGLINE} %{GREEDYDATA:parsed_msg}"] #vcenter
}
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "logstash-%{+YYYY.MM.dd}"
}
}

2

manage_template => false

This tells Logstash that you're managing the index templates yourself. Are you actually doing that?

3

No I am not, so I can remove that, restart logstash and the raw fields will show back up?

4

Yes, but only for newly created indexes.

5

So that means those fields should show up tomorrow when it creates the daily index, awesome! Thank you so much, I didn't even think to check on that parameter, sorry about that.

6

Unfortunately that didn't work :frowning: The new index was created but I still don't have the raw fields.
I'm stuck on this one and can't move forward with my dashboards until I have the raw fields.

Any ideas?

elk_raw_fields_issue2.PNG888×705 29.3 KB

7

Check the installed mapping template(s). Do they contain what you expect (i.e. list .raw subfields for string fields)? If the index template looks correct, what happens if you create a new index (e.g. on a day some time in the future), does that index get the correct mappings?

8

That's a good suggestion, I'll check on both of those. I do want to say that they showed up after I restarted elasticsearch (I only restarted logstash) but only for certain fields. Totally weird.

It should create raw entries for any custom fields I have too right? That's how it was setup before and all my stuff uses the raw fields when the data contains dashes.

9

There should be .raw subfields for string fields that don't have an explicit configuration to not have a .raw subfield.

10

I found the problem, I took some mutates out of my logstash config because I didn't think I needed them. I deleted all the indices and added the mutate statements back in. Everything is working fine now, thanks!

11

#convert our datatypes so we can report on them
mutate {
convert => { "event_id" => "string" }
}
mutate {
convert => { "event" => "string" }
}
mutate {
convert => { "logsource" => "string" }
}
mutate {
convert => { "program" => "string" }
}