# Missing core fields in index
**URL:** https://discuss.elastic.co/t/missing-core-fields-in-index/61946
**Category:** Beats
**Tags:** filebeat
**Created:** [September 30, 2016, 4:37pm UTC](https://discuss.elastic.co/t/missing-core-fields-in-index/61946 "2016-09-30T16:37:22Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@ejether](https://discuss.elastic.co/u/ejether)
#### Post date: [September 30, 2016, 4:37pm UTC](https://discuss.elastic.co/t/missing-core-fields-in-index/61946/1 "2016-09-30T16:37:22Z")
I'm having an odd issue where some of my log events getting shipped are missing the normal fields like beats.hostname, source etc. I'm not sure where to start to troubleshoot.
I have a mix of Cento6 and Centos7 all running filebeat 1.3.1 shipping to (currently) a single server running Logstash, Elasticsearch and Kibana in docker containers.
Any ideas on where to start would be much appreciated.
my filebeat conf:
> filebeat:
> prospectors:
> - paths:
> - /var/log/_/_\_perf.log
> encoding: plain
> fields\_under\_root: false
> input\_type: log
> document\_type: perf
> scan\_frequency: 10s
> harvester\_buffer\_size: 16384
> tail\_files: false
> force\_close\_files: false
> backoff: 1s
> max\_backoff: 10s
> backoff\_factor: 2
> partial\_line\_waiting: 5s
> max\_bytes: 10485760
and logstash config (erb template):
```auto
input {
beats {
port => <%= @beats_port %>
codec => multiline {
pattern => '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate => "true"
what => "previous"
}
}
}
filter {
ruby {
code => "
fields = event['message'].scan(/\S*=\S*/)
for field in fields
if field.include? '='
field = field.split('=')
if !field[0].nil? && !field[1].nil?
field[0] = field[0].gsub('.','_')
if field[1].delete('ms').to_i.to_s == field[1].delete('ms') && field[0] != 'Event'
event[field[0]] = field[1].delete('ms').to_s
else
event[field[0]] = field[1].to_s.delete(',')
end
end
end
end
"
}
#grok { match => {"message" => "%{TIMESTAMP_ISO8601:timestamp}"}}
#date { match => {"[@metadata][timestamp]" => "yyyy-MM-dd HH:mm:ss.SSS"}}
mutate { convert => {"FasaID" => "string" "ItemsProcessed" => "integer" "Total" => "integer" "count" => "integer"}}
}
output {
elasticsearch {
hosts => <%= @elasticsearch_hosts %>
#manage_template => false
#index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"
#document_type => \"%{[@metadata][type]}\"
}
#stdout { codec => rubydebug { metadata => "true" }}
}
```
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [September 30, 2016, 10:05pm UTC](https://discuss.elastic.co/t/missing-core-fields-in-index/61946/2 "2016-09-30T22:05:02Z")
I recommend moving the [multiline processing](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html) to Filebeat so that you don't need the multiline codec in Logstash.
---
### Author:  [@ejether](https://discuss.elastic.co/u/ejether)
#### Post date: [October 3, 2016, 7:48pm UTC](https://discuss.elastic.co/t/missing-core-fields-in-index/61946/3 "2016-10-03T19:48:20Z")
Do you think the multiline codec is causing this issue?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [October 4, 2016, 8:08am UTC](https://discuss.elastic.co/t/missing-core-fields-in-index/61946/4 "2016-10-04T08:08:23Z")
That is kind of the assumption as filebeat itself sends all events with this meta data. And as on the Logstash side some transformation happens, I would expect that it is removed there.
---
### Author:  [@ejether](https://discuss.elastic.co/u/ejether)
#### Post date: [October 4, 2016, 5:53pm UTC](https://discuss.elastic.co/t/missing-core-fields-in-index/61946/5 "2016-10-04T17:53:12Z")
It seems to have helped. I' not getting any more documents stacking up missing the source field. What was interesting is that it wasn't all documents or all documents from a particular prospector. It was seemingly random.
Do best practices dictate doing most of the log processing in filebeat instead of logstash?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [October 5, 2016, 6:19pm UTC](https://discuss.elastic.co/t/missing-core-fields-in-index/61946/6 "2016-10-05T18:19:05Z")
TBH I'm still confused about the random part and can't really explain it. Do you have multiple LS instances?
About the multiline and best practices: As usual, it depends.
- If you want to use as few resources as possible on the edge nodes, processing should be done in LS or ES
- If you load balance between multiple LS instances, it can happen the events arrive at different nodes and can't be properly combined
- LS can to much more then only multiline, so if you do additional processing, it could be simpler to do everything in one place.
It really depends on your use case.
---
### Author:  [@ejether](https://discuss.elastic.co/u/ejether)
#### Post date: [October 5, 2016, 7:39pm UTC](https://discuss.elastic.co/t/missing-core-fields-in-index/61946/7 "2016-10-05T19:39:28Z")
We currently have single LS instance.
As you say, I wanted to keep the config centralized, so I was putting all the processing in the LS config.
Anyway, Thanks for the tip.
Its being going strong without dropping metadata for a couple of days.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 21, 2016, 4:37pm UTC](https://discuss.elastic.co/t/missing-core-fields-in-index/61946/8 "2016-10-21T16:37:29Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.