Filebeat properties not indexed, topbeat properties all indexed

Hi all,

I am trying to figure out why all the properties generated by topbeat are indexed by default (I can create visualizations on them using kibana), while those from filebeat are not indexed. I am comparing the two templates shipped with those beats and it does not make any sense to me:

Default topbeat.template.json:

Default filebeat.template.json:

Both templates are more or less the same, except for the properties part. But from what I can see in kibana, all properties are indexed for topbeat, not only those mentioned in the templates. Is this because they are mentioned as top-level properties and hence all nested/dotted properties (eg: mem.used, not mentioned in the template) will also get indexed, regardless which one of those childs you mention in the template? I am probably asking a stupid question, but I can't get my head around this.

Notice I am using logstash for all my beats. Topbeats are forwarded without any filtering. The filebeats go through a filter to tokenize the original filebeats (nginx access and error logs for example). Might this be the differentiator that might cause them not to be indexed? From what I understood, indexation is driven by the templates, hence my confusion above...

My logstash configuration:

Screenshot of how filebeat results show up in kibana (I can't use the properties to generate visualisations):

Screenshot of how topbeat results show up in kibana (I can use all the properties to generate visualisations, even those not mentioned in topbeat.template.json):

I think I will have to create a custom template to get all fields indexed in the filebeat results, but I'd like to understand why they do get indexed for topbeat...

versions used: topbeat v1.2.3, filebeat 1.2.3, logstash 2.3.3-1, elasticsearch 2.3.3

Found the answer myself in the meanwhile:

To reload the fields, choose Logstash index and press the orange refresh button on the top right corner. The Apache log fields (path, request, agent etc.) should be displayed as indexed in the field list.


@bartvanbos Glad you found the answer and thanks for pointing others in the right direction.