Filebeat properties not indexed, topbeat properties all indexed

bartvanbos · July 6, 2016, 7:24am

Hi all,

I am trying to figure out why all the properties generated by topbeat are indexed by default (I can create visualizations on them using kibana), while those from filebeat are not indexed. I am comparing the two templates shipped with those beats and it does not make any sense to me:

Default topbeat.template.json:

gist.github.com

https://gist.github.com/boeboe/be740a759e87f77a8863135a86ac6b16

topbeat.template.json

  "topbeat" : {
    "order" : 0,
    "template" : "topbeat-*",
    "settings" : {
      "index" : {
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "_default_" : {

This file has been truncated. show original

Default filebeat.template.json:

gist.github.com

https://gist.github.com/boeboe/a0e6402700b09a9d5312cd0f94a8850c

filebeat.template.json

  "filebeat" : {
    "order" : 0,
    "template" : "filebeat-*",
    "settings" : {
      "index" : {
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "_default_" : {

This file has been truncated. show original

Both templates are more or less the same, except for the properties part. But from what I can see in kibana, all properties are indexed for topbeat, not only those mentioned in the templates. Is this because they are mentioned as top-level properties and hence all nested/dotted properties (eg: mem.used, not mentioned in the template) will also get indexed, regardless which one of those childs you mention in the template? I am probably asking a stupid question, but I can't get my head around this.

Notice I am using logstash for all my beats. Topbeats are forwarded without any filtering. The filebeats go through a filter to tokenize the original filebeats (nginx access and error logs for example). Might this be the differentiator that might cause them not to be indexed? From what I understood, indexation is driven by the templates, hence my confusion above...

My logstash configuration:

gist.github.com

https://gist.github.com/boeboe/3f7503b1246b8d963debe7208e3dd86f

logstash.conf

input {
  beats {
    port => 5044
  }
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]

This file has been truncated. show original

gist.github.com

https://gist.github.com/boeboe/d94ed8cbfddb9338397917f1bf418edd

nginx

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}
NGINXERROR1 (?<timestamp>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(?:, client: (?<client>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: %{QS:request})(?:, upstream: %{QS:upstream})?(?:, host: %{QS:host})?(?:, referrer: \"%{URI:referrer}\")
NGINXERROR2 (?<timestamp>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(?:, client: (?<clientip>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: %{QS:request}) ??(?:, host: %{QS:host})
NGINXERROR3 (?<timestamp>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(?:, client: (?<clientip>%{IP}|%{HOSTNAME}))(?:, server: %{HOSTPORT:server})

Screenshot of how filebeat results show up in kibana (I can't use the properties to generate visualisations):

Screenshot of how topbeat results show up in kibana (I can use all the properties to generate visualisations, even those not mentioned in topbeat.template.json):

I think I will have to create a custom template to get all fields indexed in the filebeat results, but I'd like to understand why they do get indexed for topbeat...

versions used: topbeat v1.2.3, filebeat 1.2.3, logstash 2.3.3-1, elasticsearch 2.3.3

bartvanbos · July 6, 2016, 11:55am

Found the answer myself in the meanwhile:
https://rafaelmt.net/en/2015/09/01/kibana-tutorial/#refresh-fields

To reload the fields, choose Logstash index and press the orange refresh button on the top right corner. The Apache log fields (path, request, agent etc.) should be displayed as indexed in the field list.

ruflin · July 11, 2016, 8:15am

@bartvanbos Glad you found the answer and thanks for pointing others in the right direction.