Custom filebeat template for JSON log lines

Elastic Stack Beats
7

Thank you Vincent! This was exactly what I needed to get started sending structured JSON logs directly to elasticsearch with filebeat. One side effect of the above procedure is that fields will be referenced as json.field in Kibana (e.g. json.source-ip following the example in your answer).

I took this a little further and figured out how to get the fields to display natively. To do this set json.keys_under_root: true in filebeat.yml and then in your mapping do not have a json section but instead put the fields directly under the "properties" section where @timestamp and beat are shown. Your example mapping updated would look like this:

"mappings":{  
      "doc":{  
         "properties":{  
            "@timestamp":{ ... },
            "beat":{ ... },
            # NOW DEFINE YOUR JSON FIELDS AND TYPES ANYWHERE IN THE TOP PROPERTIES LIST
            "source-ip":{
               "type":"ip",
               "keyword":{
                  "type": "keyword",
                  "ignore_above": 256
               }
            },
            ...
            ...etc...
            ...
            },
            "error":{ ... },
            "prospector":{ ... }
         }
      }
   }
}

FWIW I also didn't get the curl API to put my index correctly however I just saved the template then went to the Kibana console (click Dev Tools) and did a PUT elkdev and then copied and pasted the whole template after a newline (the template included the original index_patterns setting in your answer and other index settings from the output of filebeat export template > filebeat.template.json). Not sure if this would have worked if I didn't delete and recreate the index due to what Steffen noted so you should research if you want to update in other ways (saw something about reindex while looking into this stuff).

Thanks again, hope this helps you or someone else!