Custom filebeat template for JSON log lines

Elastic Stack Beats
5

It's funny: I did try to export the template using filebeat export template > filebeat.template.json. I changed the appropriate field types then I imported it to a new index and loaded a copy of my JSON log data. The data was still being written as a "text" type instead of an "ip" type. I wasn't sure why.

In the end, I found a way to make it work. Here is my procedure in case someone else runs into this issue before there is any official documentation on the subject:

  1. For filebeat.yml, set up the JSON fields but do not put the JSON event under root:
filebeat.prospectors:

- type: log

  enabled: true

  paths:
    - /var/log/elk-dev/*
  json.add_error_key: true

{...}

setup.template.name: "elk-dev"
setup.template.pattern: "elk-dev-*"

output.elasticsearch:
  - index: "elk-dev"
  1. Create a standard JSON template. You can base it off of filebeat export template. Here's an example of what it would look like:
{  
   "index_patterns":[  
      "elk-dev*"
   ],
   "mappings":{  
      "doc":{  
         "properties":{  
            "@timestamp":{ ... },
            "beat":{ ... },
            # BELOW IS WHERE YOU WILL DEFINE YOUR JSON FIELDS AND TYPES
            "json":{ 
                 "properties":{
                       "source-ip":{
                             "type":"ip",
                             "keyword":{
                                   "type": "keyword",
                                   "ignore_above": 256
                             }
                         },
                       ...
                       ...etc...
                       ...
             },
            "error":{ ... },
            "prospector":{ ... }
         }
      }
   }
}

  1. Use the curl API to put this JSON template into a new index:
    curl -H "Content-Type: application/json" -PUT 'http://$ELASTICSEARCH_IP:9200/_template/elk-dev?pretty' -d@filebeat.template.json

  2. Make sure that your filebeat.yml is outputting to elasticsearch and that you've set the appropriate index (in this example, it's "elk-dev" as shown in the last line in step 1)

  3. You can now ship data to this new index by doing sudo service filebeat restart.

  4. In my scenario, I had a timestamp in my JSON data which is different from @timestamp which filebeat adds automatically. When creating my index, I chose to use my own timestamp instead of filebeat's. I'm not sure this is required in some way to make the types work.

For me, the methods and suggestions I've found online did not work or I could not get them to work.
I could not find a way to modify fields.yml in a way that preserved custom field types although this seems like the most direct way of doing this.
Until then, this is my janky fix.

1 Like