How to disable loading template for Elasticsearch 2.x?

Elastic Stack Beats
1

Hi everyone!

@Update:
Head to first comment to see my current problem

I try to load JSON formatted logs from log file into elasticsearch. Everything works OK, but I have those json.message, json.@timestamp, json.whatever keys stored in elasticsearch. From what I've read the parameter:

json.keys_under_root: true

Handles this problem well, so I'd see normal fields. But when I add it to the yml file and restart filebeat service, filebeat can't ship logs to elasticsearch anymore. My config:

filebeat:
  prospectors:
    -
      paths:
        - /tmp/logs/logs.log
      json:
        message_key: message
        keys_under_root: true
        overwrite_keys: true
output:
  elasticsearch:
    hosts: ["localhost:9200"]
    index: "logstash-%{+yyyy.MM.dd}"

Without keys_under_root everything works properly. Elasticsearch and Kibana are on docker containers and in version 5.2.0. Filebeat is 5.2.2.

Another thing I'd like to know is:

Can I get rid of all those Beats.Name, Beats.Hostname, Type fields? I can see, they are defined in filebeat.template.json and I'd like to get rid of them, but when I delete them and restart service, filebeat stops to ship logs to ES too.

Could You please help me with those 2 matters or point me to right direction?

Edit:

To clarify, here is the example log:

logEx.png1318×784 76.2 KB

What I want to achieve is to get rid of "json" prefix on custom fields and, if possible, I'd like to remove offset, source, type, input_type and Beats.* fields as I don't need them.

Greetings, Wojtek

2

I figured out the keys_under_root parameters. Now I have problem with elasticsearch output. These are the logs from filebeat:

2017/03/10 10:30:08.451074 beat.go:267: INFO Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]
2017/03/10 10:30:08.451099 beat.go:177: INFO Setup Beat: filebeat; Version: 5.2.2
2017/03/10 10:30:08.451108 processor.go:43: DBG  Processors: 
2017/03/10 10:30:08.451114 beat.go:183: DBG  Initializing output plugins
2017/03/10 10:30:08.451171 logp.go:219: INFO Metrics logging every 30s
2017/03/10 10:30:08.451219 output.go:167: INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json
2017/03/10 10:30:08.451334 output.go:178: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /usr/share/filebeat/bin/filebeat.template-es2x.json
2017/03/10 10:30:08.451388 outputs.go:100: ERR failed to initialize elasticsearch plugin as output: Error loading template /usr/share/filebeat/bin/filebeat.template-es2x.json: open /usr/share/filebeat/bin/filebeat.template-es2x.json: no such file or directory
2017/03/10 10:30:08.451401 beat.go:288: CRIT Exiting: error initializing publisher: Error loading template /usr/share/filebeat/bin/filebeat.template-es2x.json: open /usr/share/filebeat/bin/filebeat.template-es2x.json: no such file or directory
Exiting: error initializing publisher: Error loading template /usr/share/filebeat/bin/filebeat.template-es2x.json: open /usr/share/filebeat/bin/filebeat.template-es2x.json: no such file or directory

The logs are very verbose, I really don't have this file there, but I don't want to use elasticsearch 2x, I just want 5.2. Is there a parameter to specify this in output section in filebeat.yml? My current config goes like this:

filebeat:
  prospectors:
    -
      paths:
        - /tmp/logs/logs.log
      json:
        message_key: message
        keys_under_root: true
        overwrite_keys: true

output:
  elasticsearch:
    hosts: ["localhost:9200"]
    #index: "logstash-%{+yyyy.MM.dd}"
    template:
      enabled: true
      #name: "logstash"
      path: "/etc/filebeat/filebeat.template.json"
      overwrite: true

Cheers!

3

You can add versions.2x.enabled: false under the template section, see this doc.

The reason we check for the file even if you use ES 5.x is that at startup time we don't yet have a connection to ES, so we don't know it's version. Delaying the error would make it harder to notice (by then, the process is deamonized).

1 Like
4

Thank You very much for the response, that was exactly what I was looking for. Hovewer it seems that I try to import my custom template in a wrong way. I edited the filebeat.template.json file and still I get the default messages template. Is there some proper way to input the custom template into elasticsearch? I've used

curl -XPUT 'http://localhost:9200/_template/logstash' -d@/etc/filebeat/filebeat.template.json

But without luck.

5

What did you modify in the template? The index pattern? The curl command looks good, but should not be needed, Filebeat itself can also do it. Also, keep in mind that the new template only applies to new indices, not to the existing ones.

6

I (don't know if it's correct) removed the beats.* fields, source, type, offset etc fields that filebeat adds to message. I just need my custom fields, thus I wanted to get rid of those filebeat fields somehow.

@Edit, hey, I'm doing something stupid here - I'm editing the filebeat.template.json in /etc/filebeat directory and now I have the default template there. I guess it is because I set overwrite to true and thus I do something ridiculous :smiley:

What I'd like to achieve is to have below template pattern sent to elasticsearch:

{
  "mappings": {
    "_default_": {
      "_all": {
        "norms": false
      },
      "_meta": {
        "version": "5.2.2"
      },
      "dynamic_templates": [{
        "strings_as_keyword": {
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "norms": false,
          "type": "text"
        },
        "meta": {
          "properties": {
            "cloud": {
              "properties": {
                "availability_zone": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "instance_id": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "machine_type": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "project_id": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "provider": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "region": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            }
          }
        }
      }
    }
  },
  "order": 0,
  "settings": {
    "index.mapping.total_fields.limit": 10000,
    "index.refresh_interval": "5s"
  },
  "template": "logstash-*"
}
7

I (don't know if it's correct) removed the beats.* fields, source, type, offset etc fields that filebeat adds to message. I just need my custom fields, thus I wanted to get rid of those filebeat fields somehow.

That's not correct, removing them from the template will not going to remove it from the data, just let ES choose the types for them.

See the drop_fields and include_fields processor for filtering out fields: doc link

1 Like
8

Is ithis preprocessing costly operation? I mean for CPU? I always prefer rather to send logs prepared than to operate on them and parse somehow. It's because there will be like 9,5 M of logs per hour, so, ouch...

9

Just dropping fields (without any actual parsing) is pretty cheap, and probably saves you CPU time on the JSON encoding, so you might get a net reduction in CPU. I'd say just try it. If not, you can also easily drop fields in Logstash or Elasticsearch Ingest Node.

1 Like
10

@tudor, You're life-saver here, I managed to have the output I wanted with only my fields, metadata and timestamp thanks to You. It's a shame that I can't remove "type" field as specified in documentation, but it's not that much. Thanks a ton - I can now omit Logstash and send directly to elasticsearch using less CPU, memory etc with pure filebeat :slight_smile:

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.