Filebeat strict to elasticsearch send only default fields

Elastic Stack Beats
1

HI all,
My setup has only a filebeat sending the logs to elasticsearch, I'm trying to use multiple templates with different indexes.
When filebeat starts, the indexes are created, but only with default fields (beat.name, beat.version, and stuffs like that), I'm not sure if the filebeat are sending data using my templates, but when I've purposely typed something wrong on my json, the prospector won't be able to start, so I guess my template is being read.

Here my filebeat.yml

filebeat.prospectors:
-   type: log
    json.keys_under_root: true
    json.message_key: event
    paths:
    - /var/log/*.log

-   type: log
    paths:
     - /hostfs/var/lib/docker/containers/*/*.log
    document_type: docker
    json.message_key: log

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: 'default'
  indices:
    - index: "backend"
      when.contains:
        message: "backend"
    - index: "frontend"
      when.contains:
        message: "frontend"
setup.template.name: "default"
setup.template.pattern: "default-*"
setup.template.name: "default"
setup.template.name: "backend"
setup.template.pattern: "backend-*"
setup.template.name: "frontend"
setup.template.pattern: "frontend-*"
setup.template.overwrite: true

here my backend.json

{
"mappings": {
	"_default_": {
		"_all": {
			"norms": false
		},
		"_meta": {
			"version": "5.2.1"
		},
		"dynamic_templates": [{
			"strings_as_keyword": {
				"mapping": {
					"ignore_above": 1024,
					"type": "keyword"
				},
				"match_mapping_type": "string"
			}
		}],
		"properties": {
			"@timestamp": {
				"type": "date"
			},
			"beat": {
				"properties": {
					"hostname": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"name": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"version": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"input_type": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"message": {
						"norms": false,
						"type": "text"
					},
					"instance_id": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"machine_type": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"project_id": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"provider": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"region": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"host": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"level": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"type": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"msg": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"app": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"environment": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"caller": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"error": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"request_uri": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"stack": {
						"ignore_above": 1024,
						"type": "keyword"
					},

					"offset": {
						"type": "long"
					},
					"source": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"tags": {
						"ignore_above": 1024,
						"type": "keyword"
					},
					"order": 0,
					"settings": {
						"index.mapping.total_fields.limit": 10000,
						"index.refresh_interval": "5s"
					},
					"template": "backend-*"
				}
			}
		}
	}
}
2 
setup.template.name: "default"
setup.template.pattern: "default-*"
setup.template.name: "default"
setup.template.name: "backend"
setup.template.pattern: "backend-*"
setup.template.name: "frontend"
setup.template.pattern: "frontend-*"
setup.template.overwrite: true

This doesn't work as intended. Config the config, The YAML parser will generate this:

setup.template.name: "frontend"
setup.template.pattern: "frontend-*"
setup.template.overwrite: true

When using multiple indices, you will have setup the template yourself.

3

Thanks @steffens!
But yes, that was what I did, I've created 3 different templates, but I don't think they are ben read.
How can I set something similar?
regards

4

Filebeat can not handle more then one template.

I guess you are using 5.x. See Elasticsearch Index Template docs.

Using curl you can use curl -XPUT -H 'Content-Type: application/json' http://<es_url>/_template/backend -d@backend_template.json to manually load the backend template. Do this for every template and disable template loading in filebeat. Btw. it's a good practice to version templates and index names. E.g. recent beats use the beat-version number for template files and index names.

5

Thanks for all help!

I'm gonna try, and regarding the loading, I thought by default the loading was disabled.. Am I wrong ? filebeat-template

# A template is used to set the mapping in Elasticsearch
_ # By default template loading is disabled and no template is loaded._
_ # These settings can be adjusted to load your own template or overwrite_

Last question, Can I set a specific template for each prospector?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.