Logstash Mapping

Elastic Stack Logstash
1

Hello, I'm new to logstash, and while experimenting with it I faced some problems for which I need some help to solve them.

I'm reading the data I want to put into ES, from MySQL db. I have created a template (with template_name, settings and mapping). When starting the logstash.service I get the following log messages:

Using mapping template from {:path=>"/etc/logstash/conf.d/my_mapping.json"}
Attempting to install template {...}
Installing elasticsearch template to _template/logstash

and I believe everything is ok with the installing, but after the data is being inserted into the index, the mapping isn't the same as the one provided in the template. The index has the default mapping.

I've also tried first to create the index and put the mapping into it, but again, after I insert the data using logstash, it adds the default's mapping fields to the previously put mapping.

Here's the template file:

{
	"template": "my_mapping",
	"mappings": {
		"properties": {
			"id": {
				"type": "keyword"
			},
			"uuid": {
				"type": "keyword"
			},
			"general_info": {
				"properties": {
					"first_name": {
						"type": "text"
					},
					"last_name": {
						"type": "text"
					},
					"gender": {
						"type": "text"
					},
					"note": {
						"type": "text"
					}
				}
			},
			"created_at": {
				"type": "date"
			},
			"updated_at": {
				"type": "date"
			},
			"last_contacted": {
				"type": "date"
			}
		}
	}
}

I tried to insert the data:

  • with / without 'settings' in the template file
  • template_overwrite = true / false in the config file
  • manage_template = true / false in the config file
  • remove the fields which are not specified in the mapping (such as @version and @timestamp)

but nothing seems to work.

Also, I tried to 'create' the structure of the document in the filter part of the config file, but that is not the best solution, because I can have a lot of fields and a complex structure. And, there's no way to specify the type of the field to be nested (needed for querying later).

What could be the problem?

2

This is really an elasticsearch question rather than a logstash question, but I think you need a "index_patterns" entry to tell ES which indexes the template should apply to.

3

Yes, adding the 'index_patterns' entry really helped! But, again, the data is inserted in the default mapping's format, and the default mapping is added to my mapping.

Here's how it looks now:

{
	"properties": {
		"id": {
			"type": "keyword"
		},
		"uuid": {
			"type": "keyword"
		},
		"general_info": {
			"properties": {
				"first_name": {
					"type": "text"
				},
				"last_name": {
					"type": "text"
				},
				"gender": {
					"type": "text"
				},
				"note": {
					"type": "text"
				}
			}
		},
		"created_at": {
			"type": "date"
		},
		"updated_at": {
			"type": "date"
		},
		"last_contacted": {
			"type": "date"
		},
		"first_name": {
			"type": "text",
			"fields": {
				"keyword": {
					"type": "keyword",
					"ignore_above": 256
				}
			}
		},
		"last_name": {
			"type": "text",
			"fields": {
				"keyword": {
					"type": "keyword",
					"ignore_above": 256
				}
			}
		},
		"gender": {
			"type": "text",
			"fields": {
				"keyword": {
					"type": "keyword",
					"ignore_above": 256
				}
			}
		},
		"note": {
			"type": "text",
			"fields": {
				"keyword": {
					"type": "keyword",
					"ignore_above": 256
				}
			}
		}
	}
}

As you can see, first_name / last_name / gender instead being in the general_info, they are separate fields. And, every document I inserted doesn't contain any nested fields, but it should!

4

Again, this is really an elasticsearch question and you might get a better answer in that forum.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.