Hello, I'm new to logstash, and while experimenting with it I faced some problems for which I need some help to solve them.
I'm reading the data I want to put into ES, from MySQL db. I have created a template (with template_name, settings and mapping). When starting the logstash.service I get the following log messages:
Using mapping template from {:path=>"/etc/logstash/conf.d/my_mapping.json"}
Attempting to install template {...}
Installing elasticsearch template to _template/logstash
and I believe everything is ok with the installing, but after the data is being inserted into the index, the mapping isn't the same as the one provided in the template. The index has the default mapping.
I've also tried first to create the index and put the mapping into it, but again, after I insert the data using logstash, it adds the default's mapping fields to the previously put mapping.
Here's the template file:
{
"template": "my_mapping",
"mappings": {
"properties": {
"id": {
"type": "keyword"
},
"uuid": {
"type": "keyword"
},
"general_info": {
"properties": {
"first_name": {
"type": "text"
},
"last_name": {
"type": "text"
},
"gender": {
"type": "text"
},
"note": {
"type": "text"
}
}
},
"created_at": {
"type": "date"
},
"updated_at": {
"type": "date"
}
}
}
}
I tried to insert the data:
- with / without 'settings' in the template file
- template_overwrite = true / false in the config file
- manage_template = true / false in the config file
- remove the fields which are not specified in the mapping (such as @version and @timestamp)
but nothing seems to work.
Also, I tried to 'create' the structure of the document in the filter part of the config file, but that is not the best solution, because I can have a lot of fields and a complex structure. And, there's no way to specify the type of the field to be nested (needed for querying later).
What could be the problem?
EDIT:
After I put 'index patterns' field in the mapping, the mapping was inserted correctly, but the data was again inserted with the default mapping.
Here's how it looks after putting 'index_patterns' field:
{
"properties": {
"id": {
"type": "keyword"
},
"uuid": {
"type": "keyword"
},
"general_info": {
"properties": {
"first_name": {
"type": "text"
},
"last_name": {
"type": "text"
},
"gender": {
"type": "text"
},
"note": {
"type": "text"
}
}
},
"created_at": {
"type": "date"
},
"updated_at": {
"type": "date"
},
"first_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"last_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"gender": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"note": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
As you can see, first_name / last_name / gender / note instead being in the general_info, they are separate fields. And, every document I inserted doesn't contain any nested fields, but it should!