Can Logstash Mutate Rename method use a file

Elastic Stack Logstash
1

Hi there,

I do not if it possible, but I couldn't find any information regarding the renaming of the fields based on the dictionary.

I would like to use Logstash to convert all the entry data to the 'standard' format for the further ElasticSearch storage.
Some entries can have the same logical fields, but the name of them can vary, e.g. one system can send a field 'Employee Name', the other 'EmpName' or 'EmployeeName'. So I would like to convert all these records to 'EmployeeName' field, that I have in my mapping in the ElasticSearch index.

I know there is a mutate rename field method. But it is not based on the dictionary. Is there a way to use the file like in the Translate method, as I can generate the file from the code?

Thank you!

2

Can you give an example of what you want to achieve?

3

I am sorry, if my question was not very clear!

One system can send the JSON in this format:

{
"_index": "main.entity",
"_type": "_doc",
"_id": "dc9f9286-d595-4db1-983a-0e1f36cd436c",
"_version": 1,
"_score": null,
"_source": {
"id": "dc9f9286-d595-4db1-983a-0e1f36cd436c",
"timestampUTC": "2018-12-03T20:39:51.391648Z",
"context": {
"companyNumber": 1000,
"employeeNumber": 1210
}
},
"fields": {
"timestampUTC": [
"2018-12-03T20:39:51.391Z"
]
},
"sort": [
1543869591391
]
}

the other is in this format

{
"_index": "main.entity",
"_type": "_doc",
"_id": "dc9f9286-d595-4db1-983a-0e1f36c7836c",
"_version": 1,
"_score": null,
"_source": {
"id": "dc9f9286-d595-4db1-983a-0e1f36cd436c",
"timestampUTC": "2018-12-03T20:39:51.391648Z",
"context": {
"companyNo": 1020,
"employeeNo": 1890
}
},
"fields": {
"timestampUTC": [
"2018-12-03T20:39:51.391Z"
]
},
"sort": [
1543869593391
]
}

I want to mutate any json according to the mapping via file , e.g.
"context.companyNo" -> "context.companyNumber"
"context.companyNum" -> "context.companyNumber"

The main reason why I want to have a separate file, because I have about 250 fields that I need to check. I could do it within the plugin but it is hard to maintain, if someone adds a new field, I need to update it. If I have it within a file, that I can generate a file each time when someone adds a new field.

I hope I explained what I want to achieve. Thank you for your patience and and suggestions would be appreciated.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.