How to map a (filebeat) numeric field to display a customer name?

Elastic Stack Logstash
#1

My filebeat client is sending data records with a numeric field name of 'accountID' to my elasticsearch server. When I create a (Kibana) visualization with this data, my graphs show the numberic 'accountID' values on the X-axis as expected. My goal is to map these numeric accountID values to an actual customer name. (Without the name, an accountID lookup in the database has to be executed, inefficient at best...) :confused:
My failed solution:

  1. Inner Objects - Use the elasticsearch mechanism called Inner Objects by adding the following values to the 'filebeat-index-template.json
    {
    "accountID": [
    { "ACCOUNT_ID": "1674", "SHORT_NAME": "Hickory"},
    { "ACCOUNT_ID": "887", "SHORT_NAME": "Dickory"},
    { "ACCOUNT_ID": "862", "SHORT_NAME": "Doc"},
    ]
    }
    }

Any suggestions for this novice to display a customer name, rather than my vague account_id??

#2

If you send your records via Logstash it would be trivial to use a translate filter to do this mapping.

#3

Thanks Magnus! I will investigate this option.

#4

You can't do this with ES, it needs to be done before the data is sent to it.

#5

Magnus
Thanks to your input, I installed the logstash-filter-translate plugin on my ELK server.

Based on what I read, I needed to configure the dictionary. I assume that the config file for this was located in /root/logstash-filter-translate-master/spec/filters/translate_spec.rb

I added these lines:

describe "Account_id translation" do

let(:config) do
  {
    "field"       => "accountID",
    "destination" => "translation",
    "dictionary"  => [ "1674", "Hickory",
                       "887", "Dickory",
                       "862", "Doc",
                       "1804", "Misc" ],
                       "exact"       => true,
                       "regex"       => false
  }
end

let(:event) { LogStash::Event.new("status" => 200) }

it "return the exact translation" do
  subject.register
  subject.filter(event)
  expect(event["translation"]).to eq("OK")
end

end

I restarted logstash with
service logstash restart

I expected to see the translated accountID values in Kibana, but did not.

Any suggestions on what I may have missed?

#6

Um, that's the tests for the plugin's source code. Editing it will get you nowhere. Use the plugin's dictionary_path option to select which file contains the mappings you want to make.

#7