Hello Logstash Sorcerers,
I am running Logstash v7.4.0 and Elasticsearch v7.4.0 in a nicely-working pipeline. My chain of fruit stores are sending my sales information to Logstash; Logstash then pushes that data to Elasticsearch. Here’s a subsample of my Elasticsearch fruit_sales index, plus a few example data records:
product_code qnty
====================
100 2039
200 382
300 1028
400 128
This means: I sold 2,039 units of product 100, 382 units of product 200, and so on.
My setup works great, but the trouble is it’s a pain to remember what Product Code is assigned for which fruit. On a piece of paper in my office, I have a list:
product_code name
======================
100 apples
200 oranges
300 bananas
The list currently has several hundred items. It does not change much, but I might need to modify it from time to time. Unfortunately, there is no way to modify the data that my stores send into Logstash to include these names.
What I’d love is a solution where I can input my product_code-to-name list into Logstash. And then, when LS is processing the raw data it is receiving from my stores, LS adds a new string field into each data record:
product_code *name* qnty
===============================
100 “apples” 2039
200 “oranges” 382
300 “bananas” 1028
400 “other” 128
Now in each data record, I have both the numerical product code plus a human-readable string with the fruits’ name. This makes checking queries and reading Kibana Visualizations much, much easier. Also note that there is a default value (“other”) in case I have a product which appears in the sales data, but not on the name list.
Another consideration: I don’t think I’ll need to update the product_code-to-name list much, but I can’t have a solution where I would need to stop and restart the LS service whenever I make an edit to that list. Of course, if I have 10,000 data records with a Name field already populated, and then, say, I change Product_Code 100 from “apples” to “papayas,” there’s no need to traverse through all existing records to change that. But every new data record from that point out needs to have the new string assignment.
Is there a graceful, non-computational way to do this? In SQL, you could do a join between two tables, in C you could create a product_code-to-name hash table, etc.
I assume in my LS config file, I would set up a filter that would do something crude like this:
filter {
ruby {
# Bounce each data record to a ruby script which does
# product_code lookup and adds new field accordingly
path => "/home/me/addNameField.rb"
}
}
I don’t have much Ruby experience, but this doesn’t seem that hard.
But I’d like to ask the Forum: (A) is this the best solution? And (B), if I use the Ruby script, is there an example of another Ruby script which adds a new field into a record? I’d rather adapt something that works than reinvent the wheel from scratch.
Any advice on this will be wildly appreciated. Thank you!
FULL DISCLOSURE: I am also posting a variation of this question to the Elasticsearch forum, as this task might be easier on ES than LS.