I need advise:
Everyday I create a log using logstash input-jdbc to run a SQL query on a readonly_database. In the log I have a long alpha-numeric caseID and I need a human-friendly id number instead. And this id number needs to be the same every time.
I was thinking there must be a way with a logstash input /filter plugin to lookup a list of known caseID on my ES, get a id from that, and when a new caseID is added, I get the last id+1
The log with caseID and attributes
{
caseID: "410a-8f03-317cae7710b676be2496-203b-401b",
attribute1: "foo"
},
{
caseID: "410a-8f03-317cae7710b676be2496-203b-401b",
attribute1: "bar"
},
{
caseID: "203b-401b-aaf7-fe7ddb86a6f876be-dfgd-654f",
attribute1: "foo"
},
{
caseID: "203b-401b-aaf7-fe7ddb86a6f876be-dfgd-654f",
attribute1: "bar"
},
The desired output, and id and caseID is now always linked.
{
id: "1",
caseID: "410a-8f03-317cae7710b676be2496-203b-401b",
attribute1: "foo"
},
{
id: "1",
caseID: "410a-8f03-317cae7710b676be2496-203b-401b",
attribute1: "bar"
},
{
id: "2",
caseID: "203b-401b-aaf7-fe7ddb86a6f876be-dfgd-654f",
attribute1: "foo"
},
{
id: "2",
caseID: "203b-401b-aaf7-fe7ddb86a6f876be-dfgd-654f",
attribute1: "bar"
},
How would you suggest that I attack this problem?