I am getting Json response from JIRA using Api containing issue details and loading those as logs in Elasticsearch. This activity we are doing continuously. So every time we need to provide an incremental data. In data we might have some records which already exist (Considering Jira ID e.g. ABC-1234 as primary key)and incremental data (like status changed from In-progress to done) should update the existing record instead of pushing new records with new data.
My current setup is json file is stored as log and pushed to Elastic search through Logstash. For creating visualization we are using Kibana. I need help in updating the existing record in Elastic Search.
What is your current pipeline in logstash and how does your document looks like?
I constantly do this. but I don't update just one field. I rewrite whole document.
you can do this two way.
1 rewrite whole document
for example status = submitted, to running to failed or done.
lets say you you have id=adc-1234
I scan document and on output section you just simple use
document_id => "%{Jira_ID}"
and elasticsearch will take care this.
2 just update field that you want.
if you just want to update status field. you will have to run another pipeline.
just get the jira_id and status on that record and on output section you do
document_id => "%{jita_ID}"
action => "update"
and what ever field you will have in that record will be updated.
Thanks a lot Sachin, We will try to implement it and com back to you in case I face any challenges.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.