Adding data to existing index for which dashboards are created

Elastic Stack Logstash
1

Hi All,

I am trying to add new data to an existing index in Elasticsearch by running the same config file in Logstash. I already have kibana dashboard created for that index.

My doubt is, does Logstah copy again the old data with new data? Does it affect the created dashboards?

Thanks!

2

If you process the same event through an elasticsearch output a second time then unless you are setting the document_id option you will get a second, duplicate, document in the index. It depends on what your input data looks like, but you may be able to generate a unique document_id using a fingerprint filter.

3

If Logstash is pointing to the same location for the new data, then it should ignore any files it has processed thanks to its sincedb.

And yes it will effect the dashboards, because those will be showing the additional data.

4

yes you can do it. You have to make sure you have same document_id.
everything is very much depend on uniq document_id.
When I first starting working on this I had to go through this. it is not explain properly anywhere.

Anyway what ever you do make sure you create your own document_id from your data so you can update.

on your output section you have to use
action=>"update" and it will update document if it exist

for example,
you already have three field from somewhere. job, machine_name,center in your index "job_data"

input {
read data -------- one of the field is job# which is uniq
and it also reads machine_name, center
}
output {
document_id => %{job}
}

now you said are going to read more field to add that in to record
you can read job, machine_name,center. and new field "status"

use
action => "update" on output section and it will update record and add new field for you.

it is always good to post some example that way whoever is trying to give you answer knows what you trying to achieve.

5

It's not though, you don't need to define your document ID.

6

yes you don't have to define. then it will generate automatic document_id

and now when you try to update how does it do it?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.