Can a particular record in index be updated by logstash conf file?

Elastic Stack Logstash
1

We have a scenario like this:

Server Error Count_of_Error
D1 ER1 15
D1 ER2 15

Questions:

  1. Can this count be updated in Index in logstash configuration file?
  2. Will this maintain atomicity in case this index is getting updated by multiple filebeats
  3. How index is stored inside ElasticSerach. Is this stored in files?

Presently we are getting this count from kibana visualisation. We want to maintain the same in conf file.

2

Can this count be updated in Index in logstash configuration file?

That's probably going to be hard. Where do these numbers come from? How are they getting into Logstash?

How index is stored inside ElasticSerach. Is this stored in files?

Elasticsearch uses a file system for persistence, yes.

3

magnusblack

We need to create the count too in first place. The next time same error comes it should increment.

4

Wouldn't it be easier to just use ES for this kind of counting?

5

Yeah it would definitely be easier but we have a use case where we are going to purge data every 3-4 months. So we wanted to have a summary information of all the errors occured in the past which is count.
We will be purging everything else except this count visualization as it will have only one row per error type.
Can the same aggregation kibana does on UI can be performed on logstash level so that we don't need to store all records and we will be incrementing the same count whenever a same error occur ?