# Monitor events

**URL:** <https://discuss.elastic.co/t/monitor-events/143512>\
**Category:** Kibana\
**Created:** [August 8, 2018, 12:22pm UTC](https://discuss.elastic.co/t/monitor-events/143512 "2018-08-08T12:22:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gabriela\_Diana\_Stoic](https://avatars.discourse-cdn.com/v4/letter/g/8797f3/32.png) [@Gabriela\_Diana\_Stoic](https://discuss.elastic.co/u/Gabriela_Diana_Stoic)\
**Post date:** [August 8, 2018, 12:22pm UTC](https://discuss.elastic.co/t/monitor-events/143512/1 "2018-08-08T12:22:07Z")

</div>

Hi,

I am using Logstash, Elasticsearch and Kibana to continously monitor events sent by a server via a TCP port.  
Every server event is described by: Name, ID, Subsystem where it occured, Occurence Time, Clearing Time and State.  
State can be: Uncleared or Cleared (in this case Clearing Time attribute is also populated):

{Name: Authentication Failure, ID: 87645, Subsystem: SR01, Occurence Time: 2018-08-08 14:00:00, Clearing Time: - , State: Uncleared}

After an interval of time every server event (hopefully) ends in Cleared state. In this case I want to keep only one entry of that event with the Cleared state.

Is it possible to do this?

Thanks!

---

<div class="post-metadata">

**Author:** ![Nathan\_Reese](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nathan_reese/32/84829_2.png) [@Nathan\_Reese](https://discuss.elastic.co/u/Nathan_Reese)\
**Post date:** [August 9, 2018, 1:46pm UTC](https://discuss.elastic.co/t/monitor-events/143512/2 "2018-08-09T13:46:54Z")

</div>

Yes.

When indexing your data via logstash, set the Elasticsearch [\_id](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-id-field.html) field to the `ID` field from the event. That way, future events that map to the same `ID` will not be written to a new Elasticsearch document but rather update the existing document. Besure to configure the [Elasticsearch output action](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-action) to allow for updates.

---

<div class="post-metadata">

**Author:** ![Gabriela\_Diana\_Stoic](https://avatars.discourse-cdn.com/v4/letter/g/8797f3/32.png) [@Gabriela\_Diana\_Stoic](https://discuss.elastic.co/u/Gabriela_Diana_Stoic)\
**Post date:** [August 16, 2018, 10:08am UTC](https://discuss.elastic.co/t/monitor-events/143512/3 "2018-08-16T10:08:43Z")

</div>

Hi,

Thank you for your answer! After implementing with action "update" I receive a document missing exception error like this.

- configuration:

input {  
file {  
type =\> "json"  
path =\> "/home/gabi/PycharmProjects/alarme\_logstash/alarm\_logfile.json"  
start\_position =\> "beginning"  
ignore\_older =\> 0  
}  
}

filter {  
json {  
source =\> "message"  
}  
}

output {  
stdout { codec =\> rubydebug }  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "logstash-%{+YYYY.MM.dd}"  
**document\_id =\> "%{ID}"**  
action =\> "update"

}  
}

- error

[WARN] 2018-08-14 17:58:56.170 [Ruby-0-Thread-7@[main]\>worker1: :1] elasticsearch - Could not index event to Elasticsearch. {:status=\>404, :action=\>["update", {:\_id=\>"%{ID}", :\_index=\>"logstash-2018.08.14", :\_type=\>"doc", :\_routing=\>nil, :\_retry\_on\_conflict=\>1}, #\<LogStash::Event:0xe7278f0\>], :response=\>{"update"=\>{"\_index"=\>"logstash-2018.08.14", "\_type"=\>"doc", "\_id"=\>"%{ID}", "status"=\>404, "error"=\>{"type"=\>"document\_missing\_exception", "reason"=\>"[doc][%{ID}]: document missing", "index\_uuid"=\>"3-\_Mf6gTR7ivVE46dMZ0ag", "shard"=\>"1", "index"=\>"logstash-2018.08.14"}}}}

My JSON event looks like this:

{  
"AlarmName" =\> "AlarmSlogan",  
"message" =\> "{"Occurtime": "2018-08-14 17:26:48", "Severity": "Minor", "NeType": "EQ3900", "State": "Unacknowledged Event", "AlarmName": "AlarmSlogan", "ID": "649390", "NeName": "EQ\_21", "Location": "Other details regarding the alarm"}",  
"@version" =\> "1",  
"Severity" =\> "Minor",  
"NeType" =\> "EQ3900",  
"State" =\> "Unacknowledged Event",  
"path" =\> "/home/gabi/PycharmProjects/alarme\_logstash/alarm\_logfile.json",  
"@timestamp" =\> 2018-08-14T14:58:54.009Z,  
"host" =\> "Gabi",  
"type" =\> "json",  
"Occurtime" =\> "2018-08-14 17:26:48",  
"NeName" =\> "EQ\_21",  
"Location" =\> "Other details regarding the alarm",  
**"ID" =\> "649390"**  
}

I don't know now if I should use doc\_as\_upsert to prevent trying to update a document that does not exist (for new events) or if there is an error in the way I am extracting ID field from the JSON event.

Thanks!

---

<div class="post-metadata">

**Author:** ![Nathan\_Reese](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nathan_reese/32/84829_2.png) [@Nathan\_Reese](https://discuss.elastic.co/u/Nathan_Reese)\
**Post date:** [August 16, 2018, 11:50am UTC](https://discuss.elastic.co/t/monitor-events/143512/4 "2018-08-16T11:50:31Z")

</div>

You must set the field `_id` to the value of the field `ID`. Elasticsearch tracks documents via `_id`.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 13, 2018, 11:50am UTC](https://discuss.elastic.co/t/monitor-events/143512/5 "2018-09-13T11:50:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.