# Keep only last record of each type

**URL:** <https://discuss.elastic.co/t/keep-only-last-record-of-each-type/54866>\
**Category:** Logstash\
**Created:** [July 6, 2016, 7:17pm UTC](https://discuss.elastic.co/t/keep-only-last-record-of-each-type/54866 "2016-07-06T19:17:29Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![yodog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yodog/32/4822_2.png) [@yodog](https://discuss.elastic.co/u/yodog)\
**Post date:** [July 6, 2016, 7:17pm UTC](https://discuss.elastic.co/t/keep-only-last-record-of-each-type/54866/1 "2016-07-06T19:17:30Z")

</div>

Hello all.

I have to keep 2 types of indices on elasticsearch:

1- full, regular index holding all records sent to it (`logstash-YYYY.MM.DD`)  
2- small, fast index holding only the last log message of each kind (`lastaction-YYYY.MM`)

I need some help setting up the number 2 above.

Currently i'm using the `fingerprint` filter to generate a hash (SHA1) for each  
`message` field, and using this hash as the `document_id` for the `elasticsearch` output.

```
filter {
    if ("cloned" in [tags]) {
        uuid {
            add_tag => ["lastlogin"]
            overwrite => true
            target => "@uuid"
        }
        fingerprint {
            key => "lastlogin"
            method => "SHA1"
        }
    }
}

output {
    if ("lastlogin" in [tags]) {
        elasticsearch {
            document_id => "%{fingerprint}"
            index => "lastaction-%{+YYYY.MM}"
            sniffing => true
            template_overwrite => true
        }
    }
}

```

How would you guys do it?

I'm asking because despite not seeing any duplicated records (i think this part is ok)  
the index is bigger in size when compared to de full/regular one.  
And it should not.

Also, i expected it would only update the timestamp if a newer record was inserted,  
but in fact the `@timestamp` field is always updated, even if it is a date in the past.

Any ideas?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 8, 2016, 5:50am UTC](https://discuss.elastic.co/t/keep-only-last-record-of-each-type/54866/2 "2016-07-08T05:50:54Z")

</div>

> How would you guys do it?

That's what I'd do.

> I'm asking because despite not seeing any duplicated records (i think this part is ok)  
> the index is bigger in size when compared to de full/regular one.  
> And it should not.

It could be larger than expected unless it's optimized to expunge deleted documents. I don't remember to which extent this takes place automatically; check the ES documentation. Keep in mind that ES treats updates like a delete followed by a new document.

> Also, i expected it would only update the timestamp if a newer record was inserted,  
> but in fact the `@timestamp` field is always updated, even if it is a date in the past.

I'm afraid I don't understand this part.

---

<div class="post-metadata">

**Author:** ![yodog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yodog/32/4822_2.png) [@yodog](https://discuss.elastic.co/u/yodog)\
**Post date:** [July 8, 2016, 11:31am UTC](https://discuss.elastic.co/t/keep-only-last-record-of-each-type/54866/3 "2016-07-08T11:31:19Z")

</div>

> [@magnusbaeck](#):
>
> I'm afraid I don't understand this part.

sometimes i have to feed old logs to elasticsearch.

lets say that i have the following line indexed:

```
@timestamp July 8th 2016, 08:00:20.002
message open: user edgar-allan^poe@bookwriters.com opened INBOX/Trash

```

and later i backfill a file from july 6th.

```
@timestamp July 6th 2016, 01:00:10.001
message open: user edgar-allan^poe@bookwriters.com opened INBOX/Trash

```

i would like to maintain the most recent record based on `@timestamp`, but logstash just replaces the existing record with the last one.

how can i make it evaluate the `@timestamp` field, updating the record if more recent, but discarding if older?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 8, 2016, 11:39am UTC](https://discuss.elastic.co/t/keep-only-last-record-of-each-type/54866/4 "2016-07-08T11:39:50Z")

</div>

Logstash will update the document as a whole so it seems very unlikely that it wouldn't update `@timestamp` too. Have you double-checked that the `@timestamp` field really has the correct contents when you're backfilling data?

---

<div class="post-metadata">

**Author:** ![yodog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yodog/32/4822_2.png) [@yodog](https://discuss.elastic.co/u/yodog)\
**Post date:** [July 8, 2016, 11:48am UTC](https://discuss.elastic.co/t/keep-only-last-record-of-each-type/54866/5 "2016-07-08T11:48:45Z")

</div>

> [@magnusbaeck](#):
>
> Have you double-checked that the @timestamp field really has the correct contents when you're backfilling data?

yes, the content is correct.

i am trying to mimic a SQL trigger, that just updates the @timestamp field.  
(it should never go back in time)

```
CREATE TRIGGER dbo.trgAfterUpdate ON dbo.TblLastAction
AFTER UPDATE 
AS
  UPDATE dbo.TblLastAction
  SET last_changed = GETDATE()
  FROM Inserted i

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:48am UTC](https://discuss.elastic.co/t/keep-only-last-record-of-each-type/54866/6 "2017-07-06T04:48:53Z")

</div>