Could ES store one copy if content are same?

Elastic Stack Elasticsearch
1

Hi,
Does ES support store the logs one time if they have same content? And just
record the timestamp for every log?
In this way, it could improve the disk usage.
For instance, 2 lines of log are the same, ES just store the content of the
line once, and record the timestamp for both.
When searching, ES returns the content twice with different timestamp.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

You can encode your log content and generate an Id from it.
Use that Id when you push your doc to Elasticsearch.

When a second line with same content will appear, you will generate the same Id and so, you will update the document.

Make sense?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 8 juil. 2013 à 04:34, lijionly@gmail.com a écrit :

Hi,
Does ES support store the logs one time if they have same content? And just record the timestamp for every log?
In this way, it could improve the disk usage.
For instance, 2 lines of log are the same, ES just store the content of the line once, and record the timestamp for both.
When searching, ES returns the content twice with different timestamp.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thnak you for your kindly answer.
In this way, the date fields should be a list? Every time update the same
document should add the doc date to the date list in index doc? How to do
that ?
And when searching index by date range, could ES find out the matched date
in the date list?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

You can organize documents as you need. If you need to store latest timestamp, store it in a single field. Otherwise, yes you can use an array.

About matching date, I don't think you can highlight it.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 9 juil. 2013 à 05:22, lijionly@gmail.com a écrit :

Thnak you for your kindly answer.
In this way, the date fields should be a list? Every time update the same document should add the doc date to the date list in index doc? How to do that ?
And when searching index by date range, could ES find out the matched date in the date list?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Thanks,
My goal is to use ES general function and just keep one copy if the same
when storing the logs content.
And your suggestion is that we can't match date in array?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

I did not say that.
You can store an array of dates and search in it.

The best thing to do is to test it. It's really easy to start with elasticsearch and with some curl commands you can create a document with an array of dates and search for it using RangeFilter.

HTH

David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 juil. 2013 à 04:15, lijionly@gmail.com a écrit :

Thanks,
My goal is to use ES general function and just keep one copy if the same when storing the logs content.
And your suggestion is that we can't match date in array?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Thank you,
I have tried with date array, and the range filter can get the matched
date in the date array. It's very smart.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.