Using elasticsearch to store audit log information

Elastic Stack Elasticsearch
1

Hello,
I have a use-case wherein I need to capture audit trail style information
for a large user population (~1 mil) for all their actions within multiple
applications and across web and API layers.

Questions:

  1. From my reading, elasticsearch can be used for this specific use-case.
    (Looks like github.com is possibly using them in this manner -
    http://exploringelasticsearch.com/book/elasticsearch-at-scale-interviews/interview-with-the-github-elasticsearch-team.html)

  2. Will I need to store the audit trail in a separate reliable database in
    addition to elasticsearch or is only using elasticsearch sufficient.

Any tips gratefully appreciated.

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aff233a9-1f5c-44be-bac2-57da770f29ea%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

On Monday, March 3, 2014 4:34:26 PM UTC-6, npatil wrote:

Hello,
I have a use-case wherein I need to capture audit trail style information
for a large user population (~1 mil) for all their actions within multiple
applications and across web and API layers.

Questions:

  1. From my reading, elasticsearch can be used for this specific use-case.
    (Looks like github.com is possibly using them in this manner -
    http://exploringelasticsearch.com/book/elasticsearch-at-scale-interviews/interview-with-the-github-elasticsearch-team.html
    )

Hi, glad you like my book! Yes, it is indeed used for this case by a number
of people. I myself am planning on using it with a healthcare client.

  1. Will I need to store the audit trail in a separate reliable database in
    addition to elasticsearch or is only using elasticsearch sufficient.

That is a determination you'll need to make yourself. Elasticsearch is
quite reliable and can be backed up however. This is much easier now with
the snapshot / restore API. My $0.02 is that it will work fine for this
case.

Additionally, if you need to move log data to cold storage for cost savings
at some point, consider time-period based index naming (e.g. logs-2014-02),
and periodically closing old indexes. You can then move these index files
to some sort of cold storage if need be.

Let me know if I can help answer your questions in any other way!

Any tips gratefully appreciated.

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6d320301-7392-4743-b02c-d5f6b9cbcc2a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.