How logs stored in Logstash/Elastisearch

Elastic Stack Elasticsearch
1

Hello All,

Can anyone help me on this.

  1. How data stored in logstash/elasticsearch?

  2. Where is the raw data file(path)?

  3. Is it encrypted?

  4. Can we take backup of those data and can remove and replace easily?

Regards,
Sandip Bankewar

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2
  1. It's indexed within Elasticsearch as a json document, one log entry in
    the Logstash is a document
  2. The default is /var/lib/elasticsearch/data
  3. No
  4. You can backup using the snapshot API. What do you mean by remove and
    replace though?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 16 July 2014 21:18, Sandip Bankewar sbankewar@gmail.com wrote:

Hello All,

Can anyone help me on this.

  1. How data stored in logstash/elasticsearch?

  2. Where is the raw data file(path)?

  3. Is it encrypted?

  4. Can we take backup of those data and can remove and replace easily?

Regards,
Sandip Bankewar

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YQYC9C2ZUqLigmJC8148pj_XBep%2B3GwZDWZN7qLy27qg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hello Mark,

Thanks for your response.

  1. one log entry in the Logstash is a document what do you mean by that?

  2. I mean if I have removed the raw data file as a backup purpose and then
    after few days I want to copy that again.

  3. Data stored is in Fat file right???

  4. I have the directory containing this format for data stored as *logstash-year-month-date
    ->> 0 1 2 3 4 _state *

I dont understand which file raw or fat file data stores???

Could you please help me on this?

Regards,
Sandip Bankewar

On Wednesday, 16 July 2014 17:07:26 UTC+5:30, Mark Walkom wrote:

  1. It's indexed within Elasticsearch as a json document, one log entry in
    the Logstash is a document
  2. The default is /var/lib/elasticsearch/data
  3. No
  4. You can backup using the snapshot API. What do you mean by remove and
    replace though?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 16 July 2014 21:18, Sandip Bankewar <sban...@gmail.com <javascript:>>
wrote:

Hello All,

Can anyone help me on this.

  1. How data stored in logstash/elasticsearch?

  2. Where is the raw data file(path)?

  3. Is it encrypted?

  4. Can we take backup of those data and can remove and replace easily?

Regards,
Sandip Bankewar

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/837968b9-94ed-41e5-97a7-a8a945e89187%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Hi,

On Wednesday, July 16, 2014 8:04:09 AM UTC-4, Sandip Bankewar wrote:

Hello Mark,

Thanks for your response.

  1. one log entry in the Logstash is a document what do you mean by that?

Look at Lucene. ES uses Lucene. Lucene has the notion of a "document".
In case of Logstash+ES, a log event is indexed as one Lucene document.

  1. I mean if I have removed the raw data file as a backup purpose and then

after few days I want to copy that again.

Not sure what you mean.

  1. Data stored is in Fat file right???

Not sure what you mean. Nothing is "Fat".

  1. I have the directory containing this format for data stored as *logstash-year-month-date

->> 0 1 2 3 4 _state *

I dont understand which file raw or fat file data stores???

ES indexes documents (logs in your case) using Lucene. If you can write a
bit of Java, write a simple Lucene-based indexer, that may be the simplest
way to understand what's written to disk. But if you are struggling with
Logstash+ES, you could also simply ship your logs to something like Logsene
http://sematext.com/logsene/ and not worry about dealing with indexing/ES
yourself.

Otis

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

Could you please help me on this?

Regards,
Sandip Bankewar

On Wednesday, 16 July 2014 17:07:26 UTC+5:30, Mark Walkom wrote:

  1. It's indexed within Elasticsearch as a json document, one log entry in
    the Logstash is a document
  2. The default is /var/lib/elasticsearch/data
  3. No
  4. You can backup using the snapshot API. What do you mean by remove and
    replace though?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 16 July 2014 21:18, Sandip Bankewar sban...@gmail.com wrote:

Hello All,

Can anyone help me on this.

  1. How data stored in logstash/elasticsearch?

  2. Where is the raw data file(path)?

  3. Is it encrypted?

  4. Can we take backup of those data and can remove and replace easily?

Regards,
Sandip Bankewar

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c04ad2d0-c0f4-4e7d-9d5b-8a02c79ad884%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.