# How logs stored in Logstash/Elastisearch **URL:** \ **Category:** Elasticsearch\ **Created:** [July 16, 2014, 11:18am UTC](https://discuss.elastic.co/t/how-logs-stored-in-logstash-elastisearch/18697 "2014-07-16T11:18:46Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![sbankewar](https://avatars.discourse-cdn.com/v4/letter/s/858c86/32.png) [@sbankewar](https://discuss.elastic.co/u/sbankewar)\ **Post date:** [July 16, 2014, 11:18am UTC](https://discuss.elastic.co/t/how-logs-stored-in-logstash-elastisearch/18697/1 "2014-07-16T11:18:46Z")
Hello All, Can anyone help me on this. 1. How data stored in logstash/elasticsearch? 2. Where is the raw data file(path)? 3. Is it encrypted? 4. Can we take backup of those data and can remove and replace easily? Regards, Sandip Bankewar -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 16, 2014, 11:36am UTC](https://discuss.elastic.co/t/how-logs-stored-in-logstash-elastisearch/18697/2 "2014-07-16T11:36:55Z")
1. It's indexed within Elasticsearch as a json document, one log entry in the Logstash is a document 2. The default is /var/lib/elasticsearch/data 3. No 4. You can backup using the snapshot API. What do you mean by remove and replace though? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 16 July 2014 21:18, Sandip Bankewar [sbankewar@gmail.com](mailto:sbankewar@gmail.com) wrote: > Hello All, > > Can anyone help me on this. > > 1. How data stored in logstash/elasticsearch? > > 2. Where is the raw data file(path)? > > 3. Is it encrypted? > > 4. Can we take backup of those data and can remove and replace easily? > > Regards, > Sandip Bankewar > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624YQYC9C2ZUqLigmJC8148pj\_XBep%2B3GwZDWZN7qLy27qg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624YQYC9C2ZUqLigmJC8148pj_XBep%2B3GwZDWZN7qLy27qg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![sbankewar](https://avatars.discourse-cdn.com/v4/letter/s/858c86/32.png) [@sbankewar](https://discuss.elastic.co/u/sbankewar)\ **Post date:** [July 16, 2014, 12:04pm UTC](https://discuss.elastic.co/t/how-logs-stored-in-logstash-elastisearch/18697/3 "2014-07-16T12:04:09Z")
Hello Mark, Thanks for your response. 1. one log entry in the Logstash is a document what do you mean by that? 2. I mean if I have removed the raw data file as a backup purpose and then after few days I want to copy that again. 3. Data stored is in Fat file right??? 4. I have the directory containing this format for data stored as \*logstash-year-month-date -\>\> 0 1 2 3 4 \_state \* _I dont understand which file raw or fat file data stores???_ _Could you please help me on this?_ _Regards,_ _Sandip Bankewar_ On Wednesday, 16 July 2014 17:07:26 UTC+5:30, Mark Walkom wrote: > 1. It's indexed within Elasticsearch as a json document, one log entry in > the Logstash is a document > 2. The default is /var/lib/elasticsearch/data > 3. No > 4. You can backup using the snapshot API. What do you mean by remove and > replace though? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 16 July 2014 21:18, Sandip Bankewar \<[sban...@gmail.com](mailto:sban...@gmail.com) \\> > wrote: > > > Hello All, > > > > Can anyone help me on this. > > > > 1. How data stored in logstash/elasticsearch? > > > > 2. Where is the raw data file(path)? > > > > 3. Is it encrypted? > > > > 4. Can we take backup of those data and can remove and replace easily? > > > > Regards, > > Sandip Bankewar > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/837968b9-94ed-41e5-97a7-a8a945e89187%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/837968b9-94ed-41e5-97a7-a8a945e89187%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [July 19, 2014, 4:49am UTC](https://discuss.elastic.co/t/how-logs-stored-in-logstash-elastisearch/18697/4 "2014-07-19T04:49:56Z")
Hi, On Wednesday, July 16, 2014 8:04:09 AM UTC-4, Sandip Bankewar wrote: > Hello Mark, > > Thanks for your response. > > 1. one log entry in the Logstash is a document what do you mean by that? Look at Lucene. ES uses Lucene. Lucene has the notion of a "document". In case of Logstash+ES, a log event is indexed as one Lucene document. 1. I mean if I have removed the raw data file as a backup purpose and then > after few days I want to copy that again. Not sure what you mean. > 1. Data stored is in Fat file right??? Not sure what you mean. Nothing is "Fat". 1. I have the directory containing this format for data stored as \*logstash-year-month-date > -\>\> 0 1 2 3 4 \_state \* > > _I dont understand which file raw or fat file data stores???_ ES indexes documents (logs in your case) using Lucene. If you can write a bit of Java, write a simple Lucene-based indexer, that may be the simplest way to understand what's written to disk. But if you are struggling with Logstash+ES, you could also simply ship your logs to something like Logsene [http://sematext.com/logsene/](http://sematext.com/logsene/) and not worry about dealing with indexing/ES yourself. ## Otis Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > _Could you please help me on this?_ > > _Regards,_ > _Sandip Bankewar_ > > On Wednesday, 16 July 2014 17:07:26 UTC+5:30, Mark Walkom wrote: > > > 1. It's indexed within Elasticsearch as a json document, one log entry in > > the Logstash is a document > > 2. The default is /var/lib/elasticsearch/data > > 3. No > > 4. You can backup using the snapshot API. What do you mean by remove and > > replace though? > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 16 July 2014 21:18, Sandip Bankewar [sban...@gmail.com](mailto:sban...@gmail.com) wrote: > > > > > Hello All, > > > > > > Can anyone help me on this. > > > > > > 1. How data stored in logstash/elasticsearch? > > > > > > 2. Where is the raw data file(path)? > > > > > > 3. Is it encrypted? > > > > > > 4. Can we take backup of those data and can remove and replace easily? > > > > > > Regards, > > > Sandip Bankewar > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/27143765-c12c-4238-b34f-76d9c38eca83%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c04ad2d0-c0f4-4e7d-9d5b-8a02c79ad884%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c04ad2d0-c0f4-4e7d-9d5b-8a02c79ad884%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:14am UTC](https://discuss.elastic.co/t/how-logs-stored-in-logstash-elastisearch/18697/5 "2017-07-06T01:14:46Z")