# Where data is stored?

**URL:** <https://discuss.elastic.co/t/where-data-is-stored/98441>\
**Category:** Elasticsearch\
**Created:** [August 26, 2017, 7:07am UTC](https://discuss.elastic.co/t/where-data-is-stored/98441 "2017-08-26T07:07:22Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![hack3rcon](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@hack3rcon](https://discuss.elastic.co/u/hack3rcon)\
**Post date:** [August 26, 2017, 7:07am UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/1 "2017-08-26T07:07:22Z")

</div>

Hello.  
When I send my Windows Event Logs using "Winlogbeat" directly to "Elastic" then where is my data stored? I mean is something like file.

Thank you.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 26, 2017, 9:15am UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/2 "2017-08-26T09:15:28Z")

</div>

It's stored in Elasticsearch, where depends on how you installed it - [https://www.elastic.co/guide/en/elasticsearch/reference/5.5/install-elasticsearch.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/install-elasticsearch.html)

---

<div class="post-metadata">

**Author:** ![hack3rcon](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@hack3rcon](https://discuss.elastic.co/u/hack3rcon)\
**Post date:** [August 26, 2017, 9:49am UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/3 "2017-08-26T09:49:57Z")

</div>

I installed it via "yum" command on CentOS. where is the correct directory?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 26, 2017, 10:00am UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/4 "2017-08-26T10:00:38Z")

</div>

The directory is mentioned on this page [https://www.elastic.co/guide/en/elasticsearch/reference/5.5/rpm.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/rpm.html)

---

<div class="post-metadata">

**Author:** ![hack3rcon](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@hack3rcon](https://discuss.elastic.co/u/hack3rcon)\
**Post date:** [August 26, 2017, 10:13am UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/5 "2017-08-26T10:13:56Z")

</div>

You mean is something like:

```auto
/var/lib/elasticsearch/nodes/0/indices
```

But they are not human readable!!!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 26, 2017, 10:24am UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/6 "2017-08-26T10:24:48Z")

</div>

No, why does that matter though? **Never** interact with the files that Elasticsearch creates directly on the filesystem, always use the APIs.

---

<div class="post-metadata">

**Author:** ![hack3rcon](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@hack3rcon](https://discuss.elastic.co/u/hack3rcon)\
**Post date:** [August 26, 2017, 12:40pm UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/7 "2017-08-26T12:40:46Z")

</div>

For backup?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [August 26, 2017, 3:40pm UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/8 "2017-08-26T15:40:31Z")

</div>

You shouldn't use file backups for Elasticsearch, but rather the [snapshot and restore](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/modules-snapshots.html) APIs. This can be done by way of other tools, like [Elasticsearch Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/snapshot.html).

The primary reason to not use a file-type backup approach is that the data would very likely be corrupted. The Lucene data structures in the should-never-be-touched data paths are in constant change so long as indexing is going on. If one file were backed up while another were changing, then there would be a mismatch, and corruption would ensue. Your file-based backup would be worthless.

---

<div class="post-metadata">

**Author:** ![hack3rcon](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@hack3rcon](https://discuss.elastic.co/u/hack3rcon)\
**Post date:** [August 27, 2017, 6:52am UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/9 "2017-08-27T06:52:45Z")

</div>

Thank you so much for your info. I have some questions:  
1- In "Repository" section, Code must be written in "Dev Tools" ?  
2- If I don't like to work with "Dev Tools" then I must read "Shared File System Repository" and change "elasticsearch.yml" configuration?  
3- both are same?  
4- How about restore them from file via config file?

I see two parameters `path.repo: ["/mount/backups", "/mount/longterm_backups"]` What is "/mount/longterm\_backups" and is it mandatory?

I added below line to my Elasticsearch configuration and restart "elasticsearch" service but no file created:

```auto
path.repo: ["/var/log/back","/var/log/back-long"]
```

I can see another option in my configuration file "#path.data: /path/to/data"!!!!  
Tnx.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [August 28, 2017, 1:40pm UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/10 "2017-08-28T13:40:15Z")

</div>

> [@hack3rcon](#):
>
> 1- In “Repository” section, Code must be written in “Dev Tools” ?

If you're creating the repository that way, then yes. Otherwise there's a rather undocumented tool included with Curator called `es_repo_mgr` which allows you to create `fs` and `s3` repositories at the command line. Just run `es_repo_mgr --help` and see the options you can use, which should mirror the ones in the online example.

> [@hack3rcon](#):
>
> 2- If I don’t like to work with “Dev Tools” then I must read “Shared File System Repository” and change “elasticsearch.yml” configuration?

You _must_ configure the shared filesystem repository (type `fs`) in _both_ places. It must be done in the API _and_ with `path.repo` in `elasticsearch.yml`.

> [@hack3rcon](#):
>
> 4- How about restore them from file via config file?

Restore is done using the API, as linked above, or the [Restore](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/restore.html) action in Curator, which does use a YAML configuration file.

> [@hack3rcon](#):
>
> I added below line to my Elasticsearch configuration and restart “elasticsearch” service but no file created:

That does not create a path. It merely tells Elasticsearch that use of that path is acceptable. The path must still be added via the API (or a tool like the aforementioned `es_repo_mgr`, which does the API calls for you).

---

<div class="post-metadata">

**Author:** ![hack3rcon](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@hack3rcon](https://discuss.elastic.co/u/hack3rcon)\
**Post date:** [August 29, 2017, 12:03pm UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/11 "2017-08-29T12:03:33Z")

</div>

API? Can you give me an example?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [August 29, 2017, 3:11pm UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/12 "2017-08-29T15:11:12Z")

</div>

The examples are in the snapshot and restore link I mentioned previously, but here's a [link to one](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/modules-snapshots.html#_repositories):

```auto
PUT /_snapshot/my_backup
{
  "type": "fs",
  "settings": {
        ... repository specific settings ...
  }
}

```

---

<div class="post-metadata">

**Author:** ![hack3rcon](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@hack3rcon](https://discuss.elastic.co/u/hack3rcon)\
**Post date:** [September 6, 2017, 5:51am UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/13 "2017-09-06T05:51:26Z")

</div>

Excuse me, I can't understand ` ... repository specific settings ...` !!!  
I like to create a backup of "server1-_" and "server2-_".

```auto
# curl -XGET 'http://localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open server2-2017.08.28 8KF2JptfS9q6qNgsscqDLQ 5 1 6 0 121.2kb 121.2kb
yellow open server1-2017.09.01 hQl7hSIfRPmbL6wBjqSwpw 5 1 17 0 217kb 217kb
yellow open server1-2017.08.28 00yeQwotQj-s3ZEHXvM-SA 5 1 2 0 40.2kb 40.2kb
yellow open .kibana qx6nf2-4Q9O7jU_ron7eNw 1 1 3 0 23.3kb 23.3kb
```

My elasticsearch config is as below:

```auto
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
path.repo: ["/var/log/back","/var/log/back-long"]
#
# Path to log files:
#
path.logs: /var/log/elastic
#
```

I'm thankful if you show me a good code.

Thank you.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [September 6, 2017, 3:14pm UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/14 "2017-09-06T15:14:01Z")

</div>

> [@hack3rcon](#):
>
> I like to create a backup of “server1-" and "server2-”

That's not how snapshots work. You snapshot selected indices from _all_ nodes (the entire cluster), or not at all.

> [@hack3rcon](#):
>
> Excuse me, I can’t understand `... repository specific settings ...`

Did you visit [the link](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/modules-snapshots.html#_repositories) I sent already? It has examples which show `location` and `compression` as potential options for `settings`:

```auto
        "location": "/mount/backups/my_backup",
        "compress": true

```

I'm a bit concerned that you're defining your `path.repo` as `/var/log/back` or `/var/log/back-long`. Are these shared file systems that just happen to be mounted in `/var/log`? If not, then you will not be able to create a snapshot repository. A snapshot repository _must_ be a _shared_ filesystem, like NFS, to which each master and data node has read and write access.

---

<div class="post-metadata">

**Author:** ![hack3rcon](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@hack3rcon](https://discuss.elastic.co/u/hack3rcon)\
**Post date:** [September 10, 2017, 7:28am UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/15 "2017-09-10T07:28:14Z")

</div>

Thank you.  
It just a test and I know "/var/log" is not a good location. I like to create a backup and then remove server and restore my backup.  
I read the link but I'm a beginner and... I don't know what is ` ... repository specific settings ...`. is it parameters or...  
I'm thankful if you provide the commands.

---

<div class="post-metadata">

**Author:** ![hack3rcon](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@hack3rcon](https://discuss.elastic.co/u/hack3rcon)\
**Post date:** [September 11, 2017, 12:29pm UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/16 "2017-09-11T12:29:03Z")

</div>

Can `... repository specific settings ...` be:

```auto
"compress": true,
"location": "/mount/backups/my_backup"
```

?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [September 11, 2017, 1:06pm UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/17 "2017-09-11T13:06:36Z")

</div>

Yes. Exactly.

---

<div class="post-metadata">

**Author:** ![hack3rcon](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@hack3rcon](https://discuss.elastic.co/u/hack3rcon)\
**Post date:** [September 11, 2017, 2:06pm UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/18 "2017-09-11T14:06:32Z")

</div>

Is `"location"` vs `path.repo` in my config file?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [September 11, 2017, 2:09pm UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/19 "2017-09-11T14:09:54Z")

</div>

Yes, `location` should match one of the entries in `path.repo`. `path.repo` is where you tell Elasticsearch that it is acceptable to use that given mount point as a `location` for a repository. It is a hard, config-file based whitelist.

---

<div class="post-metadata">

**Author:** ![hack3rcon](https://avatars.discourse-cdn.com/v4/letter/h/96bed5/32.png) [@hack3rcon](https://discuss.elastic.co/u/hack3rcon)\
**Post date:** [September 16, 2017, 10:25am UTC](https://discuss.elastic.co/t/where-data-is-stored/98441/20 "2017-09-16T10:25:13Z")

</div>

I have written:

```auto
PUT /_snapshot/my_backup
{
  "type": "fs",
  "settings": {
        "compress": true,
		"location": "/var/log/back"
  }
}
```

and I got:

```auto
{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "failed to create blob container"
      }
    ],
    "type": "exception",
    "reason": "failed to create blob container",
    "caused_by": {
      "type": "access_denied_exception",
      "reason": "/var/log/back/tests-CHKsf_FGS2muTXMF555buA"
    }
  },
  "status": 500
}
```

Why?

[Next page](https://discuss.elastic.co/t/where-data-is-stored/98441.md?page=2)