# Simple elasticsearch database designe help
**URL:** https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935
**Category:** Elasticsearch
**Created:** [January 20, 2019, 12:09pm UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935 "2019-01-20T12:09:14Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@Mart](https://discuss.elastic.co/u/Mart)
#### Post date: [January 20, 2019, 12:09pm UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/1 "2019-01-20T12:09:14Z")
Hello! I have a question about elastic database design. My data is 10 different collections spread to tapes. Every tape can have one or more collections stored and on every tape there are 100K+ unique files. In elastic I would like to store only metadata about `tapeName`, `collectionName`, `fileName`, `fileInfo`, `fileSize`, `fileDate`, `fileHash`. Searches would be mainly collection specific but also sometimes all collections wide. Physical layout:
**tape1**
collection1
|-\> file1
|-\> ..
**tape2**
collection1
|-\> file1x
|-\> ..
collection2
|-\> file1
|-\>..
Below is what I have in mind so far.. or should it be different index for every collection or smb else?
```
PUT tapes
{
"settings" : {
"number_of_shards" : 1,
"number_of_replicas" : 0
},
"mappings": {
"_doc": {
"properties": {
"tapeName": {"type": "text"},
"collectionName": {"type": "keyword"},
"fileName": {"type": "text"},
"fileInfo": {"type": "text"},
"fileSize": {"type": "integer"},
"fileDate": {"type": "date"},
"fileHash": {"type": "keyword"}
}
}
}
}
```
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 21, 2019, 2:51am UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/2 "2019-01-21T02:51:19Z")
Will every document have a `failDate`?
---
### Author:  [@Mart](https://discuss.elastic.co/u/Mart)
#### Post date: [January 21, 2019, 4:24am UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/3 "2019-01-21T04:24:06Z")
Yes, every document have `fileName`, `fileSize`, `fileDate`, `fileHash` always filled. `fileInfo` is filled very rarerly. And all the documents are associated with `tapeName` and `collectionName`.
`fileDate` is the file modification date from filesystem. In case there are more then one with same `fileName` and they have different `fileHash`, I can compare `fileDate` to make some decision.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 21, 2019, 4:25am UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/4 "2019-01-21T04:25:27Z")
Fail or file?
---
### Author:  [@Mart](https://discuss.elastic.co/u/Mart)
#### Post date: [January 21, 2019, 4:27am UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/5 "2019-01-21T04:27:53Z")
sry, yes file π
fixed above. In my mother language fail translates to file π
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 21, 2019, 6:32am UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/6 "2019-01-21T06:32:35Z")
I'd suggest you use time based indices, even if yearly. eg `tapes-YYYY`, and then use the `fileDate` as the timestamp.
Otherwise what you are suggesting looks fine, and it's great that you are thinking this far ahead π
---
### Author:  [@Mart](https://discuss.elastic.co/u/Mart)
#### Post date: [January 21, 2019, 11:16am UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/7 "2019-01-21T11:16:12Z")
Thank you very much for your suggestion! I will seriously consider it!
Beside brand new files, old data is rotated ie. 5-8 years old tapes are deleted and rewritten to new ones. When I use indices per year, then it would be very easy to delete old years. Nice.
Priority is to find from `fileName` value and usually it is known from witch collection it should be. Also `collectionName` and `tapeName` are important. Other data (info, date, size, hash) is secondary.
Do I understand correctly that if I have indices per year, then I have to search through all collections to find ie. `fileName`?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 21, 2019, 9:14pm UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/8 "2019-01-21T21:14:45Z")
Yes, but that's not really anything worth worrying about.
---
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [January 21, 2019, 9:19pm UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/9 "2019-01-21T21:19:15Z")
Warkolm,
so you are saying indice name is tapename-yyyy ?
if he will have 1000 tapes then he will have 1000 indice.
what if he use same indice and use document\_id = tapes-YYYY., and fileDate as timestamp ?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 21, 2019, 10:05pm UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/10 "2019-01-21T22:05:01Z")
> [@elasticforme](#):
>
> if he will have 1000 tapes then he will have 1000 indice.
If you have one per tape, yes. But I suggested that they group all tapes by year.
---
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [January 21, 2019, 10:17pm UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/11 "2019-01-21T22:17:54Z")
oh ok I misunderstood.
index name is Tapes-2019, then Tapes-2020 got it
---
### Author:  [@Mart](https://discuss.elastic.co/u/Mart)
#### Post date: [January 22, 2019, 5:29am UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/12 "2019-01-22T05:29:58Z")
I can think of two possibilities for yearly indices:
a) current year (tapes-2019, tapes-2020..) and that makes total indices count to 5-8 (time window for rewrite tapes). Itβs easy to discard old indices (simply delete from file system) but individual indices size will grow gradually (new data is added constantly and old is rewritten).
b) `fileDate` year (tapes-1990, .. tapes-2019) and that makes total indices count to 29 as of today. Indices size would be more spread out but discarded tapes data have to be deleted inside indices separately. Is it a problem?
As there are 100000+ files per tape, should I look for [join datatype](https://www.elastic.co/guide/en/elasticsearch/reference/current/parent-join.html) for example `tapeName` as parent and all other fields as child? Or `collectionName` as parent and all others as child?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 22, 2019, 5:31am UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/13 "2019-01-22T05:31:07Z")
Just make it flat. Each document would contain all the fields required.
---
### Author:  [@Mart](https://discuss.elastic.co/u/Mart)
#### Post date: [January 22, 2019, 5:45am UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/14 "2019-01-22T05:45:49Z")
Thanks for helping me! I preciate it and happy smile π
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 19, 2019, 5:45am UTC](https://discuss.elastic.co/t/simple-elasticsearch-database-designe-help/164935/15 "2019-02-19T05:45:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.