# Question about structure

**URL:** <https://discuss.elastic.co/t/question-about-structure/182201>\
**Category:** Elasticsearch\
**Created:** [May 22, 2019, 10:31am UTC](https://discuss.elastic.co/t/question-about-structure/182201 "2019-05-22T10:31:04Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![yan014](https://avatars.discourse-cdn.com/v4/letter/y/278dde/32.png) [@yan014](https://discuss.elastic.co/u/yan014)\
**Post date:** [May 22, 2019, 10:31am UTC](https://discuss.elastic.co/t/question-about-structure/182201/1 "2019-05-22T10:31:04Z")

</div>

Hello everyone,

I'm developing a program that uses elastic search as a search engine.

I found it interesting to have the opinion of the community on how to structure my data.

The project is quite simple, we have documents (a document can be in several languages ​​and have several versions) that are categorized and have dynamic meta data according to their category.

I have already done some test with "elasticsearch ingest attachment" to send the document and parse directly?

But I do not see how deal with meta data that is dynamic according to the category of the document.

Can you advise me on it?

Thank you in advance,

Have a good day,

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 22, 2019, 2:01pm UTC](https://discuss.elastic.co/t/question-about-structure/182201/2 "2019-05-22T14:01:32Z")

</div>

Are you providing the metadata by yourself?

---

<div class="post-metadata">

**Author:** ![yan014](https://avatars.discourse-cdn.com/v4/letter/y/278dde/32.png) [@yan014](https://discuss.elastic.co/u/yan014)\
**Post date:** [May 22, 2019, 2:44pm UTC](https://discuss.elastic.co/t/question-about-structure/182201/3 "2019-05-22T14:44:09Z")

</div>

Hello 🙂

Yes, fields encoded by me like dates, text, numbers, multiple values...

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 22, 2019, 3:16pm UTC](https://discuss.elastic.co/t/question-about-structure/182201/4 "2019-05-22T15:16:54Z")

</div>

So you can do something like:

```auto
PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data"
      }
    }
  ]
}
PUT my_index/_doc/my_id?pipeline=attachment
{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
  "meta": {
    "foo1": "bar1",
    "foo2": "bar2"
  }
}

```

And that should work.

---

<div class="post-metadata">

**Author:** ![yan014](https://avatars.discourse-cdn.com/v4/letter/y/278dde/32.png) [@yan014](https://discuss.elastic.co/u/yan014)\
**Post date:** [May 23, 2019, 8:20am UTC](https://discuss.elastic.co/t/question-about-structure/182201/5 "2019-05-23T08:20:59Z")

</div>

Thank you, I thought to do this 😃

And for categories, I add them like this:

```
PUT my_index/_doc/my_id?pipeline=attachment
{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
  "category": {
      "id": 1,
      "name": "Category name "
  },
  "meta": {
    "foo1": "bar1",
    "foo2": "bar2"
  }
}

```

OR

```
PUT my_index/_doc/my_id?pipeline=attachment
{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
  "category": "Category name ",
  "meta": {
    "foo1": "bar1",
    "foo2": "bar2"
  }
}

```

Or like this and I have a way of making a relationship with another index containing the categories?

```
PUT my_index/_doc/my_id?pipeline=attachment
{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
  "category_id": 1,
  "meta": {
    "foo1": "bar1",
    "foo2": "bar2"
  }
}

```

Thanks a lot 😃

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 23, 2019, 1:02pm UTC](https://discuss.elastic.co/t/question-about-structure/182201/6 "2019-05-23T13:02:04Z")

</div>

> [@yan014](#):
>
> Or like this and I have a way of making a relationship with another index containing the categories?

I'd not do that. I'd do:

```auto
PUT my_index/_doc/my_id?pipeline=attachment
{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
  "category": "Category name ",
  "meta": {
    "foo1": "bar1",
    "foo2": "bar2"
  }
}

```

---

<div class="post-metadata">

**Author:** ![yan014](https://avatars.discourse-cdn.com/v4/letter/y/278dde/32.png) [@yan014](https://discuss.elastic.co/u/yan014)\
**Post date:** [May 23, 2019, 1:22pm UTC](https://discuss.elastic.co/t/question-about-structure/182201/7 "2019-05-23T13:22:21Z")

</div>

Thanks 😉

If my document and my categories are multilingual, I index the document for each language?

No need to have this :

```
PUT my_index/_doc/my_id?pipeline=attachment
{
  "data": {
    "en": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
    "fr": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
 },
  "category": {
    "en": "Category name EN",
    "fr": "Category name FR"
 },
  "meta": {
    "foo1": "bar1",
    "foo2": "bar2"
  }
}

```

but

```
PUT my_index/_doc/my_id?pipeline=attachment
{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
  "category": "Category name EN ",
  "meta": {
    "foo1": "bar1",
    "foo2": "bar2"
  }
}

```

AND the file in french :

```
PUT my_index/_doc/my_id?pipeline=attachment
{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
  "category": "Category name FR ",
  "meta": {
    "foo1": "bar1",
    "foo2": "bar2"
  }
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 23, 2019, 1:36pm UTC](https://discuss.elastic.co/t/question-about-structure/182201/8 "2019-05-23T13:36:16Z")

</div>

I prefer the later form but I'd use 2 indices: `my_index_fr` and `my_index_en`. Unless there is a need to have an absolute relationship between both versions of the same document.

---

<div class="post-metadata">

**Author:** ![yan014](https://avatars.discourse-cdn.com/v4/letter/y/278dde/32.png) [@yan014](https://discuss.elastic.co/u/yan014)\
**Post date:** [May 23, 2019, 1:49pm UTC](https://discuss.elastic.co/t/question-about-structure/182201/9 "2019-05-23T13:49:55Z")

</div>

Why the advantages to use 2 indices ?

And what do you mean by "an absolute relationship", how can i do that with ES ?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 23, 2019, 2:07pm UTC](https://discuss.elastic.co/t/question-about-structure/182201/10 "2019-05-23T14:07:52Z")

</div>

I mean that if you want to have within the same elasticsearch document both versions, then you have to do something like:

```auto
PUT my_index/_doc/my_id?pipeline=attachment
{
  "data": {
    "en": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
    "fr": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
 },
  "category": {
    "en": "Category name EN",
    "fr": "Category name FR"
 },
  "meta": {
    "foo1": "bar1",
    "foo2": "bar2"
  }
}

```

---

<div class="post-metadata">

**Author:** ![yan014](https://avatars.discourse-cdn.com/v4/letter/y/278dde/32.png) [@yan014](https://discuss.elastic.co/u/yan014)\
**Post date:** [May 23, 2019, 2:13pm UTC](https://discuss.elastic.co/t/question-about-structure/182201/11 "2019-05-23T14:13:10Z")

</div>

Okey 🙂

No I do not need it " i think 😛 " , otherwise I also have a database mysql behind which can do it.

And what the advantages to use 2 indices ?

Thanks a lot for all the informations and your time 😃

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 23, 2019, 2:40pm UTC](https://discuss.elastic.co/t/question-about-structure/182201/12 "2019-05-23T14:40:00Z")

</div>

> And what the advantages to use 2 indices ?

Well. It reduces the number of fields within one index. Not a big deal here I guess as you have a few of them.  
What I'd think about is "reindex" needs. If something goes wrong with a specific lang, ie `FR` and you need to change the text analyzer (which means reindex). Do you want to reindex both languages or only one?

I prefer one but it's really up to you.

Just giving my 2 cents here 😉

---

<div class="post-metadata">

**Author:** ![yan014](https://avatars.discourse-cdn.com/v4/letter/y/278dde/32.png) [@yan014](https://discuss.elastic.co/u/yan014)\
**Post date:** [June 3, 2019, 10:35am UTC](https://discuss.elastic.co/t/question-about-structure/182201/13 "2019-06-03T10:35:19Z")

</div>

Thanks a lot for the informations 😃

Dummy question about "PUT \_ingest/pipeline/attachment", i need to call once or for each "PUT  
my\_index/\_doc/my\_id?pipeline=attachment" in different request ?

Thanks 😉

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 3, 2019, 10:47am UTC](https://discuss.elastic.co/t/question-about-structure/182201/14 "2019-06-03T10:47:40Z")

</div>

Only once.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 1, 2019, 10:47am UTC](https://discuss.elastic.co/t/question-about-structure/182201/15 "2019-07-01T10:47:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.