# Do not use lot of types per index?

**URL:** <https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121>\
**Category:** Elasticsearch\
**Created:** [October 4, 2016, 1:46am UTC](https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121 "2016-10-04T01:46:14Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [October 4, 2016, 1:46am UTC](https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121/1 "2016-10-04T01:46:14Z")

</div>

In this presentation ([https://www.elastic.co/elasticon/2015/sf/scaling-elasticsearch-for-production-at-verizon](https://www.elastic.co/elasticon/2015/sf/scaling-elasticsearch-for-production-at-verizon)) they said to not use a lot of types per index.

I have now 41 types in filebeat-xx index, is this bad?

I am using the document\_type to set the name of each log file where logs are coming from, should i put this on a field or other place?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 4, 2016, 2:57am UTC](https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121/2 "2016-10-04T02:57:44Z")

</div>

Are they different sorts of logs?

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [October 4, 2016, 3:16am UTC](https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121/3 "2016-10-04T03:16:05Z")

</div>

yes they are all text logs but they have different times format, some are multilines other not, and they are spread over different files

so what i am doing is log1-\* is doc\_type log1, log2-\* is doc\_type log2, etc... then i can search by log type.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 4, 2016, 4:25am UTC](https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121/4 "2016-10-04T04:25:54Z")

</div>

In the future, types are probably going to be removed. See [https://github.com/elastic/elasticsearch/issues/15613](https://github.com/elastic/elasticsearch/issues/15613)  
It will take a long time but I believe it will happen.

Having one type per index seems to be a better option.

You can either put within a log\_type1 object every content coming from type1 and so on or use one index per type.

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [October 4, 2016, 4:34am UTC](https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121/5 "2016-10-04T04:34:19Z")

</div>

what is the first option "put within a log\_type1 object every content coming from type1"? is not having an index per filebeat log file too much?

At the end the logs file cotent is different but the documents are not, they should not have different mapping... basically we are not parsing them on different fields, we are only insterested in the full message line. We just use date to process filter to update date timestamp.

In this case what would be the best option? Would be better to just add a field property indicating the log file name so we can query for all logs line of type1, type2, etc and leave document type as "log" for all?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 4, 2016, 7:09am UTC](https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121/6 "2016-10-04T07:09:16Z")

</div>

> [@agonzalez](#):
>
> what is the first option "put within a log\_type1 object every content coming from type1"?

Yeah sorry. Was on a mobile and hard to explain. I meant:

```auto
{
  "type1": {
    // Whatever structure related to type1 data
  },
  "type2": {
    // Whatever structure related to type2 data
  },
  "type3": {
    // Whatever structure related to type3 data
  }
}

```

> [@agonzalez](#):
>
> is not having an index per filebeat log file too much?

It depends on your retention factor, the total number of shards...

> [@agonzalez](#):
>
> At the end the logs file cotent is different but the documents are not, they should not have different mapping...

In that case, I'd simply add a `type` field inside the document:

```auto
{
  "type": "type1",
  "message": "content"
}

```

It's the best solution IMO in your case.

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [October 4, 2016, 10:20am UTC](https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121/7 "2016-10-04T10:20:22Z")

</div>

how do i add a type field inside document from filebeat? I know i can add fields.type, something like:

```
  paths:
    - /var/log/httpd/error_log
  fields:
   type: httpd_error_log

```

instead of using:

```
  paths:
    - /var/log/httpd/error_log
  document_type: httpd_error_log

```

Is this more efficent to just query logs of httpd\_error\_log and retrieve all messages of that type?

These are two sample docs i have now:

tasker log type:

```
{
  "_index": "filebeat-2016.10.04",
  "_type": "tasker",
  "_id": "AVePp7jp1UM0xgwSbV9J",
  "_score": null,
  "_source": {
    "@timestamp": "2016-10-04T10:25:16.388Z",
    "beat": {
      "hostname": "rc02",
      "name": "rc02"
    },
    "fields": {
      "asset_tag": "822-101-xxxx"
    },
    "input_type": "log",
    "message": "Oct 04 12:25:15 : [LOG0] Deleting task #42508 which has expired",
    "offset": 825908,
    "source": "/usr/bp/logs.dir/tasker-9.log",
    "type": "tasker"
  },
  "fields": {
    "@timestamp": [
      1475576716388
    ]
  },
  "sort": [
    1475576716388
  ]
}

```

bpserver-backup type log:

```
{
  "_index": "filebeat-2016.10.04",
  "_type": "bpserver-backup",
  "_id": "AVePp2Em1UM0xgwSbV2N",
  "_score": null,
  "_source": {
    "@timestamp": "2016-10-04T10:24:54.050Z",
    "beat": {
      "hostname": "rc02",
      "name": "rc02"
    },
    "fields": {
      "asset_tag": "822-101-xxx"
    },
    "input_type": "log",
    "message": "Oct 04 12:24:48 : [LOG0] Load average....: 0.6\n ",
    "offset": 10858,
    "source": "/usr/bp/logs.dir/bpserver-backup-9.log",
    "type": "bpserver-backup"
  },
  "fields": {
    "@timestamp": [
      1475576694050
    ]
  },
  "sort": [
    1475576694050
  ]
}

```

thanks

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 4, 2016, 11:29am UTC](https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121/8 "2016-10-04T11:29:18Z")

</div>

I believe you can't do that in beats in general.  
Before 5.0, you have to process that with Logstash. So forward from filebeat to logstash which mutates your fields and then ships to elasticsearch.  
From 5.0, it's easier as you can define an ingest pipeline directly in elasticsearch. The pipeline will process the original data and mutate it inside elasticsearch before it gets indexed.

---

<div class="post-metadata">

**Author:** ![agonzalez](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@agonzalez](https://discuss.elastic.co/u/agonzalez)\
**Post date:** [October 4, 2016, 11:36am UTC](https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121/9 "2016-10-04T11:36:27Z")

</div>

I am doing it currently with beats adding a field variable to identify the source id in filebeat 5.0 whithout logstash or ingest node pipeline, I am using in the beats config this:

```
"fields": {
  "asset_tag": "822-101-xxx"
}

```

so if i use this:

```
paths:
    - /var/log/httpd/error_log
  fields:
   type: httpd_error_log
   asset_tag: ${ASSET_TAG}

```

I will get this inside my documents:

```
"fields": {
  "asset_tag": "822-101-xxx",
  "type": "httpd_error_log"
}

```

and then I can search by fields.type: httpd\_error\_log

Just wanted to know if this is better and more efficient than having the document\_type/type thing per log as all my logs document are the same format.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 4, 2016, 3:34pm UTC](https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121/10 "2016-10-04T15:34:55Z")

</div>

It looks very good to me. As soon as other logs share the same structure, that's the ideal way IMO.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:15pm UTC](https://discuss.elastic.co/t/do-not-use-lot-of-types-per-index/62121/11 "2017-07-05T22:15:03Z")

</div>