# Index different source files to different indices?

**URL:** <https://discuss.elastic.co/t/index-different-source-files-to-different-indices/124052>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [March 15, 2018, 9:23am UTC](https://discuss.elastic.co/t/index-different-source-files-to-different-indices/124052 "2018-03-15T09:23:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [March 15, 2018, 9:23am UTC](https://discuss.elastic.co/t/index-different-source-files-to-different-indices/124052/1 "2018-03-15T09:23:39Z")

</div>

Hi,

currently my first trial of a track looks like this:

```
{
  "description": "my first benchmark",
  "indices": [
    {
      "name": "activemq_queue_count",
	  "auto-managed": false,
	  "body": "mappings/activemq_queue_count.json",
      "types": [
        {
          "name": "docs"
        }
      ]
    }
  ],
  "corpora": [
    {
      "name": "mytest",
      "documents": [
        {
          "source-file": "activemq_queue_count.json",
          "document-count": 63495,
          "uncompressed-bytes": 34230751
        }
      ]
    }
  ],
  "challenge": {
    "name": "index-only",
    "schedule": [
      {
        "operation": "delete-index"
      },
      {
        "operation": {
          "operation-type": "create-index",
          "settings": {
            "index.number_of_replicas": 0
          }
        }
      },
      {
        "operation": {
          "operation-type": "cluster-health",
          "request-params": {
            "wait_for_status": "yellow"
          }
        }
      },
      {
        "operation": {
          "operation-type": "bulk",
          "bulk-size": 5000
        },
        "warmup-time-period": 5,
        "clients": 8
      }
    ]
  }
}

```

We have many different logs. Coming from ES 5.x we had multiple types in a single index. Now with ES 6.x I would like to test the differences (especially index / query performance, storage needs, etc.) between 1 type per index and all types of logs in one index (but stored as one type (docs) and filterable by a custom field logType).

In the example above I only call bulk. Since there is only one index and only one document source, there is no problem. But how can I define which file should be bulk indexed to which index if I define more than one indices / files?

---

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [March 16, 2018, 8:34am UTC](https://discuss.elastic.co/t/index-different-source-files-to-different-indices/124052/2 "2018-03-16T08:34:06Z")

</div>

Hello,

My understanding is that you'd like to benchmark the performance between two scenarios, one having multiple indices with only one type and the second using one index with a custom type field, as explained in the elasticsearch docs [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html#_alternatives_to_mapping_types).

I have prepared a [complete gist](https://gist.github.com/dliappis/f90f30cd605ce9a115799296e044af42) with an example `track.json`, mappings and document files, loosely based on your example above, and commands to test the two scenarios as separate challenges.

Below I have broken down my suggestions for the track file per scenario, with references in the gist code:

#### 1. Many indices, one type, separate document files per index

For this case you'll define your indices separately in the `indices` array and specify your document files in the `documents` array of a corpus. Each element in the `documents` array [can specify an index](https://esrally.readthedocs.io/en/stable/track.html#corpora). In my gist this is accomplished [here](https://gist.github.com/dliappis/f90f30cd605ce9a115799296e044af42#file-track-json-L27-L40).

You can then have a dedicated challenge for this scenario. The bulk operation to index the documents to the corresponding indices references the corpus defined earlier, as shown in my [gist section](https://gist.github.com/dliappis/f90f30cd605ce9a115799296e044af42#file-track-json-L98-L106).

#### 2. One index with custom type field

In this case it's sufficient to specify the custom type field in the [combined document](https://gist.github.com/dliappis/f90f30cd605ce9a115799296e044af42#file-documents-combined-json-L1-L4), pretty much mirroring the custom type field example in the [elasticsearch docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html#_custom_type_field).

In the same tracks file this can be implemented with an [additional corpus](https://gist.github.com/dliappis/f90f30cd605ce9a115799296e044af42#file-track-json-L42-L53) targeting the index and a [dedicated challenge](https://gist.github.com/dliappis/f90f30cd605ce9a115799296e044af42#file-track-json-L109-L148); the bulk operation again specifies the corpus which we've already defined which [index to target](https://gist.github.com/dliappis/f90f30cd605ce9a115799296e044af42#file-track-json-L49).

Dimitris

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 13, 2018, 8:34am UTC](https://discuss.elastic.co/t/index-different-source-files-to-different-indices/124052/3 "2018-04-13T08:34:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.