# Rally error while running against custom track and custom data

**URL:** <https://discuss.elastic.co/t/rally-error-while-running-against-custom-track-and-custom-data/114963>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [January 10, 2018, 9:17pm UTC](https://discuss.elastic.co/t/rally-error-while-running-against-custom-track-and-custom-data/114963 "2018-01-10T21:17:06Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![dipanjan44](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@dipanjan44](https://discuss.elastic.co/u/dipanjan44)\
**Post date:** [January 10, 2018, 9:17pm UTC](https://discuss.elastic.co/t/rally-error-while-running-against-custom-track-and-custom-data/114963/1 "2018-01-10T21:17:07Z")

</div>

INFO] Writing logs to /Users/.rally/logs/rally\_out\_20180110T211006Z.log  
[INFO] Preparing for race ...  
[INFO] Rally will delete the benchmark candidate after the benchmark  
[INFO] Racing on track [contactdata], challenge [createcontactindex] and car ['defaults'] with version [7.0.0-alpha1].

[INFO] Preparing file offset table for [/Users/.rally/benchmarks/data/contactdata/contactdata.json] ... [OK]

[ERROR] Cannot race. ('Could not execute benchmark', RequestError(400, 'mapper\_parsing\_exception', {'error': {'root\_cause': [{'type': 'mapper\_parsing\_exception', 'reason': 'normalizer [lowercase\_normalizer] not found for field [country]'}], 'type': 'mapper\_parsing\_exception', 'reason': 'Failed to parse mapping [contact]: normalizer [lowercase\_normalizer] not found for field [country]', 'caused\_by': {'type': 'mapper\_parsing\_exception', 'reason': 'normalizer [lowercase\_normalizer] not found for field [country]'}}, 'status': 400}))

# The error logs only say this:

2018-01-10 21:11:37,510 PID:21118 rally.driver ERROR Main driver encountered a fatal exception. Shutting down.  
Traceback (most recent call last):  
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/esrally/driver/driver.py", line 160, in receiveMessage  
expected\_status=None, new\_status=None, transition=self.after\_track\_prepared)  
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/esrally/actor.py", line 90, in transition\_when\_all\_children\_responded  
transition()  
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/esrally/driver/driver.py", line 257, in after\_track\_prepared  
self.coordinator.after\_track\_prepared()  
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/esrally/driver/driver.py", line 364, in after\_track\_prepared  
self.prepare\_cluster()  
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/esrally/driver/driver.py", line 475, in prepare\_cluster  
setup\_index(es, index, self.challenge.index\_settings)  
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/esrally/driver/driver.py", line 844, in setup\_index  
es.indices.create([index=index.name](http://index=index.name), body=body)  
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 73, in \_wrapped  
return func(\*args, params=params, \*\*kwargs)  
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/elasticsearch/client/indices.py", line 107, in create  
params=params, body=body)  
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/elasticsearch/transport.py", line 312, in perform\_request  
status, headers, data = connection.perform\_request(method, url, params, body, ignore=ignore, timeout=timeout)  
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/elasticsearch/connection/http\_urllib3.py", line 128, in perform\_request  
self.\_raise\_error(response.status, raw\_data)  
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 125, in \_raise\_error  
raise HTTP\_EXCEPTIONS.get(status\_code, TransportError)(status\_code, error\_message, additional\_info)  
elasticsearch.exceptions.RequestError: TransportError(400, 'mapper\_parsing\_exception', 'normalizer [lowercase\_normalizer] not found for field [country]')

==========================

Could anyone please guide whether it is a problem with the test data or the mapping.json for index creation is wrong?

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [January 11, 2018, 7:13am UTC](https://discuss.elastic.co/t/rally-error-while-running-against-custom-track-and-custom-data/114963/2 "2018-01-11T07:13:58Z")

</div>

This is a problem with the mapping file. It say that it cannot find the `lowercase_normalizer` that you have defined for the field `country`.

You should be able to reproduce this by issuing the following curl statements:

```auto
# create an empty index
curl -XPUT 'localhost:9200/name_of_your_index_here?pretty' -H 'Content-Type: application/json' -d'
{}
'

# issue the PUT mapping call
curl -XPUT 'localhost:9200/name_of_your_index_here/_mapping/contact?pretty' -H 'Content-Type: application/json' -d @mapping.json

```

---

<div class="post-metadata">

**Author:** ![dipanjan44](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@dipanjan44](https://discuss.elastic.co/u/dipanjan44)\
**Post date:** [January 11, 2018, 3:58pm UTC](https://discuss.elastic.co/t/rally-error-while-running-against-custom-track-and-custom-data/114963/3 "2018-01-11T15:58:44Z")

</div>

Hello Daniel,

My mapping file looks like this:

{  
"settings": {  
"analysis": {  
"normalizer": {  
"lowercase\_normalizer": {  
"type": "custom",  
"char\_filter": [],  
"filter": ["lowercase"]  
}  
}  
},  
"index": {  
"number\_of\_shards": 5,  
"number\_of\_replicas": 2  
},  
"index.requests.cache.enable": true  
},  
"mappings": {  
"contact": {  
"dynamic": "false",  
"\_routing": {  
"required": true  
},  
"properties": {  
"type": {  
"type": "keyword",  
"normalizer": "lowercase\_normalizer"  
},  
"joinField": {  
"type": "join",  
"relations": {  
"contact": "location"  
}  
},  
"country": {  
"type": "keyword",  
"normalizer": "lowercase\_normalizer"  
}  
}  
}  
}  
}

Rally throws the below error, when i try to use it. However it works fine when i use this mapping file directly against ES.

[INFO] Writing logs to /Users/dipanjan.haldar/.rally/logs/rally\_out\_20180111T154711Z.log  
[INFO] Preparing for race ...  
[INFO] Rally will delete the benchmark candidate after the benchmark  
[INFO] Racing on track [contactdata], challenge [createcontactindex] and car ['defaults'] with version [6.1.1].

[ERROR] Cannot race. ('Could not execute benchmark', RequestError(400, 'mapper\_parsing\_exception', {'error': {'root\_cause': [{'type': 'mapper\_parsing\_exception', 'reason': 'Root mapping definition has unsupported parameters: [index.requests.cache.enable : true] [index : {number\_of\_shards=5, number\_of\_replicas=2}] [analysis : {normalizer={lowercase\_normalizer={filter=[lowercase], char\_filter=[], type=custom}}}]'}], 'type': 'mapper\_parsing\_exception', 'reason': 'Failed to parse mapping [settings]: Root mapping definition has unsupported parameters: [index.requests.cache.enable : true] [index : {number\_of\_shards=5, number\_of\_replicas=2}] [analysis : {normalizer={lowercase\_normalizer={filter=[lowercase], char\_filter=[], type=custom}}}]', 'caused\_by': {'type': 'mapper\_parsing\_exception', 'reason': 'Root mapping definition has unsupported parameters: [index.requests.cache.enable : true] [index : {number\_of\_shards=5, number\_of\_replicas=2}] [analysis : {normalizer={lowercase\_normalizer={filter=[lowercase], char\_filter=[], type=custom}}}]'}}, 'status': 400}))

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [January 11, 2018, 4:53pm UTC](https://discuss.elastic.co/t/rally-error-while-running-against-custom-track-and-custom-data/114963/4 "2018-01-11T16:53:37Z")

</div>

Hi Dipanjan,

I got the problem now. Unfortunately, Rally 0.8.1 does not allow you do combine settings and mappings into one file. This will be fixed in the upcoming release Rally 0.9.0 but to make it work with Rally 0.8.1, you need to use a somewhat akward syntax

1. In your challenge, you need to define an `index-settings` property (also see the [docs](http://esrally.readthedocs.io/en/stable/track.html#challenge)):

```auto
  "index-settings": {
    "analysis": {
      "normalizer": {
        "lowercase_normalizer": {
          "type": "custom",
          "char_filter": [],
          "filter": ["lowercase"]
        }
      }
    },
    "index": {
      "number_of_shards": 5,
      "number_of_replicas": 2
    },
    "index.requests.cache.enable": true    
  }

```

Then you need to store your mappings for the contact type in a mapping file, let's call it `contact-mapping.json`. The contents of the file are as follows:

```auto
{
  "contact": {
    "dynamic": "false",
    "_routing": {
      "required": true
    },
    "properties": {
      "type": {
        "type": "keyword",
        "normalizer": "lowercase_normalizer"
      },
      "joinField": {
        "type": "join",
        "relations": {
          "contact": "location"
        }
      },
      "country": {
        "type": "keyword",
        "normalizer": "lowercase_normalizer"
      }
    }
  }
}

```

Then you need to reference this file in your index definition. E.g.:

```auto
  "indices": [
    {
      "name": "my-index-name",
      "types": [
        {
          "name": "contact",
          "mapping": "contact-mapping.json",
          "other properties here...": "..."
        }
      ]
    }
  ]

```

That should work. And again: With 0.9.0 things will work much more straightforward and you can just provide one file for the whole index body (as you already tried).

---

<div class="post-metadata">

**Author:** ![dipanjan44](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@dipanjan44](https://discuss.elastic.co/u/dipanjan44)\
**Post date:** [January 11, 2018, 8:26pm UTC](https://discuss.elastic.co/t/rally-error-while-running-against-custom-track-and-custom-data/114963/5 "2018-01-11T20:26:48Z")

</div>

Thanks Daniel, the issue is resolved and I am able to run the test

---

<div class="post-metadata">

**Author:** ![dipanjan44](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@dipanjan44](https://discuss.elastic.co/u/dipanjan44)\
**Post date:** [January 11, 2018, 9:42pm UTC](https://discuss.elastic.co/t/rally-error-while-running-against-custom-track-and-custom-data/114963/6 "2018-01-11T21:42:31Z")

</div>

Hi Daniel,

If I use a datafile with the following content, rally doesnt upload the data to ES. However this works fine when used directly against ES.

{"index": {"\_type": "contact", "\_id": 1122333333, "\_index": "contact\_current"}}  
{ "firstName": "Bob", "organizationId": 1, "lastName": "test", "middleInitial": "test", "createdDate": "2016-07-15T00:19:02.940Z", "lastModifiedDate": "2017-02-10T19:01:43.580Z", "type": "contact", "joinField": "contact"}

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [January 12, 2018, 6:40am UTC](https://discuss.elastic.co/t/rally-error-while-running-against-custom-track-and-custom-data/114963/7 "2018-01-12T06:40:40Z")

</div>

Hi Dipanjan,

how does Rally behave exactly? Your data file contains:

An action and metadata line which is:

```auto
{"index": {"_type": "contact", "_id": 1122333333, "_index": "contact_current"}}

```

and the actual data:

```auto
{ "firstName": "Bob", "organizationId": 1, "lastName": "test", "middleInitial": "test", "createdDate": "2016-07-15T00:19:02.940Z", "lastModifiedDate": "2017-02-10T19:01:43.580Z", "type": "contact", "joinField": "contact"}

```

Note that - by default - Rally assumes your data file only contains data and generates the action and metadata line on the fly for you. If your data file already contains that line, then you can stop Rally from generating it, by setting `includes-action-and-meta-data` to `true` in your `indices` section. E.g.:

```auto
"indices": [
    {
      "name": "contact_current",
      "types": [
        {
          "name": "contact",
          "mapping": "contact-mapping.json",
          "documents": "documents.json.bz2",
          "includes-action-and-meta-data": true,
          "other properties here...": "..."
        }
      ]
    }
  ]

```

See the [track reference docs](http://esrally.readthedocs.io/en/stable/track.html#indices) for more details. Note that in that case the document count that you need to specify in the `indices` section is only half the number of lines of your file, because the other half are action and metadata lines. E.g. if your file has 1000 lines, the document count is 500 because 500 lines contain actual documents, the other 500 lines are action and metadata lines.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 9, 2018, 6:40am UTC](https://discuss.elastic.co/t/rally-error-while-running-against-custom-track-and-custom-data/114963/8 "2018-02-09T06:40:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.