Rally error while running against custom track and custom data

Elastic Stack Elasticsearch
1

INFO] Writing logs to /Users/.rally/logs/rally_out_20180110T211006Z.log
[INFO] Preparing for race ...
[INFO] Rally will delete the benchmark candidate after the benchmark
[INFO] Racing on track [contactdata], challenge [createcontactindex] and car ['defaults'] with version [7.0.0-alpha1].

[INFO] Preparing file offset table for [/Users/.rally/benchmarks/data/contactdata/contactdata.json] ... [OK]

[ERROR] Cannot race. ('Could not execute benchmark', RequestError(400, 'mapper_parsing_exception', {'error': {'root_cause': [{'type': 'mapper_parsing_exception', 'reason': 'normalizer [lowercase_normalizer] not found for field [country]'}], 'type': 'mapper_parsing_exception', 'reason': 'Failed to parse mapping [contact]: normalizer [lowercase_normalizer] not found for field [country]', 'caused_by': {'type': 'mapper_parsing_exception', 'reason': 'normalizer [lowercase_normalizer] not found for field [country]'}}, 'status': 400}))

The error logs only say this:

2018-01-10 21:11:37,510 PID:21118 rally.driver ERROR Main driver encountered a fatal exception. Shutting down.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/esrally/driver/driver.py", line 160, in receiveMessage
expected_status=None, new_status=None, transition=self.after_track_prepared)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/esrally/actor.py", line 90, in transition_when_all_children_responded
transition()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/esrally/driver/driver.py", line 257, in after_track_prepared
self.coordinator.after_track_prepared()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/esrally/driver/driver.py", line 364, in after_track_prepared
self.prepare_cluster()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/esrally/driver/driver.py", line 475, in prepare_cluster
setup_index(es, index, self.challenge.index_settings)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/esrally/driver/driver.py", line 844, in setup_index
es.indices.create(index=index.name, body=body)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 73, in _wrapped
return func(*args, params=params, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/elasticsearch/client/indices.py", line 107, in create
params=params, body=body)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/elasticsearch/transport.py", line 312, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 128, in perform_request
self._raise_error(response.status, raw_data)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 125, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, 'mapper_parsing_exception', 'normalizer [lowercase_normalizer] not found for field [country]')

==========================

Could anyone please guide whether it is a problem with the test data or the mapping.json for index creation is wrong?

2

This is a problem with the mapping file. It say that it cannot find the lowercase_normalizer that you have defined for the field country.

You should be able to reproduce this by issuing the following curl statements:

# create an empty index
curl -XPUT 'localhost:9200/name_of_your_index_here?pretty' -H 'Content-Type: application/json' -d'
{}
'

# issue the PUT mapping call
curl -XPUT 'localhost:9200/name_of_your_index_here/_mapping/contact?pretty' -H 'Content-Type: application/json' -d @mapping.json
3

Hello Daniel,

My mapping file looks like this:

{
"settings": {
"analysis": {
"normalizer": {
"lowercase_normalizer": {
"type": "custom",
"char_filter": [],
"filter": ["lowercase"]
}
}
},
"index": {
"number_of_shards": 5,
"number_of_replicas": 2
},
"index.requests.cache.enable": true
},
"mappings": {
"contact": {
"dynamic": "false",
"_routing": {
"required": true
},
"properties": {
"type": {
"type": "keyword",
"normalizer": "lowercase_normalizer"
},
"joinField": {
"type": "join",
"relations": {
"contact": "location"
}
},
"country": {
"type": "keyword",
"normalizer": "lowercase_normalizer"
}
}
}
}
}

Rally throws the below error, when i try to use it. However it works fine when i use this mapping file directly against ES.

[INFO] Writing logs to /Users/dipanjan.haldar/.rally/logs/rally_out_20180111T154711Z.log
[INFO] Preparing for race ...
[INFO] Rally will delete the benchmark candidate after the benchmark
[INFO] Racing on track [contactdata], challenge [createcontactindex] and car ['defaults'] with version [6.1.1].

[ERROR] Cannot race. ('Could not execute benchmark', RequestError(400, 'mapper_parsing_exception', {'error': {'root_cause': [{'type': 'mapper_parsing_exception', 'reason': 'Root mapping definition has unsupported parameters: [index.requests.cache.enable : true] [index : {number_of_shards=5, number_of_replicas=2}] [analysis : {normalizer={lowercase_normalizer={filter=[lowercase], char_filter=[], type=custom}}}]'}], 'type': 'mapper_parsing_exception', 'reason': 'Failed to parse mapping [settings]: Root mapping definition has unsupported parameters: [index.requests.cache.enable : true] [index : {number_of_shards=5, number_of_replicas=2}] [analysis : {normalizer={lowercase_normalizer={filter=[lowercase], char_filter=[], type=custom}}}]', 'caused_by': {'type': 'mapper_parsing_exception', 'reason': 'Root mapping definition has unsupported parameters: [index.requests.cache.enable : true] [index : {number_of_shards=5, number_of_replicas=2}] [analysis : {normalizer={lowercase_normalizer={filter=[lowercase], char_filter=[], type=custom}}}]'}}, 'status': 400}))

4

Hi Dipanjan,

I got the problem now. Unfortunately, Rally 0.8.1 does not allow you do combine settings and mappings into one file. This will be fixed in the upcoming release Rally 0.9.0 but to make it work with Rally 0.8.1, you need to use a somewhat akward syntax

  1. In your challenge, you need to define an index-settings property (also see the docs):
  "index-settings": {
    "analysis": {
      "normalizer": {
        "lowercase_normalizer": {
          "type": "custom",
          "char_filter": [],
          "filter": ["lowercase"]
        }
      }
    },
    "index": {
      "number_of_shards": 5,
      "number_of_replicas": 2
    },
    "index.requests.cache.enable": true    
  }

Then you need to store your mappings for the contact type in a mapping file, let's call it contact-mapping.json. The contents of the file are as follows:

{
  "contact": {
    "dynamic": "false",
    "_routing": {
      "required": true
    },
    "properties": {
      "type": {
        "type": "keyword",
        "normalizer": "lowercase_normalizer"
      },
      "joinField": {
        "type": "join",
        "relations": {
          "contact": "location"
        }
      },
      "country": {
        "type": "keyword",
        "normalizer": "lowercase_normalizer"
      }
    }
  }
}

Then you need to reference this file in your index definition. E.g.:

  "indices": [
    {
      "name": "my-index-name",
      "types": [
        {
          "name": "contact",
          "mapping": "contact-mapping.json",
          "other properties here...": "..."
        }
      ]
    }
  ]

That should work. And again: With 0.9.0 things will work much more straightforward and you can just provide one file for the whole index body (as you already tried).

5

Thanks Daniel, the issue is resolved and I am able to run the test

6

Hi Daniel,

If I use a datafile with the following content, rally doesnt upload the data to ES. However this works fine when used directly against ES.

{"index": {"_type": "contact", "_id": 1122333333, "_index": "contact_current"}}
{ "firstName": "Bob", "organizationId": 1, "lastName": "test", "middleInitial": "test", "createdDate": "2016-07-15T00:19:02.940Z", "lastModifiedDate": "2017-02-10T19:01:43.580Z", "type": "contact", "joinField": "contact"}

7

Hi Dipanjan,

how does Rally behave exactly? Your data file contains:

An action and metadata line which is:

{"index": {"_type": "contact", "_id": 1122333333, "_index": "contact_current"}}

and the actual data:

{ "firstName": "Bob", "organizationId": 1, "lastName": "test", "middleInitial": "test", "createdDate": "2016-07-15T00:19:02.940Z", "lastModifiedDate": "2017-02-10T19:01:43.580Z", "type": "contact", "joinField": "contact"}

Note that - by default - Rally assumes your data file only contains data and generates the action and metadata line on the fly for you. If your data file already contains that line, then you can stop Rally from generating it, by setting includes-action-and-meta-data to true in your indices section. E.g.:

"indices": [
    {
      "name": "contact_current",
      "types": [
        {
          "name": "contact",
          "mapping": "contact-mapping.json",
          "documents": "documents.json.bz2",
          "includes-action-and-meta-data": true,
          "other properties here...": "..."
        }
      ]
    }
  ]

See the track reference docs for more details. Note that in that case the document count that you need to specify in the indices section is only half the number of lines of your file, because the other half are action and metadata lines. E.g. if your file has 1000 lines, the document count is 500 because 500 lines contain actual documents, the other 500 lines are action and metadata lines.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.