Rally _bulk error 400 "type is missing"

Elastic Stack Elasticsearch
1

Hey there,
I'm trying to import a custom track exported from an old ES version (still using 'type' field) into another ES cluster (same version).
This is the error I faced:

[ERROR] Cannot race. Error in load generator [0]
("Request returned an error. Error type: transport, Description: action_request_validation_exception ({'error': {'root_cause': [{'type': 'action_request_validation_exception', 'reason': 'Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;22: type is missing;23: type is missing;24: type is missing;25: type is missing;26: type is missing;27: type is missing;28: type is missing;29: type is missing;30: type is missing;31: type is missing;32: type is missing;33: type is missing;34: type is missing;35: type is missing;36: type is missing;37: type is missing;38: type is missing;39: type is missing;40: type is missing;41: type is missing;42: type is missing;43: type is missing;44: type is missing;45: type is missing;46: type is missing;47: type is missing;48: type is missing;49: type is missing;50: type is missing;51: type is missing;52: type is missing;53: type is missing;54: type is missing;55: type is missing;56: type is missing;57: type is missing;58: type is missing;59: type is missing;60: type is missing;61: type is missing;62: type is missing;63: type is missing;64: type is missing;65: type is missing;66: type is missing;67: type is missing;68: type is missing;69: type is missing;70: type is missing;71: type is missing;72: type is missing;73: type is missing;74: type is missing;75: type is missing;76: type is missing;77: type is missing;78: type is missing;79: type is missing;80: type is missing;81: type is missing;82: type is missing;83: type is missing;84: type is missing;85: type is missing;86: type is missing;87: type is missing;88: type is missing;89: type is missing;90: type is missing;91: type is missing;92: type is missing;93: type is missing;94: type is missing;95: type is missing;96: type is missing;97: type is missing;98: type is missing;99: type is missing;100: type is missing;101: type is missing;102: type is missing;103: type is missing;104: type is missing;105: type is missing;106: type is missing;107: type is missing;108: type is missing;109: type is missing;110: type is missing;111: type is missing;112: type is missing;113: type is missing;114: type is missing;115: type is missing;116: type is missing;117: type is missing;118: type is missing;119: type is missing;120: type is missing;121: type is missing;122: type is missing;123: type is missing;124: type is missing;125: type is missing;'}], 'type': 'action_request_validation_exception', 'reason': 'Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;22: type is missing;23: type is missing;24: type is missing;25: type is missing;26: type is missing;27: type is missing;28: type is missing;29: type is missing;30: type is missing;31: type is missing;32: type is missing;33: type is missing;34: type is missing;35: type is missing;36: type is missing;37: type is missing;38: type is missing;39: type is missing;40: type is missing;41: type is missing;42: type is missing;43: type is missing;44: type is missing;45: type is missing;46: type is missing;47: type is missing;48: type is missing;49: type is missing;50: type is missing;51: type is missing;52: type is missing;53: type is missing;54: type is missing;55: type is missing;56: type is missing;57: type is missing;58: type is missing;59: type is missing;60: type is missing;61: type is missing;62: type is missing;63: type is missing;64: type is missing;65: type is missing;66: type is missing;67: type is missing;68: type is missing;69: type is missing;70: type is missing;71: type is missing;72: type is missing;73: type is missing;74: type is missing;75: type is missing;76: type is missing;77: type is missing;78: type is missing;79: type is missing;80: type is missing;81: type is missing;82: type is missing;83: type is missing;84: type is missing;85: type is missing;86: type is missing;87: type is missing;88: type is missing;89: type is missing;90: type is missing;91: type is missing;92: type is missing;93: type is missing;94: type is missing;95: type is missing;96: type is missing;97: type is missing;98: type is missing;99: type is missing;100: type is missing;101: type is missing;102: type is missing;103: type is missing;104: type is missing;105: type is missing;106: type is missing;107: type is missing;108: type is missing;109: type is missing;110: type is missing;111: type is missing;112: type is missing;113: type is missing;114: type is missing;115: type is missing;116: type is missing;117: type is missing;118: type is missing;119: type is missing;120: type is missing;121: type is missing;122: type is missing;123: type is missing;124: type is missing;125: type is missing;'}, 'status': 400})", None)

Is there anyone can help me? Can I set anything in the bulk operation?
I am using a standard configuration:

 {
      "operation": {
        "operation-type": "bulk",
        "bulk-size": {{bulk_size | default(5000)}},
        "ingest-percentage": {{ingest_percentage | default(100)}}
      },
      "clients": {{bulk_indexing_clients | default(8)}}
    }
2

Hi @rschirin,

What versions of ES and Rally are you using? Also, is the failure during the bulk phase or doing a create-index phase?

3

I used Rally 2.2.1 with ES 7.x (src to create custom track and also dst) and it works.
Then I used Rally 2.2.1 with ES 2.4 to create the custom track and then I used Rally 1.4.1 to test race on ES 2.4 and I faced that error during the bulk phase. Index is correctly deleted and created.

I know that ES 2.x is out of support/life/everything but I have to perform again some tests. :sob: :sob:

4

I can't guarantee anything as we are talking about very old versions, and the create-track functionality was introduced much later than ES 2.x, but my suggestion is to try to use Rally 2.0.2 to create the track as it was the last version to support benchmarking 2.x clusters. If this doesn't work, though, I'm afraid support options would be limited to help generate the track for you for that old of an ES cluster

5

using Rally 2.0.2 it fails with the same result.
Probably I should try to create the custom-track with the same Rally version that I will use to import data.
Unfortunately, trying to create custom track with Rally:2.0.2 fails with the following error:

Extracting documents for index [indexname] for test mode... 1000/1000 docs [100.0% done][ERROR] Cannot create-track. 'skipped'.

and then

2021-09-01 17:05:58,311 -not-actor-/PID:1 esrally.rally ERROR A fatal error occurred while running subcommand [create-track].
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/esrally/rally.py", line 774, in dispatch_sub_command
tracker.create_track(cfg)
File "/usr/local/lib/python3.8/site-packages/esrally/tracker/tracker.py", line 78, in create_track
indices, corpora = extract_mappings_and_corpora(client, output_path, indices)
File "/usr/local/lib/python3.8/site-packages/esrally/tracker/tracker.py", line 51, in extract_mappings_and_corpora
c = corpus.extract(client, output_path, i["name"])
File "/usr/local/lib/python3.8/site-packages/esrally/tracker/corpus.py", line 62, in extract
dump_documents(client, index, get_doc_outpath(output_path, index, "-1k"), min(total_docs, 1000), " for test mode")
File "/usr/local/lib/python3.8/site-packages/esrally/tracker/corpus.py", line 83, in dump_documents
for n, doc in enumerate(helpers.scan(client, query=query, index=index)):
File "/usr/local/lib/python3.8/site-packages/elasticsearch/helpers/actions.py", line 518, in scan
if (resp["_shards"]["successful"] + resp["_shards"]["skipped"]) < resp[
KeyError: 'skipped'

could be it related to docker image?
on ES 2.4 is there a way to create a custom track?
:roll_eyes:

6

Ok, that issue is not related to docker image or Python Rally module since I have tested also the latter.
I found also this git issue opened here.
Currently I have fixed the problem bypassing the check in the file /usr/local/lib/python3.8/site-packages/elasticsearch/helpers/actions.py starting from line 518

#if (resp["_shards"]["successful"] + resp["_shards"]["skipped"]) < resp[
    519              #   "_shards"
    520             #]["total"]:
    521             #    logger.warning(
    522             #        "Scroll request has only succeeded on %d (+%d skipped) shards out of %d.",
    523             #        resp["_shards"]["successful"],
    524             #        resp["_shards"]["skipped"],
    525             #        resp["_shards"]["total"],
    526             #    )
    527             #    if raise_on_error:
    528             #        raise ScanError(
    529             #            scroll_id,
    530             #            "Scroll request has only succeeded on %d (+%d skiped) shards out of %d."
    531             #            % (
    532             #                resp["_shards"]["successful"],
    533             #                resp["_shards"]["skipped"],
    534             #                resp["_shards"]["total"],
    535             #            ),
    536             #        )

I know that it is not a good way to work, probably "skipped" field is not existing in the old ES version, is it?

7

regarding the first issue, thanks to this post I found the solution.
It was related to missing "target-type": "doc", in the corpora track section

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.