# Rally \_bulk error 400 "type is missing"

**URL:** <https://discuss.elastic.co/t/rally-bulk-error-400-type-is-missing/283073>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [September 1, 2021, 3:59pm UTC](https://discuss.elastic.co/t/rally-bulk-error-400-type-is-missing/283073 "2021-09-01T15:59:29Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\
**Post date:** [September 1, 2021, 3:59pm UTC](https://discuss.elastic.co/t/rally-bulk-error-400-type-is-missing/283073/1 "2021-09-01T15:59:29Z")

</div>

Hey there,  
I'm trying to import a custom track exported from an old ES version (still using 'type' field) into another ES cluster (same version).  
This is the error I faced:

> [ERROR] Cannot race. Error in load generator [0]  
> ("Request returned an error. Error type: transport, Description: action\_request\_validation\_exception ({'error': {'root\_cause': [{'type': 'action\_request\_validation\_exception', 'reason': 'Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;22: type is missing;23: type is missing;24: type is missing;25: type is missing;26: type is missing;27: type is missing;28: type is missing;29: type is missing;30: type is missing;31: type is missing;32: type is missing;33: type is missing;34: type is missing;35: type is missing;36: type is missing;37: type is missing;38: type is missing;39: type is missing;40: type is missing;41: type is missing;42: type is missing;43: type is missing;44: type is missing;45: type is missing;46: type is missing;47: type is missing;48: type is missing;49: type is missing;50: type is missing;51: type is missing;52: type is missing;53: type is missing;54: type is missing;55: type is missing;56: type is missing;57: type is missing;58: type is missing;59: type is missing;60: type is missing;61: type is missing;62: type is missing;63: type is missing;64: type is missing;65: type is missing;66: type is missing;67: type is missing;68: type is missing;69: type is missing;70: type is missing;71: type is missing;72: type is missing;73: type is missing;74: type is missing;75: type is missing;76: type is missing;77: type is missing;78: type is missing;79: type is missing;80: type is missing;81: type is missing;82: type is missing;83: type is missing;84: type is missing;85: type is missing;86: type is missing;87: type is missing;88: type is missing;89: type is missing;90: type is missing;91: type is missing;92: type is missing;93: type is missing;94: type is missing;95: type is missing;96: type is missing;97: type is missing;98: type is missing;99: type is missing;100: type is missing;101: type is missing;102: type is missing;103: type is missing;104: type is missing;105: type is missing;106: type is missing;107: type is missing;108: type is missing;109: type is missing;110: type is missing;111: type is missing;112: type is missing;113: type is missing;114: type is missing;115: type is missing;116: type is missing;117: type is missing;118: type is missing;119: type is missing;120: type is missing;121: type is missing;122: type is missing;123: type is missing;124: type is missing;125: type is missing;'}], 'type': 'action\_request\_validation\_exception', 'reason': 'Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;22: type is missing;23: type is missing;24: type is missing;25: type is missing;26: type is missing;27: type is missing;28: type is missing;29: type is missing;30: type is missing;31: type is missing;32: type is missing;33: type is missing;34: type is missing;35: type is missing;36: type is missing;37: type is missing;38: type is missing;39: type is missing;40: type is missing;41: type is missing;42: type is missing;43: type is missing;44: type is missing;45: type is missing;46: type is missing;47: type is missing;48: type is missing;49: type is missing;50: type is missing;51: type is missing;52: type is missing;53: type is missing;54: type is missing;55: type is missing;56: type is missing;57: type is missing;58: type is missing;59: type is missing;60: type is missing;61: type is missing;62: type is missing;63: type is missing;64: type is missing;65: type is missing;66: type is missing;67: type is missing;68: type is missing;69: type is missing;70: type is missing;71: type is missing;72: type is missing;73: type is missing;74: type is missing;75: type is missing;76: type is missing;77: type is missing;78: type is missing;79: type is missing;80: type is missing;81: type is missing;82: type is missing;83: type is missing;84: type is missing;85: type is missing;86: type is missing;87: type is missing;88: type is missing;89: type is missing;90: type is missing;91: type is missing;92: type is missing;93: type is missing;94: type is missing;95: type is missing;96: type is missing;97: type is missing;98: type is missing;99: type is missing;100: type is missing;101: type is missing;102: type is missing;103: type is missing;104: type is missing;105: type is missing;106: type is missing;107: type is missing;108: type is missing;109: type is missing;110: type is missing;111: type is missing;112: type is missing;113: type is missing;114: type is missing;115: type is missing;116: type is missing;117: type is missing;118: type is missing;119: type is missing;120: type is missing;121: type is missing;122: type is missing;123: type is missing;124: type is missing;125: type is missing;'}, 'status': 400})", None)

Is there anyone can help me? Can I set anything in the bulk operation?  
I am using a standard configuration:

```auto
 {
      "operation": {
        "operation-type": "bulk",
        "bulk-size": {{bulk_size | default(5000)}},
        "ingest-percentage": {{ingest_percentage | default(100)}}
      },
      "clients": {{bulk_indexing_clients | default(8)}}
    }

```

---

<div class="post-metadata">

**Author:** ![RickBoyd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rickboyd/32/80022_2.png) [@RickBoyd](https://discuss.elastic.co/u/RickBoyd)\
**Post date:** [September 1, 2021, 4:10pm UTC](https://discuss.elastic.co/t/rally-bulk-error-400-type-is-missing/283073/2 "2021-09-01T16:10:48Z")

</div>

Hi @rschirin,

What versions of ES and Rally are you using? Also, is the failure during the bulk phase or doing a create-index phase?

---

<div class="post-metadata">

**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\
**Post date:** [September 1, 2021, 4:15pm UTC](https://discuss.elastic.co/t/rally-bulk-error-400-type-is-missing/283073/3 "2021-09-01T16:15:38Z")

</div>

I used Rally 2.2.1 with ES 7.x (src to create custom track and also dst) and it works.  
Then I used Rally 2.2.1 with ES 2.4 to create the custom track and then I used Rally 1.4.1 to test race on ES 2.4 and I faced that error during the bulk phase. Index is correctly deleted and created.

I know that ES 2.x is out of support/life/everything but I have to perform again some tests. 😭 😭

---

<div class="post-metadata">

**Author:** ![RickBoyd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rickboyd/32/80022_2.png) [@RickBoyd](https://discuss.elastic.co/u/RickBoyd)\
**Post date:** [September 1, 2021, 4:29pm UTC](https://discuss.elastic.co/t/rally-bulk-error-400-type-is-missing/283073/4 "2021-09-01T16:29:00Z")

</div>

I can't guarantee anything as we are talking about very old versions, and the `create-track` functionality was introduced much later than ES 2.x, but my suggestion is to try to use Rally 2.0.2 to create the track as it was the last version to support benchmarking 2.x clusters. If this doesn't work, though, I'm afraid support options would be limited to help generate the track for you for that old of an ES cluster

---

<div class="post-metadata">

**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\
**Post date:** [September 1, 2021, 5:08pm UTC](https://discuss.elastic.co/t/rally-bulk-error-400-type-is-missing/283073/5 "2021-09-01T17:08:02Z")

</div>

using Rally 2.0.2 it fails with the same result.  
Probably I should try to create the custom-track with the same Rally version that I will use to import data.  
Unfortunately, trying to create custom track with Rally:2.0.2 fails with the following error:

> Extracting documents for index [indexname] for test mode... 1000/1000 docs [100.0% done][ERROR] Cannot create-track. 'skipped'.

and then

> 2021-09-01 17:05:58,311 -not-actor-/PID:1 esrally.rally ERROR A fatal error occurred while running subcommand [create-track].  
> Traceback (most recent call last):  
> File "/usr/local/lib/python3.8/site-packages/esrally/rally.py", line 774, in dispatch\_sub\_command  
> tracker.create\_track(cfg)  
> File "/usr/local/lib/python3.8/site-packages/esrally/tracker/tracker.py", line 78, in create\_track  
> indices, corpora = extract\_mappings\_and\_corpora(client, output\_path, indices)  
> File "/usr/local/lib/python3.8/site-packages/esrally/tracker/tracker.py", line 51, in extract\_mappings\_and\_corpora  
> c = corpus.extract(client, output\_path, i["name"])  
> File "/usr/local/lib/python3.8/site-packages/esrally/tracker/corpus.py", line 62, in extract  
> dump\_documents(client, index, get\_doc\_outpath(output\_path, index, "-1k"), min(total\_docs, 1000), " for test mode")  
> File "/usr/local/lib/python3.8/site-packages/esrally/tracker/corpus.py", line 83, in dump\_documents  
> for n, doc in enumerate(helpers.scan(client, query=query, index=index)):  
> File "/usr/local/lib/python3.8/site-packages/elasticsearch/helpers/actions.py", line 518, in scan  
> if (resp["\_shards"]["successful"] + resp["\_shards"]["skipped"]) \< resp[  
> KeyError: 'skipped'

could be it related to docker image?  
on ES 2.4 is there a way to create a custom track?  
🙄

---

<div class="post-metadata">

**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\
**Post date:** [September 1, 2021, 6:49pm UTC](https://discuss.elastic.co/t/rally-bulk-error-400-type-is-missing/283073/6 "2021-09-01T18:49:23Z")

</div>

Ok, that issue is not related to docker image or Python Rally module since I have tested also the latter.  
I found also this git issue opened [here](https://github.com/elastic/elasticsearch-py/issues/1450).  
Currently I have fixed the problem bypassing the check in the file `/usr/local/lib/python3.8/site-packages/elasticsearch/helpers/actions.py` starting from line 518

```auto
#if (resp["_shards"]["successful"] + resp["_shards"]["skipped"]) < resp[
    519 # "_shards"
    520 #]["total"]:
    521 # logger.warning(
    522 # "Scroll request has only succeeded on %d (+%d skipped) shards out of %d.",
    523 # resp["_shards"]["successful"],
    524 # resp["_shards"]["skipped"],
    525 # resp["_shards"]["total"],
    526 # )
    527 # if raise_on_error:
    528 # raise ScanError(
    529 # scroll_id,
    530 # "Scroll request has only succeeded on %d (+%d skiped) shards out of %d."
    531 # % (
    532 # resp["_shards"]["successful"],
    533 # resp["_shards"]["skipped"],
    534 # resp["_shards"]["total"],
    535 # ),
    536 # )

```

I know that it is not a good way to work, probably "skipped" field is not existing in the old ES version, is it?

---

<div class="post-metadata">

**Author:** ![rschirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rschirin/32/45283_2.png) [@rschirin](https://discuss.elastic.co/u/rschirin)\
**Post date:** [September 2, 2021, 12:39pm UTC](https://discuss.elastic.co/t/rally-bulk-error-400-type-is-missing/283073/7 "2021-09-02T12:39:31Z")

</div>

regarding the first issue, thanks to [this post](https://discuss.elastic.co/t/run-custom-tracks-get-type-error/271520/2) I found the solution.  
It was related to missing `"target-type": "doc",` in the `corpora` track section

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 30, 2021, 12:40pm UTC](https://discuss.elastic.co/t/rally-bulk-error-400-type-is-missing/283073/8 "2021-09-30T12:40:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.