How can i race 4 create-index operations in parallel?

Elastic Stack Elasticsearch
1

I am tring to run bencmark with custom track. The following is the custom track.json file. I get the following error when I run benchmark it with 25 clients in parallel 4 create-index operation.

How do I fix this?

Is it the same process of 4 parallel create-index operations created with 25 clients and a create-index operation created with 100 clients?

Error is :


[ERROR] Cannot race. Error in load generator [0]
        ("Request returned an error. Error type: transport, Description: resource_already_exists_exception ({'error': {'root_cause': [{'type': 'resource_alrea                                                dy_exists_exception', 'reason': 'index [logs111/FK3f6IPSS2uAwUVaridYxg] already exists', 'index_uuid': 'FK3f6IPSS2uAwUVaridYxg', 'index': 'logs111'}], '                                                type': 'resource_already_exists_exception', 'reason': 'index [logs111/FK3f6IPSS2uAwUVaridYxg] already exists', 'index_uuid': 'FK3f6IPSS2uAwUVaridYxg', 'ind                                                ex': 'logs111'}, 'status': 400})", None)

track.json

{
    "version": 2,
    "description": "Benchmark for Rally-logs",
    "indices": [
        {
            "name": "logs1",
            "body": "index.json",
            "types": [
                "logs_type"
            ]
        },
        {
            "name": "logs2",
            "body": "index.json",
            "types": [
                "logs_type"
            ]
        },
        {
            "name": "logs3",
            "body": "index.json",
            "types": [
                "logs_type"
            ]
        },
        {
            "name": "logs4",
            "body": "index.json",
            "types": [
                "logs_type"
            ]
        }
    ],
    "corpora": [
        {
            "name": "rally-logs",
            "documents": [
                {
                    "source-file": "documents.json",
                    "document-count": 2300000,
                    "uncompressed-bytes": 10878999999,
                    "target-index": "logs1",
                    "target-type":"logs_type"
                },
                {
                    "source-file": "documents.json",
                    "document-count": 2300000,
                    "uncompressed-bytes": 10878999999,
                    "target-index": "logs2",
                    "target-type":"logs_type"
                },
                {
                    "source-file": "documents.json",
                    "document-count": 2300000,
                    "uncompressed-bytes": 10878999999,
                    "target-index": "logs3",
                    "target-type":"logs_type"
                },
                {
                    "source-file": "documents.json",
                    "document-count": 2300000,
                    "uncompressed-bytes": 10878999999,
                    "target-index": "logs4",
                    "target-type":"logs_type"
                }
            ]
        }
    ],
    "schedule": [
        {
            "operation": {
                "operation-type": "delete-index",
                "only-if-exists": true,
                "index": "logs*"
            }
        },
        {
            "parallel": {
                "warmup-iterations": 50,
                "iterations": 100,
                "tasks": [
                    {
                        "name": "create-index-1",
                        "operation": "create-index",
                        "clients": 25,
                        "target-throughput": 2500,
                        "include-in-reporting": true,
                        "index": "logs1"
                    },
                    {
                        "name": "create-index-2",
                        "operation": "create-index",
                        "clients": 25,
                        "target-throughput": 2500,
                        "include-in-reporting": true,
                        "index": "logs2"
                    },
                    {
                        "name": "create-index-3",
                        "operation": "create-index",
                        "clients": 25,
                        "target-throughput": 2500,
                        "include-in-reporting": true,
                        "index": "logs3"
                    },
                    {
                        "name": "create-index-4",
                        "operation": "create-index",
                        "clients": 25,
                        "target-throughput": 2500,
                        "include-in-reporting": true,
                        "index": "logs4"
                    }
                ]
            }
        }
    ]
}
2

Hi,

this seems expected? You are attempting to create the same index 25 times concurrently and only the first attempt to create an index will succeed and for the other 24 you will get this error. You only need to create an index once and then you add data to it (usually by bulk-indexing documents). Why do you want to do this? Please also check our tutorial on how to create custom tracks in the docs.

Daniel

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.