Config elastic 2.2.0 with 5 nodes

Elastic Stack Elasticsearch
1

Hi all,

I'm new in elastic.
I thought number of master should be a quorum of data nodes ( so 3 master for 5 nodes)
In fact I have 5 nodes for an elastic cluster and only one master.
Is it a correct config ? May I have to add a 6th node as a load balancer to have several master ?

thanks all

2

You can have multiple master eligible nodes - which you want 3 of - but there is only ever one active master.

So as long as that is what you have set you will be ok

3

because we're facing performance problem with elastic, I'm wondering if it is a good case for an architecture like:
a load balancer
3 master
5 nodes
Is it complex to implement ?

4

No it is not.

5

3 dedicated master node may help with stability and resilience. It may or may not help with your performance problems, as that depends on what the source of this is. Do you have monitoring installed?

6

The monitoring is not working for a basic problem of a closed network port. I have around 4 applications on this cluster.
I'm going to push the demand on network team

7

What kind of performance problems are you having? If you can provide the output from the cluster stats API, it can give an indication of the health of the cluster.

8

I hope the stats are not too heavy :wink:
performance problem is from spark streaming point of view... an application is about 90 doc / second to feed elastic
each node is 1Tb data size

{
    "timestamp": 1506679598966,
    "cluster_name": "cluster-search",
    "status": "yellow",
    "indices":
    {
        "count": 77,
        "shards":
        {
            "total": 475,
            "primaries": 239,
            "replication": 0.9874476987447699,
            "index":
            {
                "shards":
                {
                    "min": 6,
                    "max": 10,
                    "avg": 6.1688311688311686
                },
                "primaries":
                {
                    "min": 3,
                    "max": 5,
                    "avg": 3.103896103896104
                },
                "replication":
                {
                    "min": 0.6,
                    "max": 1,
                    "avg": 0.9922077922077921
                }
            }
        },
        "docs":
        {
            "count": 720437714,
            "deleted": 238283836
        },
        "store":
        {
            "size_in_bytes": 643221726243,
            "throttle_time_in_millis": 0
        },
        "fielddata":
        {
            "memory_size_in_bytes": 0,
            "evictions": 0
        },
        "query_cache":
        {
            "memory_size_in_bytes": 17969448,
            "total_count": 61291197,
            "hit_count": 1452970,
            "miss_count": 59838227,
            "cache_size": 466,
            "cache_count": 466,
            "evictions": 0
        },
        "completion":
        {
            "size_in_bytes": 0
        },
        "segments":
        {
            "count": 4400,
            "memory_in_bytes": 1458472413,
            "terms_memory_in_bytes": 1227576573,
            "stored_fields_memory_in_bytes": 194672360,
            "term_vectors_memory_in_bytes": 0,
            "norms_memory_in_bytes": 2878592,
            "doc_values_memory_in_bytes": 33344888,
            "index_writer_memory_in_bytes": 0,
            "index_writer_max_memory_in_bytes": 33032695663,
            "version_map_memory_in_bytes": 0,
            "fixed_bit_set_memory_in_bytes": 183375808
        },
        "percolate":
        {
            "total": 0,
            "time_in_millis": 0,
            "current": 0,
            "memory_size_in_bytes": -1,
            "memory_size": "-1b",
            "queries": 0
        }
    },
    "nodes":
    {
        "count":
        {
            "total": 5,
            "master_only": 0,
            "data_only": 0,
            "master_data": 5,
            "client": 0
        },
        "versions":
        [
            "2.2.0"
        ],
        "os":
        {
            "available_processors": 30,
            "allocated_processors": 30,
            "mem":
            {
                "total_in_bytes": 0
            },
            "names":
            [
                {
                    "name": "Linux",
                    "count": 5
                }
            ]
        },
        "process":
        {
            "cpu":
            {
                "percent": 2
            },
            "open_file_descriptors":
            {
                "min": 379,
                "max": 2547,
                "avg": 1636
            }
        },
        "jvm":
        {
            "max_uptime_in_millis": 85858165,
            "versions":
            [
                {
                    "version": "1.7.0_79",
                    "vm_name": "Java HotSpot(TM) 64-Bit Server VM",
                    "vm_version": "24.79-b02",
                    "vm_vendor": "Oracle Corporation",
                    "count": 5
                }
            ],
            "mem":
            {
                "heap_used_in_bytes": 20736842272,
                "heap_max_in_bytes": 53458632704
            },
            "threads": 426
        },
        "fs":
        {
            "total_in_bytes": 5494842347520,
            "free_in_bytes": 4851197722624,
            "available_in_bytes": 4851197722624,
            "spins": "true"
        },
        "plugins":
        [
        ]
    }
}
9

It looks like you have an update heavy workload and that a lot of heap is used by the index writers. Do you have any non-standard settings on these nodes? How many of these indices are you concurrently writing into?

Are the performance problems related to indexing speed or querying?

10

it is speed indexing... there is a few queries actually.
The config is very basic
Indices are partitionned; they are about 5 in use
I don't know how to read these stats beause you have to divide all by 5.
Nodes are 32GB ram and 60% for elastic (little more than 16GB)

11

How are you indexing? Are you using bulk requests? If so, how large are they? What indexing/update rate are you seeing? How many fields do you have in your mappings? Are the mappings static or dynamic?

12

I have 4 applications writing in spark streaming scala. I think it is a java call to a lib called samuel.
It sends a json (simplified compare to the original message).
Only one appli is using a static mapping with 50 fields
it is the whole applications which are slows

13

So most of the applications generate fields dynamically? What is the size of the mappings for those indices? How large are the documents in terms of size and number of fields?

14

the largest one is around 50 fields (all not analysed type) and the others are around 20 fields.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.