Elasticsearch index

Elastic Stack Elasticsearch
21

Pl find below output.

1.)curl xx.x.xxx.aa:9200/_cat/health
646382008 13:50:08 apiman red 2 2 163 82 0 1 8 0

2.)curl 'xx.x.xxx.aa:9200/_cat/health?v&ts=0'
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks
1646382084 13:51:24 apiman red 2 2 163 82 0 1 8 0

3.)curl xx.x.xxx.aa:9200:9200/_cat/shards

test           4 p STARTED              0    144b xx.x.xxx.bb Sinister
test           4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
test           0 p STARTED              0    144b xx.x.xxx.bb Sinister
test           0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
test           3 p STARTED              0    144b xx.x.xxx.bb Sinister
test           3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
test           1 p STARTED              0    144b xx.x.xxx.bb Sinister
test           1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
test           2 p STARTED              0    144b xx.x.xxx.bb Sinister
test           2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
srrvas1.html   2 p STARTED              0    144b xx.x.xxx.bb Sinister
srrvas1.html   2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
srrvas1.html   0 p STARTED              0    144b xx.x.xxx.bb Sinister
srrvas1.html   0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
srrvas1.html   3 p STARTED              0    144b xx.x.xxx.bb Sinister
srrvas1.html   3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
srrvas1.html   1 p STARTED              0    144b xx.x.xxx.bb Sinister
srrvas1.html   1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
srrvas1.html   4 p STARTED              0    144b xx.x.xxx.bb Sinister
srrvas1.html   4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
servlet        4 p STARTED              0    144b xx.x.xxx.bb Sinister
servlet        4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
servlet        0 p STARTED              0    144b xx.x.xxx.bb Sinister
servlet        0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
servlet        3 p STARTED              0    144b xx.x.xxx.bb Sinister
servlet        3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
servlet        1 p STARTED              0    144b xx.x.xxx.bb Sinister
servlet        1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
servlet        2 p STARTED              0    144b xx.x.xxx.bb Sinister
servlet        2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
spipe          4 p STARTED              0    144b xx.x.xxx.bb Sinister
spipe          4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
spipe          0 p STARTED              0    144b xx.x.xxx.bb Sinister
spipe          0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
spipe          3 p STARTED              0    144b xx.x.xxx.bb Sinister
spipe          3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
spipe          1 p STARTED              0    144b xx.x.xxx.bb Sinister
spipe          1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
spipe          2 p STARTED              0    144b xx.x.xxx.bb Sinister
spipe          2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
blazeds        4 p STARTED              0    144b xx.x.xxx.bb Sinister
blazeds        4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
blazeds        0 p STARTED              0    144b xx.x.xxx.bb Sinister
blazeds        0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
blazeds        3 p STARTED              0    144b xx.x.xxx.bb Sinister
blazeds        3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
blazeds        1 p STARTED              0    144b xx.x.xxx.bb Sinister
blazeds        1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
blazeds        2 p STARTED              0    144b xx.x.xxx.bb Sinister
blazeds        2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
messagebroker  4 p STARTED              0    144b xx.x.xxx.bb Sinister
messagebroker  4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
messagebroker  0 p STARTED              0    144b xx.x.xxx.bb Sinister
messagebroker  0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
messagebroker  3 p STARTED              0    144b xx.x.xxx.bb Sinister
messagebroker  3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
messagebroker  1 p STARTED              0    144b xx.x.xxx.bb Sinister
messagebroker  1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
messagebroker  2 p STARTED              0    144b xx.x.xxx.bb Sinister
messagebroker  2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
phppath        4 p STARTED              0    144b xx.x.xxx.bb Sinister
phppath        4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
phppath        0 p STARTED              0    144b xx.x.xxx.bb Sinister
phppath        0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
phppath        3 p STARTED              0    144b xx.x.xxx.bb Sinister
phppath        3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
phppath        1 p STARTED              0    144b xx.x.xxx.bb Sinister
phppath        1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
phppath        2 p STARTED              0    144b xx.x.xxx.bb Sinister
phppath        2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
flex2gateway   2 p STARTED              0    144b xx.x.xxx.bb Sinister
flex2gateway   2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
flex2gateway   0 p STARTED              0    144b xx.x.xxx.bb Sinister
flex2gateway   0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
flex2gateway   3 p STARTED              0    144b xx.x.xxx.bb Sinister
flex2gateway   3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
flex2gateway   1 p STARTED              0    144b xx.x.xxx.bb Sinister
flex2gateway   1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
flex2gateway   4 p STARTED              0    144b xx.x.xxx.bb Sinister
flex2gateway   4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
.kibana        0 p STARTED             11  29.6kb xx.x.xxx.bb Sinister
.kibana        0 r STARTED             11  29.6kb xx.x.xxx.aa Jackson Arvad
lcds           4 p STARTED              0    144b xx.x.xxx.bb Sinister
lcds           4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
lcds           0 p STARTED              0    144b xx.x.xxx.bb Sinister
lcds           0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
lcds           3 p STARTED              0    144b xx.x.xxx.bb Sinister
lcds           3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
lcds           1 p STARTED              0    144b xx.x.xxx.bb Sinister
lcds           1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
lcds           2 p STARTED              0    144b xx.x.xxx.bb Sinister
lcds           2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
webui          4 p STARTED              0    144b xx.x.xxx.bb Sinister
webui          4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
webui          0 p STARTED              0    144b xx.x.xxx.bb Sinister
webui          0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
webui          3 p STARTED              0    144b xx.x.xxx.bb Sinister
webui          3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
webui          1 p STARTED              0    144b xx.x.xxx.bb Sinister
webui          1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
webui          2 p STARTED              0    144b xx.x.xxx.bb Sinister
webui          2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
cgi-bin        2 p STARTED              0    144b xx.x.xxx.bb Sinister
cgi-bin        2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
cgi-bin        0 p STARTED              0    144b xx.x.xxx.bb Sinister
cgi-bin        0 r STARTED              0    144b xx.x.xxx.aa Jackson Arvad
cgi-bin        3 p STARTED              0    144b xx.x.xxx.bb Sinister
cgi-bin        3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
cgi-bin        1 p STARTED              0    144b xx.x.xxx.bb Sinister
cgi-bin        1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
cgi-bin        4 p STARTED              0    144b xx.x.xxx.bb Sinister
cgi-bin        4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
apiman_metrics 2 p UNASSIGNED
apiman_metrics 2 r UNASSIGNED
apiman_metrics 0 p STARTED      340611781  54.7gb xx.x.xxx.bb Sinister
apiman_metrics 0 r INITIALIZING                   xx.x.xxx.aa Jackson Arvad
apiman_metrics 3 p UNASSIGNED
apiman_metrics 3 r UNASSIGNED
apiman_metrics 1 p UNASSIGNED
apiman_metrics 1 r UNASSIGNED
apiman_metrics 4 p UNASSIGNED
apiman_metrics 4 r UNASSIGNED
perl           4 p STARTED              0    144b xx.x.xxx.bb Sinister
perl           4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
perl           0 p STARTED              0    144b xx.x.xxx.bb Sinister
perl           0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
perl           3 p STARTED              0    144b xx.x.xxx.bb Sinister
perl           3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
perl           1 p STARTED              0    144b xx.x.xxx.bb Sinister
perl           1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
perl           2 p STARTED              0    144b xx.x.xxx.bb Sinister
perl           2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmillcl.exe  2 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmillcl.exe  2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmillcl.exe  0 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmillcl.exe  0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmillcl.exe  3 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmillcl.exe  3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmillcl.exe  1 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmillcl.exe  1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmillcl.exe  4 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmillcl.exe  4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
apiman_gateway 4 p STARTED            155 226.7kb xx.x.xxx.bb Sinister
apiman_gateway 4 r STARTED            155 227.2kb xx.x.xxx.aa Jackson Arvad
apiman_gateway 0 p STARTED            151 209.9kb xx.x.xxx.bb Sinister
apiman_gateway 0 r STARTED            151 210.2kb xx.x.xxx.aa Jackson Arvad
apiman_gateway 3 p STARTED            198 245.1kb xx.x.xxx.bb Sinister
apiman_gateway 3 r STARTED            198 239.6kb xx.x.xxx.aa Jackson Arvad
apiman_gateway 1 p STARTED            174 220.4kb xx.x.xxx.bb Sinister
apiman_gateway 1 r STARTED            174 219.6kb xx.x.xxx.aa Jackson Arvad
apiman_gateway 2 p STARTED            165 199.3kb xx.x.xxx.bb Sinister
apiman_gateway 2 r STARTED            165   184kb xx.x.xxx.aa Jackson Arvad
jsonrpc        4 p STARTED              0    144b xx.x.xxx.bb Sinister
jsonrpc        4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
jsonrpc        0 p STARTED              0    144b xx.x.xxx.bb Sinister
jsonrpc        0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
jsonrpc        3 p STARTED              0    144b xx.x.xxx.bb Sinister
jsonrpc        3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
jsonrpc        1 p STARTED              0    144b xx.x.xxx.bb Sinister
jsonrpc        1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
jsonrpc        2 p STARTED              0    144b xx.x.xxx.bb Sinister
jsonrpc        2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmill6cl.exe 2 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmill6cl.exe 2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmill6cl.exe 0 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmill6cl.exe 0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmill6cl.exe 3 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmill6cl.exe 3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmill6cl.exe 1 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmill6cl.exe 1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmill6cl.exe 4 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmill6cl.exe 4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
22

I suspect you do not have enough space to allocate all the shards...

To be clear I am not sure whether we can easily help you fix this.

Run

Please run with the /?v

and

24

So, Pl find below output for above command

OUTPUT FOR REFERENCE
OUTPUT FOR REFERENCE933×288 9.07 KB

sudo curl 'xx.x.xxx.aa:9200/_cat/allocation/?v'

shards disk.used disk.avail disk.total disk.percent host ip node
82 175.9gb 93gb 268.9gb 65 abc.de.fgh xx.x.xxx.bb Sinister
82 79.1gb 139.7gb 218.9gb 36 xyz.de.fgh xx.x.xxx.aa Jackson Arvad
8 UNASSIGNED

sudo curl xx.x.xxx.aa:9200/_cat/nodes/?v

host ip heap.percent ram.percent load node.role master name
abc.de.fgh xx.x.xxx.bb 41 44 0.76 d * Sinister
xyz.de.fgh xx.x.xxx.aa 42 39 0.27 d m Jackson Arvad

25

ANY UPDATE ON THIS

26

Unrelated note but it looks like your cluster has been exposed on internet with no security at all.
It looks compromised.

27

So, what's next?

28

@stephenb Pl help on this

29

I'd start a new cluster with 8.0 which is secured by default and reindex all the data there.
Or I'd switch to cloud.elastic.co which is even easier.

30

@stephenb Any update?

31

No not really.

Your cluster is so old That of Many of the tools I would use to try to fix it don't exist in that version.

It looks like it may have been neglected for a long time.

I don't know why those shards are unassigned, data may be missing all together.

In addition, I agree with @dadoonet that your cluster is most likely been compromised.

I think your best bet is to rebuild from scratch or use elastic cloud or look for.

We can't always fix everything from the community forums.