Elasticsearch index

umangpatel · March 4, 2022, 8:45am

Pl find below output.

1.)curl xx.x.xxx.aa:9200/_cat/health
646382008 13:50:08 apiman red 2 2 163 82 0 1 8 0

2.)curl 'xx.x.xxx.aa:9200/_cat/health?v&ts=0'
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks
1646382084 13:51:24 apiman red 2 2 163 82 0 1 8 0

3.)curl xx.x.xxx.aa:9200:9200/_cat/shards

test           4 p STARTED              0    144b xx.x.xxx.bb Sinister
test           4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
test           0 p STARTED              0    144b xx.x.xxx.bb Sinister
test           0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
test           3 p STARTED              0    144b xx.x.xxx.bb Sinister
test           3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
test           1 p STARTED              0    144b xx.x.xxx.bb Sinister
test           1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
test           2 p STARTED              0    144b xx.x.xxx.bb Sinister
test           2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
srrvas1.html   2 p STARTED              0    144b xx.x.xxx.bb Sinister
srrvas1.html   2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
srrvas1.html   0 p STARTED              0    144b xx.x.xxx.bb Sinister
srrvas1.html   0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
srrvas1.html   3 p STARTED              0    144b xx.x.xxx.bb Sinister
srrvas1.html   3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
srrvas1.html   1 p STARTED              0    144b xx.x.xxx.bb Sinister
srrvas1.html   1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
srrvas1.html   4 p STARTED              0    144b xx.x.xxx.bb Sinister
srrvas1.html   4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
servlet        4 p STARTED              0    144b xx.x.xxx.bb Sinister
servlet        4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
servlet        0 p STARTED              0    144b xx.x.xxx.bb Sinister
servlet        0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
servlet        3 p STARTED              0    144b xx.x.xxx.bb Sinister
servlet        3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
servlet        1 p STARTED              0    144b xx.x.xxx.bb Sinister
servlet        1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
servlet        2 p STARTED              0    144b xx.x.xxx.bb Sinister
servlet        2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
spipe          4 p STARTED              0    144b xx.x.xxx.bb Sinister
spipe          4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
spipe          0 p STARTED              0    144b xx.x.xxx.bb Sinister
spipe          0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
spipe          3 p STARTED              0    144b xx.x.xxx.bb Sinister
spipe          3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
spipe          1 p STARTED              0    144b xx.x.xxx.bb Sinister
spipe          1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
spipe          2 p STARTED              0    144b xx.x.xxx.bb Sinister
spipe          2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
blazeds        4 p STARTED              0    144b xx.x.xxx.bb Sinister
blazeds        4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
blazeds        0 p STARTED              0    144b xx.x.xxx.bb Sinister
blazeds        0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
blazeds        3 p STARTED              0    144b xx.x.xxx.bb Sinister
blazeds        3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
blazeds        1 p STARTED              0    144b xx.x.xxx.bb Sinister
blazeds        1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
blazeds        2 p STARTED              0    144b xx.x.xxx.bb Sinister
blazeds        2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
messagebroker  4 p STARTED              0    144b xx.x.xxx.bb Sinister
messagebroker  4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
messagebroker  0 p STARTED              0    144b xx.x.xxx.bb Sinister
messagebroker  0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
messagebroker  3 p STARTED              0    144b xx.x.xxx.bb Sinister
messagebroker  3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
messagebroker  1 p STARTED              0    144b xx.x.xxx.bb Sinister
messagebroker  1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
messagebroker  2 p STARTED              0    144b xx.x.xxx.bb Sinister
messagebroker  2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
phppath        4 p STARTED              0    144b xx.x.xxx.bb Sinister
phppath        4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
phppath        0 p STARTED              0    144b xx.x.xxx.bb Sinister
phppath        0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
phppath        3 p STARTED              0    144b xx.x.xxx.bb Sinister
phppath        3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
phppath        1 p STARTED              0    144b xx.x.xxx.bb Sinister
phppath        1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
phppath        2 p STARTED              0    144b xx.x.xxx.bb Sinister
phppath        2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
flex2gateway   2 p STARTED              0    144b xx.x.xxx.bb Sinister
flex2gateway   2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
flex2gateway   0 p STARTED              0    144b xx.x.xxx.bb Sinister
flex2gateway   0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
flex2gateway   3 p STARTED              0    144b xx.x.xxx.bb Sinister
flex2gateway   3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
flex2gateway   1 p STARTED              0    144b xx.x.xxx.bb Sinister
flex2gateway   1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
flex2gateway   4 p STARTED              0    144b xx.x.xxx.bb Sinister
flex2gateway   4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
.kibana        0 p STARTED             11  29.6kb xx.x.xxx.bb Sinister
.kibana        0 r STARTED             11  29.6kb xx.x.xxx.aa Jackson Arvad
lcds           4 p STARTED              0    144b xx.x.xxx.bb Sinister
lcds           4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
lcds           0 p STARTED              0    144b xx.x.xxx.bb Sinister
lcds           0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
lcds           3 p STARTED              0    144b xx.x.xxx.bb Sinister
lcds           3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
lcds           1 p STARTED              0    144b xx.x.xxx.bb Sinister
lcds           1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
lcds           2 p STARTED              0    144b xx.x.xxx.bb Sinister
lcds           2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
webui          4 p STARTED              0    144b xx.x.xxx.bb Sinister
webui          4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
webui          0 p STARTED              0    144b xx.x.xxx.bb Sinister
webui          0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
webui          3 p STARTED              0    144b xx.x.xxx.bb Sinister
webui          3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
webui          1 p STARTED              0    144b xx.x.xxx.bb Sinister
webui          1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
webui          2 p STARTED              0    144b xx.x.xxx.bb Sinister
webui          2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
cgi-bin        2 p STARTED              0    144b xx.x.xxx.bb Sinister
cgi-bin        2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
cgi-bin        0 p STARTED              0    144b xx.x.xxx.bb Sinister
cgi-bin        0 r STARTED              0    144b xx.x.xxx.aa Jackson Arvad
cgi-bin        3 p STARTED              0    144b xx.x.xxx.bb Sinister
cgi-bin        3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
cgi-bin        1 p STARTED              0    144b xx.x.xxx.bb Sinister
cgi-bin        1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
cgi-bin        4 p STARTED              0    144b xx.x.xxx.bb Sinister
cgi-bin        4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
apiman_metrics 2 p UNASSIGNED
apiman_metrics 2 r UNASSIGNED
apiman_metrics 0 p STARTED      340611781  54.7gb xx.x.xxx.bb Sinister
apiman_metrics 0 r INITIALIZING                   xx.x.xxx.aa Jackson Arvad
apiman_metrics 3 p UNASSIGNED
apiman_metrics 3 r UNASSIGNED
apiman_metrics 1 p UNASSIGNED
apiman_metrics 1 r UNASSIGNED
apiman_metrics 4 p UNASSIGNED
apiman_metrics 4 r UNASSIGNED
perl           4 p STARTED              0    144b xx.x.xxx.bb Sinister
perl           4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
perl           0 p STARTED              0    144b xx.x.xxx.bb Sinister
perl           0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
perl           3 p STARTED              0    144b xx.x.xxx.bb Sinister
perl           3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
perl           1 p STARTED              0    144b xx.x.xxx.bb Sinister
perl           1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
perl           2 p STARTED              0    144b xx.x.xxx.bb Sinister
perl           2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmillcl.exe  2 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmillcl.exe  2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmillcl.exe  0 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmillcl.exe  0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmillcl.exe  3 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmillcl.exe  3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmillcl.exe  1 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmillcl.exe  1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmillcl.exe  4 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmillcl.exe  4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
apiman_gateway 4 p STARTED            155 226.7kb xx.x.xxx.bb Sinister
apiman_gateway 4 r STARTED            155 227.2kb xx.x.xxx.aa Jackson Arvad
apiman_gateway 0 p STARTED            151 209.9kb xx.x.xxx.bb Sinister
apiman_gateway 0 r STARTED            151 210.2kb xx.x.xxx.aa Jackson Arvad
apiman_gateway 3 p STARTED            198 245.1kb xx.x.xxx.bb Sinister
apiman_gateway 3 r STARTED            198 239.6kb xx.x.xxx.aa Jackson Arvad
apiman_gateway 1 p STARTED            174 220.4kb xx.x.xxx.bb Sinister
apiman_gateway 1 r STARTED            174 219.6kb xx.x.xxx.aa Jackson Arvad
apiman_gateway 2 p STARTED            165 199.3kb xx.x.xxx.bb Sinister
apiman_gateway 2 r STARTED            165   184kb xx.x.xxx.aa Jackson Arvad
jsonrpc        4 p STARTED              0    144b xx.x.xxx.bb Sinister
jsonrpc        4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
jsonrpc        0 p STARTED              0    144b xx.x.xxx.bb Sinister
jsonrpc        0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
jsonrpc        3 p STARTED              0    144b xx.x.xxx.bb Sinister
jsonrpc        3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
jsonrpc        1 p STARTED              0    144b xx.x.xxx.bb Sinister
jsonrpc        1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
jsonrpc        2 p STARTED              0    144b xx.x.xxx.bb Sinister
jsonrpc        2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmill6cl.exe 2 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmill6cl.exe 2 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmill6cl.exe 0 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmill6cl.exe 0 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmill6cl.exe 3 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmill6cl.exe 3 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmill6cl.exe 1 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmill6cl.exe 1 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad
sawmill6cl.exe 4 p STARTED              0    144b xx.x.xxx.bb Sinister
sawmill6cl.exe 4 r STARTED              0    108b xx.x.xxx.aa Jackson Arvad

stephenb · March 4, 2022, 3:54pm

I suspect you do not have enough space to allocate all the shards...

To be clear I am not sure whether we can easily help you fix this.

Run

Please run with the /?v

and

umangpatel · March 7, 2022, 5:28am

So, Pl find below output for above command

sudo curl 'xx.x.xxx.aa:9200/_cat/allocation/?v'

shards disk.used disk.avail disk.total disk.percent host ip node
82 175.9gb 93gb 268.9gb 65 abc.de.fgh xx.x.xxx.bb Sinister
82 79.1gb 139.7gb 218.9gb 36 xyz.de.fgh xx.x.xxx.aa Jackson Arvad
8 UNASSIGNED

sudo curl xx.x.xxx.aa:9200/_cat/nodes/?v

host ip heap.percent ram.percent load node.role master name
abc.de.fgh xx.x.xxx.bb 41 44 0.76 d * Sinister
xyz.de.fgh xx.x.xxx.aa 42 39 0.27 d m Jackson Arvad

umangpatel · March 7, 2022, 9:51am

ANY UPDATE ON THIS

dadoonet · March 7, 2022, 10:07am

Unrelated note but it looks like your cluster has been exposed on internet with no security at all.
It looks compromised.

umangpatel · March 7, 2022, 10:19am

So, what's next?

umangpatel · March 7, 2022, 10:21am

@stephenb Pl help on this

dadoonet · March 7, 2022, 10:43am

I'd start a new cluster with 8.0 which is secured by default and reindex all the data there.
Or I'd switch to cloud.elastic.co which is even easier.

umangpatel · March 8, 2022, 4:38am

@stephenb Any update?

stephenb · March 8, 2022, 5:42am

No not really.

Your cluster is so old That of Many of the tools I would use to try to fix it don't exist in that version.

It looks like it may have been neglected for a long time.

I don't know why those shards are unassigned, data may be missing all together.

In addition, I agree with @dadoonet that your cluster is most likely been compromised.

I think your best bet is to rebuild from scratch or use elastic cloud or look for.

We can't always fix everything from the community forums.

system · April 5, 2022, 5:42am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.