[Solved] "message":"write EPIPE","name":"Error"

Elastic Stack Kibana
1

Dear Al

After I removed mij filebeat index from kibana, kibana doesn't load the webinterface anymore

1916×1044 21.3 KB

This is the error I have i my logs:

{"type":"log","@timestamp":"2016-11-17T07:45:32Z","tags":["connection","client","error"],"pid":20281,"level":"error","message":"write EPIPE","error":{"message":"write EPIPE","name":"Error","stack":"Error: write EPIPE\n at exports._errnoException (util.js:1026:11)\n at WriteWrap.afterWrite (net.js:799:14)","code":"EPIPE"}}

I have no Idea what went wrong. Just removed the filebeat index. I still have 5 other indexes in kibana, but as you can see, nothing is loaded...

2

Are you able to go to the Management console and change the default index pattern? Is this an on-premise or cloud deployment? Is this a fresh 5.0.0 install or did you upgrade from a previous version?

3

This is on premise. I can't do anything. Only have the screen in the screen shot and on all the possible menu items, I get the error shown in the logs

4

The reason I asked these questions is that I had a very similar issue a while ago, but has not able to reproduce it once I got it resolved. I was able to 'unlock' the UI by inserting a dummy record into an index covered by the default index pattern and then change the default index pattern in the management panel.

Would it be possible for you to upload Kibana and Elasticsearch logs somewhere so we can have a look at them?

5

You can find the kibana logs here. This is all the log I have after falsing the silent option in kibana:

https://nextcloud.brusselsairport.be/index.php/s/ust0hBzCtvvvMBQ

No logging is showing up in elasticsearch

6

Did the workaround 'unlock' Kibana or are you still stuck?

7

I can't do anything in the Management UI... So... :slight_smile: No, not yet. I reran the script for creating the beats dashboard and index in kibana, bu t no luck...

8

I just had to add some data into one of the default indices. I did not make any changes to Kibana. Can you turn on a beat to get some data indexed? Once you have changed the default index pattern you can always delete it again.

9

There is data coming in for the filebeat index in elasticsearch. I just can't do anything in the gui to change or add the index :frowning: .

Is there a way to change the default index or add the index for filebeat trough the command line?

10

So there is now data in the indices that make up the default index pattern? Have you tried restarting Kibana?

11

Yes, there was never no data in the index. I removed the index because there was a problem with the geo mapping. And I wanted to force a new index for filebeat (Like I did with the previous version). But this time the interface went blank :smiley:

12

Timelion is the only thing that is still working.

[root@elasticsearch01prod filebeat]# curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open maillog-2016.10.14 XfmXZ89iQXGolI5h7PHhjg 2 1 548701 0 566.5mb 283.2mb
green open ovirt-2016.10.25 s8Qkw4knSfWEhKGqyv1fyw 5 1 5315342 0 4.1gb 2gb
green open syslog-2016.11.04 y7IrzfmaRsKUE-cUmPfpkw 5 1 12 0 160.8kb 80.4kb
green open syslog-2016.10.14 YdUtHsRiTfGiDHvmf3KSAw 5 1 791209 0 539.7mb 269.8mb
green open maillog-2016.10.25 T0fT6wo7SQCEGVGjBm6P_g 2 1 553579 0 575.9mb 287.9mb
green open basis-2016.11.15 LF7NM0zKTJaoL6G5TWo1og 5 1 26009 0 48.7mb 24.2mb
green open .marvel-es-1-2016.10.12 gP4581yRQqOeS7e66PAnYA 1 1 0 0 318b 159b
green open spipe 6tIo_k97RbSwnYKKy8-YeQ 5 1 0 0 1.5kb 795b
green open basis-2016.10.15 gu9g4oNTT4y-Zf8FQf6Djg 5 1 30829 0 61.8mb 30.9mb
green open .marvel-es-1-2016.10.17 OgP7drHYQpq-zUSpYqC7lg 1 1 304561 8330 290mb 145mb
green open .marvel-es-1-2016.10.11 auPZP4MfQmG4EEQbeZu0eA 1 1 0 0 318b 159b
green open maillog-2016.10.26 zW8pinIuRLGHCfi5Vy3L1w 2 1 562399 0 586.5mb 293.2mb
green open syslog-2016.11.12 -kF0sB0iRDadBgJjCkjXug 5 1 24487 0 25.5mb 12.6mb
green open syslog-2016.09.27 DfS7Cvq8TT2Jj-0xmdXywA 5 1 763356 0 505mb 252.5mb
green open syslog-2016.09.17 JknGRtIwS2-nSgJ4ocrVJg 5 1 675540 0 445.4mb 222.7mb
green open filebeat-2016.11.16 1DFKRGmRTouH1FvbGg5YBg 5 1 47891656 0 25.8gb 12.9g

13

Try inserting a single dummy record , e.g. '{"field1":1}' into an index covered by the default pattern you have configured. That worked for me. If it does not for you, it is most likely a different issue.

14

[root@elasticsearch01prod ~]# curl -XPUT 'localhost:9200/filebeat/foo/1?pretty' -d'

{"field1":1}'
{
"_index" : "filebeat",
"_type" : "foo",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"created" : true
}

Still same error in kibana.

15

And your default index pattern was'filebeat*' and not'filebeat-*' or something similar?

16

No you are right... The deafult index is filebeat-... But what am I doing wrong here...?
[root@elasticsearch01prod ~]# curl -XPUT 'http://localhost:9200/filebeat-/foo/1?pretty' -d'
{"field1":1}'
{
"error" : {
"root_cause" : [
{
"type" : "invalid_index_name_exception",
"reason" : "Invalid index name [filebeat-], must not contain the following characters [ , ", , \, <, |, ,, >, /, ?]",
"index_uuid" : "na",
"index" : "filebeat-"
}
],
"type" : "invalid_index_name_exception",
"reason" : "Invalid index name [filebeat-], must not contain the following characters [ , ", , \, <, |, ,, >, /, ?]",
"index_uuid" : "na",
"index" : "filebeat-"
},
"status" : 400
}

17

You can not have a index name with a star in it. Index into filebeat-2016.11.17 instead, which should match the pattern.

18

Well, that I did first:

[root@elasticsearch01prod ~]# curl -XPUT 'localhost:9200/filebeat-2016.11.17/foo/1?pretty' -d'
{"field1":1}'
{
"_index" : "filebeat-2016.11.17",
"_type" : "foo",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"created" : true
}

But that still gave nothing... So I thought I tried the next best thing :-)...
Restarted kibana... Still the same... :frowning:
{"type":"log","@timestamp":"2016-11-17T10:47:13Z","tags":["connection","client","error"],"pid":31767,"level":"error","message":"write EPIPE","error":{"message":"write EPIPE","name":"Error","stack":"Error: write EPIPE\n at exports._errnoException (util.js:1026:11)\n at WriteWrap.afterWrite (net.js:799:14)","code":"EPIPE"}}
{"type":"log","@timestamp":"2016-11-17T11:14:52Z","tags":["connection","client","error"],"pid":31767,"level":"error","message":"write EPIPE","error":{"message":"write EPIPE","name":"Error","stack":"Error: write EPIPE\n at exports._errnoException (util.js:1026:11)\n at WriteWrap.afterWrite (net.js:799:14)","code":"EPIPE"}}

19

Does Kibana connect directly to Elasticsearch or is there some kind of proxy in between? Are they on the same host?

20

Directly.