Unable to fetch data from reporting collector"

legolas_bilbao · February 28, 2019, 9:57am

Hi everybody,

the kibana doesnt start in the last days

the error is the following:

feb 28 09:50:25 ganondorf kibana[14813]: {"type":"error","@timestamp":"2019-02-28T09:50:25Z","tags":["warning","stats-collection"],"pid":14813,"level":"error","error":{"message":"[illegal_argument_exception]
feb 28 09:50:25 ganondorf kibana[14813]: {"type":"log","@timestamp":"2019-02-28T09:50:25Z","tags":["warning","stats-collection"],"pid":14813,"message":"Unable to fetch data from reporting collector"}
feb 28 09:50:35 ganondorf kibana[14813]: {"type":"error","@timestamp":"2019-02-28T09:50:35Z","tags":["warning","stats-collection"],"pid":14813,"level":"error","error":{"message":"[illegal_argument_exception]
feb 28 09:50:35 ganondorf kibana[14813]: {"type":"log","@timestamp":"2019-02-28T09:50:35Z","tags":["warning","stats-collection"],"pid":14813,"message":"Unable to fetch data from reporting collector"}
feb 28 09:50:45 ganondorf kibana[14813]: {"type":"error","@timestamp":"2019-02-28T09:50:45Z","tags":["warning","stats-collection"],"pid":14813,"level":"error","error":{"message":"[illegal_argument_exception]
feb 28 09:50:45 ganondorf kibana[14813]: {"type":"log","@timestamp":"2019-02-28T09:50:45Z","tags":["warning","stats-collection"],"pid":14813,"message":"Unable to fetch data from reporting collector"}
feb 28 09:50:55 ganondorf kibana[14813]: {"type":"error","@timestamp":"2019-02-28T09:50:55Z","tags":["warning","stats-collection"],"pid":14813,"level":"error","error":{"message":"[illegal_argument_exception]
feb 28 09:50:55 ganondorf kibana[14813]: {"type":"log","@timestamp":"2019-02-28T09:50:55Z","tags":["warning","stats-collection"],"pid":14813,"message":"Unable to fetch data from reporting collector"}
feb 28 09:51:05 ganondorf kibana[14813]: {"type":"error","@timestamp":"2019-02-28T09:51:05Z","tags":["warning","stats-collection"],"pid":14813,"level":"error","error":{"message":"[illegal_argument_exception]
feb 28 09:51:05 ganondorf kibana[14813]: {"type":"log","@timestamp":"2019-02-28T09:51:05Z","tags":["warning","stats-collection"],"pid":14813,"message":"Unable to fetch data from reporting collector"}

the cluster health is the following:

root@ganondorf:/etc/kibana# curl -X GET "localhost:9200/_cluster/stats?human&pretty"
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "elasticsearch",
"timestamp" : 1551347578159,
"status" : "yellow",
"indices" : {
"count" : 2,
"shards" : {
"total" : 6,
"primaries" : 6,
"replication" : 0.0,
"index" : {
"shards" : {
"min" : 1,
"max" : 5,
"avg" : 3.0
},
"primaries" : {
"min" : 1,
"max" : 5,
"avg" : 3.0
},
"replication" : {
"min" : 0.0,
"max" : 0.0,
"avg" : 0.0
}
}
},
"docs" : {
"count" : 38505389,
"deleted" : 4
},
"store" : {
"size" : "8.6gb",
"size_in_bytes" : 9245665985
},
"fielddata" : {
"memory_size" : "784b",
"memory_size_in_bytes" : 784,
"evictions" : 0
},
"query_cache" : {
"memory_size" : "1.2mb",
"memory_size_in_bytes" : 1353057,
"total_count" : 8120,
"hit_count" : 121,
"miss_count" : 7999,
"cache_size" : 87,
"cache_count" : 112,
"evictions" : 25
},
"completion" : {
"size" : "0b",
"size_in_bytes" : 0
},
"segments" : {
"count" : 117,
"memory" : "19.1mb",
"memory_in_bytes" : 20033176,
"terms_memory" : "13.9mb",
"terms_memory_in_bytes" : 14624801,
"stored_fields_memory" : "3.6mb",
"stored_fields_memory_in_bytes" : 3844168,
"term_vectors_memory" : "0b",
"term_vectors_memory_in_bytes" : 0,
"norms_memory" : "70.9kb",
"norms_memory_in_bytes" : 72640,
"points_memory" : "1.3mb",
"points_memory_in_bytes" : 1432371,
"doc_values_memory" : "57.8kb",
"doc_values_memory_in_bytes" : 59196,
"index_writer_memory" : "0b",
"index_writer_memory_in_bytes" : 0,
"version_map_memory" : "0b",
"version_map_memory_in_bytes" : 0,
"fixed_bit_set" : "0b",
"fixed_bit_set_memory_in_bytes" : 0,
"max_unsafe_auto_id_timestamp" : 1547970321160,
"file_sizes" : { }
}
},
"nodes" : {
"count" : {
"total" : 1,
"data" : 1,
"coordinating_only" : 0,
"master" : 1,
"ingest" : 1
},
"versions" : [
"6.4.2"
],
"os" : {
"available_processors" : 4,
"allocated_processors" : 4,
"names" : [
{
"name" : "Linux",
"count" : 1
}
],
"mem" : {
"total" : "7.7gb",
"total_in_bytes" : 8364498944,
"free" : "562.3mb",
"free_in_bytes" : 589635584,
"used" : "7.2gb",
"used_in_bytes" : 7774863360,
"free_percent" : 7,
"used_percent" : 93
}
},
"process" : {
"cpu" : {
"percent" : 22
},
"open_file_descriptors" : {
"min" : 309,
"max" : 309,
"avg" : 309
}
},
"jvm" : {
"max_uptime" : "40d",
"max_uptime_in_millis" : 3460854107,
"versions" : [
{
"version" : "1.8.0_191",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "25.191-b12",
"vm_vendor" : "Oracle Corporation",
"count" : 1
}
],
"mem" : {
"heap_used" : "656.9mb",
"heap_used_in_bytes" : 688877104,
"heap_max" : "990.7mb",
"heap_max_in_bytes" : 1038876672
},
"threads" : 148
},
"fs" : {
"total" : "491.1gb",
"total_in_bytes" : 527367929856,
"free" : "440.6gb",
"free_in_bytes" : 473110618112,
"available" : "415.6gb",
"available_in_bytes" : 446250455040
},
"plugins" : ,
"network_types" : {
"transport_types" : {
"security4" : 1
},
"http_types" : {
"security4" : 1
}
}
}
}

any idea?

nickpeihl · February 28, 2019, 6:01pm

It looks like the lines of the Kibana error logs you copied from stdout are truncated. Can you copy and paste the full lines of the errors from the kibana log file? You may have to set the logging.dest parameter in your kibana.yml file to create the log file.

legolas_bilbao · March 1, 2019, 8:11am

Configuration en kibana.yml

Enables you specify a file where Kibana stores log output.

logging.dest: /home//kibana.txt

Set the value of this setting to true to suppress all logging output.

#logging.silent: false

Set the value of this setting to true to suppress all logging output other than error messages.

#logging.quiet: false

Set the value of this setting to true to log all events, including system usage information

and all requests.

logging.verbose: true

system kibana status after reboot

root@ganondorf:/home/jallende# service kibana status
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2019-03-01 08:10:11 UTC; 2s ago
Process: 2282 ExecStart=/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml (code=exited, status=1/FAILURE)
Main PID: 2282 (code=exited, status=1/FAILURE)

mar 01 08:10:11 ganondorf systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
mar 01 08:10:11 ganondorf systemd[1]: kibana.service: Failed with result 'exit-code'.
mar 01 08:10:11 ganondorf systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
mar 01 08:10:11 ganondorf systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
mar 01 08:10:11 ganondorf systemd[1]: Stopped Kibana.
mar 01 08:10:11 ganondorf systemd[1]: kibana.service: Start request repeated too quickly.
mar 01 08:10:11 ganondorf systemd[1]: kibana.service: Failed with result 'exit-code'.
mar 01 08:10:11 ganondorf systemd[1]: Failed to start Kibana.

I dont what is happening now and the file log is empty

nickpeihl · March 1, 2019, 4:17pm

Can you post the contents of your /etc/kibana/kibana.yml file?

legolas_bilbao · March 3, 2019, 6:20pm

jallende@hostname:~$ sudo grep -v "#" /etc/kibana/kibana.yml

server.host: "localhost"

logging.dest: /home/path/kibana.txt

logging.verbose: true

legolas_bilbao · March 4, 2019, 2:44pm

I solved the issue but i dont know why its works. Change the profile for the log from to root to kibana user and it works

regards

system · April 1, 2019, 2:44pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to fetch data from rollups collector Kibana	5	4232	February 22, 2019
Kibana update error Kibana	3	658	April 13, 2019
Error in Kibana when looking back 60 days Kibana	4	789	July 6, 2017
Kibana Can not start Kibana	4	8739	June 3, 2019
Follow up: Unable to fetch data from kibana collector Kibana	5	2406	February 4, 2019