WARN#011[elasticsearch]#011elasticsearch/client.go:408#011Cannot index event publisher.Event

Elastic Stack Elasticsearch
1

Hi,

ES is flooting syslog.

Nov 7 13:57:04 nuc-mini-server filebeat[757]: 2020-11-07T13:57:04.044+0100#011WARN#011[elasticsearch]#011elasticsearch/client.go:408#011Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfe1c446f77d192f, ext:86400206004069, loc:(*time.Location)(0x60807a0)}, Meta:{"pipeline":"filebeat-7.9.3-system-syslog-pipeline"}, Fields:{"agent":{"ephemeral_id":"4187a8d4-ae01-4962-99c0-662fdf3b3266","hostname":"nuc-mini-server","id":"c445a486-4727-4600-abfa-eaf7de13141c","name":"nuc-mini-server","type":"filebeat","version":"7.9.3"},"ecs":{"version":"1.5.0"},"event":{"dataset":"system.syslog","module":"system","timezone":"+01:00"},"fileset":{"name":"syslog"},"host":{"architecture":"x86_64","containerized":false,"hostname":"nuc-mini-server","id":"b88bd0c7768c49b491f9ca611ef3b04e","ip":["192.168.178.32","2001:16b8:c202:200:1e69:7aff:fe68:d0ee","fd00::1e69:7aff:fe68:d0ee","fe80::1e69:7aff:fe68:d0ee","172.17.0.1"],"mac":["1c:69:7a:68:d0:ee","84:c5:a6:9e:43:87","02:42:73:ab:90:9f"],"name":"nuc-mini-server","os":{"codename":"focal","family":"debian","kernel":"5.4.0-52-generic","name":"Ubuntu","platform":"ubuntu","version":"20.04.1 LTS (Focal Fossa)"}},"input":{"type":"log"},"log":{"file":{"path":"/var/log/syslog"},"offset":32053019954},"message":"Nov 7 13:53:38 nuc-mini-server filebeat[757]: 2020-11-07T13:53:38.539+0100#011WARN#011[elasticsearch]#011elasticsearch/client.go:408#011Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfe1c413a152a6b2, ext:86194834123495, loc:(*time.Location)(0x60807a0)}, Meta:{\"pipeline\":\"filebeat-7.9.3-system-syslog-pipeline\"}, Fields:{\"agent\":{\"ephemeral_id\":\"4187a8d4-ae01-4962-99c0-662fdf3b3266\",\"hostname\":\"nuc-mini-server\",\"id\":\"c445a486-4727-4600-abfa-eaf7de13141c\",\"name\":\"nuc-mini-server\",\"type\":\"filebeat\",\"version\":\"7.9.3\"},\"ecs\":{\"version\":\"1.5.0\"},\"event\":{\"dataset\":\"system.syslog\",\"module\":\"system\",\"timezone\":\"+01:00\"},\"fileset\":{\"name\":\"syslog\"},\"host\":{\"architecture\":\"x86_64\",\"containerized\":false,\"hostname\":\"nuc-mini-server\",\"id\":\"b88bd0c7768c49b491f9ca611ef3b04e\",\"ip\":[\"192.168.178.32\",\"2001:16b8:c202:200:1e69:7aff:fe68:d0ee\",\"fd00::1e69:7aff:fe68:d0ee\",\"fe80::1e69:7aff:fe68:d0ee\",\"172.17.0.1\"],\"mac\":[\"1c:69:7a:68:d0:ee\",\"84:c5:a6:9e:43:87\",\"02:42:73:ab:90:9f\"],\"name\":\"nuc-mini-server\",\"os\":{\"codename\":\"focal\",\"family\":\"debian\",\"kernel\":\"5.4.0-52-generic\",\"name\":\"Ubuntu\",\"platform\":\"ubuntu\",\"version\":\"20.04.1 LTS (Focal Fossa)\"}},\"input\":{\"type\":\"log\"},\"log\":{\"file\":{\"path\":\"/var/log/syslog\"},\"offset\":30339979340},\"message\":\"Nov 7 13:50:16 nuc-mini-server filebeat[757]: 2020-11-07T13:50:16.128+0100#011WARN#011[elasticsearch]#011elasticsearch/client.go:408#011Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfe1c3e111737881, ext:85992567838909, loc:(*time.Location)(0x60807a0)}, Meta:{\\\"pipeline\\\":\\\"filebeat-7.9.3-system-syslog-pipeline\\\"}, Fields:{\\\"agent\\\":{\\\"ephemeral_id\\\":\\\"4187a8d4-ae01-4962-99c0-662fdf3b3266\\\",\\\"hostname\\\":\\\"nuc-mini-server\\\",\\\"id\\\":\\\"c445a486-4727-4600-abfa-eaf7de13141c\\\",\\\"name\\\":\\\"nuc-mini-server\\\",\\\"type\\\":\\\"filebeat\\\",\\\"version\\\":\\\"7.9.3\\\"},\\\"ecs\\\":{\\\"version\\\":\\\"1.5.0\\\"},\\\"event\\\":{\\\"dataset\\\":\\\"system.syslog\\\",\\\"module\\\":\\\"system\\\",\\\"timezone\\\":\\\"+01:00\\\"},\\\"fileset\\\":{\\\"name\\\":\\\"syslog\\\"},\\\"host\\\":{\\\"architecture\\\":\\\"x86_64\\\",\\\"containerized\\\":false,\\\"hostname\\\":\\\"nuc-mini-server\\\",\\\"id\\\":\\\"b88bd0c7768c49b491f9ca611ef3b04e\\\",\\\"ip\\\":[\\\"192.168.178.32\\\",\\\"2001:16b8:c202:200:1e69:7aff:fe68:d0ee\\\",\\\"fd00::1e69:7aff:fe68:d0ee\\\",\\\"fe80::1e69:7aff:fe68:d0ee\\\",\\\"172.17.0.1\\\"],\\\"mac\\\":[\\\"1c:69:7a:68:d0:ee\\\",\\\"84:c5:a6:9e:43:87\\\",\\\"02:42:73:ab:90:9f\\\"],\\\"name\\\":\\\"nuc-mini-server\\\",\\\"os\\\":{\\\"codename\\\":\\\"focal\\\",\\\"family\\\":\\\"debian\\\",\\\"kernel\\\":\\\"5.4.0-52-generic\\\",\\\"name\\\":\\\"Ubuntu\\\",\\\"platform\\\":\\\"ubuntu\\\",\\\"version\\\":\\\"20.04.1 LTS (Focal Fossa)\\\"}},\\\"input\\\":{\\\"type\\\":\\\"log\\\"},\\\"log\\\":{\\\"file\\\":{\\\"path\\\":\\\"/var/log/syslog\\\"},\\\"offset\\\":28658103739},\\\"message\\\":\\\"Nov 7 13:46:55 nuc-mini-server filebeat[757]: 2020-11-07T13:46:55.638+0100#011WARN#011[elasticsearch]#011elasticsearch/client.go:408#011Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfe1c3aef0307854, ext:85792083541647, loc:(*time.Location)(0x60807a0)}, Meta:{\\\\\\\"pipeline\\\\\\\":\\\\\\\"filebeat-7.9.3-system-syslog-pipeline\\\\\\\"}, Fields:{\\\\\\\"agent\\\\\\\":{\\\\\\\"ephemeral_id\\\\\\\":\\\\\\\"4187a8d4-ae01-4962-99c0-662fdf3b3266\\\\\\\",\\\\\\\"hostname\\\\\\\":\\\\\\\"nuc-mini-server\\\\\\\",\\\\\\\"id\\\\\\\":\\\\\\\"c445a486-4727-4600-abfa-eaf7de13141c\\\\\\\",\\\\\\\"name\\\\\\\":\\\\\\\"nuc-mini-server\\\\\\\",\\\\\\\"type\\\\\\\":\\\\\\\"filebeat\\\\\\\",\\\\\\\"version\\\\\\\":\\\\\\\"7.9.3\\\\\\\"},\\\\\\\"ecs\\\\\\\":{\\\\\\\"version\\\\\\\":\\\\\\\"1.5.0\\\\\\\"},\\\\\\\"event\\\\\\\":{\\\\\\\"dataset\\\\\\\":\\\\\\\"system.syslog\\\\\\\",\\\\\\\"module\\\\\\\":\\\\\\\"system\\\\\\\",\\\\\\\"timezone\\\\\\\":\\\\\\\"+01:00\\\\\\\"},\\\\\\\"fileset\\\\\\\":{\\\\\\\"name\\\\\\\":\\\\\\\"syslog\\\\\\\"},\\\\\\\"host\\\\\\\":{\\\\\\\"architecture\\\\\\\":\\\\\\\"x86_64\\\\\\\",\\\\\\\"containerized\\\\\\\":false,\\\\\\\"hostname\\\\\\\":\\\\\\\"nuc-mini-server\\\\\\\",\\\\\\\"id\\\\\\\":\\\\\\\"b88bd0c7768c49b491f9ca611ef3b04e\\\\\\\",\\\\\\\"ip\\\\\\\":[\\\\\\\"192.168.178.32\\\\\\\",\\\\\\\"2001:16b8:c202:200:1e69:7aff:fe68:d0ee\\\\\\\",\\\\\\\"fd00::1e69:7aff:fe68:d0ee\\\\\\\",\\\\\\\"fe80::1e69:7aff:fe68:d0ee\\\\\\\",\\\\\\\"172.17.0.1\\\\\\\"],\\\\\\\"mac\\\\\\\":[\\\\\\\"1c:69:7a:68:d0:ee\\\\\\\",\\\\\\\"84:c5:a6:9e:43:87\\\\\\\",\\\\\\\"02:42:73:ab:90:9f\\\\\\\"],\\\\\\\"name\\\\\\\":\\\\\\\"nuc-mini-server\\\\\\\",\\\\\\\"os\\\\\\\":{\\\\\\\"codename\\\\\\\":\\\\\\\"focal\\\\\\\",\\\\\\\"family\\\\\\\":\\\\\\\"debian\\\\\\\",\\\\\\\"kernel\\\\\\\":\\\\\\\"5.4.0-52-generic\\\\\\\",\\\\\\\"name\\\\\\\":\\\\\\\"Ubuntu\\\\\\\",\\\\\\\"platform\\\\\\\":\\\\\\\"ubuntu\\\\\\\",\\\\\\\"version\\\\\\\":\\\\\\\"20.04.1 LTS (Focal Fossa)\\\\\\\"}},\\\\\\\"input\\\\\\\":{\\\\\\\"type\\\\\\\":\\\\\\\"log\\\\\\\"},\\\\\\\"log\\\\\\\":{\\\\\\\"file\\\\\\\":{\\\\\\\"path\\\\\\\":\\\\\\\"/var/log/syslog\\\\\\\"},\\\\\\\"offset\\\\\\\":27022000377},\\\\\\\"message\\\\\\\":\\\\\\\"Nov 7 13:43:40 nuc-mini-server filebeat[757]: 2020-11-07T13:43:40.846+0100#011WARN#011[elasticsearch]#011elasticsearch/client.go:408#011Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfe1c37e4736ed9b, ext:85597396099029, loc:(*time.Location)(0x60807a0)}, Meta:{\\\\\\\\\\\\\\\"pipeline\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"filebeat-7.9.3-system-syslog-pipeline\\\\\\\\\\\\\\\"}, Fields:{\\\\\\\\\\\\\\\"agent\\\\\\\\\\\\\\\":{\\\\\\\\\\\\\\\"ephemeral_id\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"4187a8d4-ae01-4962-99c0-662fdf3b3266\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"hostname\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"nuc-mini-server\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"id\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"c445a486-4727-4600-abfa-eaf7de13141c\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"name\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"nuc-mini-server\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"type\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"filebeat\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"version\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"7.9.3\\\\\\\\\\\\\\\"},\\\\\\\\\\\\\\\"ecs\\\\\\\\\\\\\\\":{\\\\\\\\\\\\\\\"version\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"1.5.0\\\\\\\\\\\\\\\"},\\\\\\\\\\\\\\\"event\\\\\\\\\\\\\\\":{\\\\\\\\\\\\\\\"dataset\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"system.syslog\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"module\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"system\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"timezone\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"+01:00\\\\\\\\\\\\\\\"},\\\\\\\\\\\\\\\"fileset\\\\\\\\\\\\\\\":{

Memory & Space looks good. But start to swap.

image
image746×274 14.6 KB

And the cores are working ...

image

Regards,
Rafal

2

Is that the Elasticsearch log?

3

Not. This was the syslog. And the size war ~30GB after 1-2 hours. I removed the current elasticsearch index and the problem was gone.

Sorry, fot the late answer.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.