Filebeat logs not visible in kibana dashboards

Elastic Stack Beats
1

Hi,

After a long back and forth with my issue on this link:

I've been forwarded to here.

My question is that I'm receiving logs in the kibana:

logs
logs1905×822 187 KB

But still the tables don't show anything neither events from SEIM.

data2
data21853×689 42.2 KB

events
events1744×606 73.7 KB

1- Is this because I set the dashboards (sudo filebeat setup --dashboards) from my side where they are set up automatically so no need to do them from my side?

Or is there anything else?

Here is part of filebeat logs:

amdin@amdin-virtual-machine:~$ journalctl --unit=filebeat -f
-- Logs begin at Wed 2020-03-25 12:38:30 EET. --
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.296+0300        INFO        [index-management.ilm]        ilm/std.go:139        do not generate ilm policy: exists=true, overwrite=false
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.296+0300        INFO        [index-management]        idxmgmt/std.go:271        ILM policy successfully loaded.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.296+0300        INFO        [index-management]        idxmgmt/std.go:410        Set setup.template.name to '{filebeat-7.6.2 {now/d}-000001}' as ILM is enabled.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.296+0300        INFO        [index-management]        idxmgmt/std.go:415        Set setup.template.pattern to 'filebeat-7.6.2-*' as ILM is enabled.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.297+0300        INFO        [index-management]        idxmgmt/std.go:449        Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.6.2 {now/d}-000001} as ILM is enabled.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.297+0300        INFO        [index-management]        idxmgmt/std.go:453        Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.299+0300        INFO        template/load.go:89        Template filebeat-7.6.2 already exists and will not be overwritten.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.300+0300        INFO        [index-management]        idxmgmt/std.go:295        Loaded index template.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.644+0300        INFO        [index-management]        idxmgmt/std.go:306        Write alias successfully generated.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.664+0300        INFO        pipeline/output.go:105        Connection to backoff(elasticsearch(http://192.168.2.220:9200)) established
May 13 12:17:48 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:48.251+0300        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":{"ms":165}},"total":{"ticks":290,"time":{"ms":298},"value":290},"user":{"ticks":130,"time":{"ms":133}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":24},"info":{"ephemeral_id":"94251c67-a6e0-4d82-9e6a-a39d2d673bcd","uptime":{"ms":30055}},"memstats":{"gc_next":22033904,"memory_alloc":14765896,"memory_total":32108248,"rss":58568704},"runtime":{"goroutines":94}},"filebeat":{"events":{"added":1896,"done":1896},"harvester":{"files":{"0172a8b4-4475-42ee-adc8-db2470ecdb4b":{"last_event_published_time":"2020-05-13T12:17:21.266Z","last_event_timestamp":"2020-05-13T12:17:21.264Z","name":"/var/log/vmware-vmtoolsd-root.log","read_offset":522,"size":522,"start_time":"2020-05-13T12:17:18.264Z"},"32240ca2-74c7-4f54-9f3e-fc1c34304173":{"last_event_published_time":"2020-05-13T12:17:29.277Z","last_event_timestamp":"2020-05-13T12:17:29.277Z","name":"/var/log/auth.log","read_offset":5376,"size":5032,"start_time":"2020-05-13T12:17:18.266Z"},"45c6754f-a096-4f5a-905d-f1357834f969":{"last_event_published_time":"2020-05-13T12:17:21.297Z","last_event_timestamp":"2020-05-13T12:17:21.296Z","name":"/var/log/vmware-vmsvc-root.1.log","read_offset":10155,"size":10155,"start_time":"2020-05-13T12:17:18.246Z"},"877f3d78-e374-44f9-9d21-2a738bc28e5e":{"last_event_published_time":"2020-05-13T12:17:21.292Z","last_event_timestamp":"2020-05-13T12:17:21.292Z","name":"/var/log/fontconfig.log","read_offset":5873,"size":5873,"start_time":"2020-05-13T12:17:18.266Z"},"8c84e524-a0d4-4b34-9b91-637d04838925":{"last_event_published_time":"2020-05-13T12:17:21.250Z","last_event_timestamp":"2020-05-13T12:17:21.250Z","name":"/var/log/vmware-network.log","read_offset":3211,"size":3211,"start_time":"2020-05-13T12:17:18.265Z"},"987901bf-bda9-4c2c-a7ce-db52450b4c54":{"last_event_published_time":"2020-05-13T12:17:21.304Z","last_event_timestamp":"2020-05-13T12:17:21.304Z","name":"/var/log/bootstrap.log","read_offset":56751,"size":56751,"start_time":"2020-05-13T12:17:18.268Z"},"a4a04d7d-ab34-45f5-aa07-3294edd56066":{"last_event_published_time":"2020-05-13T12:17:36.302Z","last_event_timestamp":"2020-05-13T12:17:36.302Z","name":"/var/log/vmware-vmsvc-root.log","read_offset":31095,"size":30941,"start_time":"2020-05-13T12:17:18.261Z"},"a4ca167e-0d07-4952-9c2b-db52d716b6cb":{"last_event_published_time":"2020-05-13T12:17:21.253Z","last_event_timestamp":"2020-05-13T12:17:21.252Z","name":"/var/log/vmware-network.2.log","read_offset":685,"size":685,"start_time":"2020-05-13T12:17:18.264Z"},"a7e321af-51d6-4d17-8a31-f7794121ec4f":{"last_event_published_time":"2020-05-13T12:17:21.294Z","last_event_timestamp":"2020-05-13T12:17:21.294Z","name":"/var/log/vmware-network.1.log","read_offset":3211,"size":3211,"start_time":"2020-05-13T12:17:18.268Z"},"af44e01c-1e0b-4e3f-aaec-7e07a1b639c6":{"last_event_published_time":"2020-05-13T12:17:21.245Z","last_event_timestamp":"2020-05-13T12:17:21.245Z","name":"/var/log/kern.log","read_offset":5636,"size":5636,"start_time":"2020-05-13T12:17:18.266Z"},"b34efc54-ca12-4fd5-b42a-a4301786817b":{"last_event_published_time":"2020-05-13T12:17:21.289Z","last_event_timestamp":"2020-05-13T12:17:21.289Z","name":"/var/log/gpu-manager.log","read_offset":1163,"size":1163,"start_time":"2020-05-13T12:17:18.246Z"},"b4ddd1d8-84a7-4b40-8796-f4cd600364e9":{"last_event_published_time":"2020-05-13T12:17:21.289Z","last_event_timestamp":"2020-05-13T12:17:21.289Z","name":"/var/log/dpkg.log","read_offset":1535,"size":1535,"start_time":"2020-05-13T12:17:18.260Z"},"bcaee23f-de79-4729-878d-b0c049c97e3d":{"last_event_published_time":"2020-05-13T12:17:21.304Z","last_event_timestamp":"2020-05-13T12:17:21.304Z","name":"/var/log/vmware-vmsvc-root.2.log","read_offset":6113,"size":6113,"start_time":"2020-05-13T12:17:18.260Z"},"cded127f-e23c-444b-a074-1c6336b0ada1":{"last_event_published_time":"2020-05-13T12:17:21.284Z","last_event_timestamp":"2020-05-13T12:17:21.282Z","name":"/var/log/alternatives.log","read_offset":3485,"size":3485,"start_time":"2020-05-13T12:17:18.246Z"}},"open_files":14,"running":14,"started":14}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"acked":1818,"batches":39,"total":1818},"read":{"bytes":26755},"type":"elasticsearch","write":{"bytes":1411798}},"pipeline":{"clients":1,"events":{"active":0,"filtered":78,"published":1818,"retry":50,"total":1896},"queue":{"acked":1818}}},"registrar":{"states":{"current":14,"update":1896},"writes":{"success":52,"total":52}},"system":{"cpu":{"cores":1},"load":{"1":0.32,"15":0.04,"5":0.14,"norm":{"1":0.32,"15":0.04,"5":0.14}}}}}}

Your help is appreciated.

Thanks

2

Hey @ethical20,

In order to use the dashboards provided by the modules, in general you should be using modules.
Are you using the system module or you are collecting these logs directly with an input?

Modules parse logs, providing more structure data, and also add these event.module and event.dataset fields you are missing.

3

Hi,

I'm not using modules, I'm collecting these logs directly . As i've read that using modules is not a must.

I used not to use modules before and everything was fine!

4

Yes, you are right, using modules is not a must, but they help parsing logs and providing additional metadata in events. For example most of the fields that appear empty in this events view, are provided by modules.
event.module and event.dataset are filled by beats when using modules.
user.name, source.ip and destination.ip are usually parsed from logs. This parsing is usually done with modules, but you can also processors, logstash or ingest pipelines to do it without modules.
Most filebeat dashboards require these fields to be able to display some data, they are usually thought to be used with modules.

Even if you don't use modules or do any parsing, you can still use the discover or logs UI views to visualize the logs, but they won't contain a lot of structured data, that is specially useful in the SIEM app for example.

What do you mean by fine? Did you have the missing fields? Were you parsing with Logstash or your own ingest pipelines?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.