Filebeat logs not visible in kibana dashboards

[ethical20]

Hi,

After a long back and forth with my issue on this link:

Kibana dashboard No result found

Hi, Yes I've checked the filebeat.yml, edited this part to true and now the logs are sent to elasticsearch. - type: log # Change to true to enable this input configuration. enabled: false (changed to true) But still the tables don't show anything neither events from SEIM. [data2] [events] 1- Is this because I set the dashboards (sudo filebeat setup --dashboards) from my side where they are set up automatically so no need to do them from my side? 2- Is this because I didn't installe…

I've been forwarded to here.

My question is that I'm receiving logs in the kibana:

But still the tables don't show anything neither events from SEIM.

1- Is this because I set the dashboards (sudo filebeat setup --dashboards) from my side where they are set up automatically so no need to do them from my side?

Or is there anything else?

Here is part of filebeat logs:

amdin@amdin-virtual-machine:~$ journalctl --unit=filebeat -f
-- Logs begin at Wed 2020-03-25 12:38:30 EET. --
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.296+0300        INFO        [index-management.ilm]        ilm/std.go:139        do not generate ilm policy: exists=true, overwrite=false
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.296+0300        INFO        [index-management]        idxmgmt/std.go:271        ILM policy successfully loaded.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.296+0300        INFO        [index-management]        idxmgmt/std.go:410        Set setup.template.name to '{filebeat-7.6.2 {now/d}-000001}' as ILM is enabled.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.296+0300        INFO        [index-management]        idxmgmt/std.go:415        Set setup.template.pattern to 'filebeat-7.6.2-*' as ILM is enabled.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.297+0300        INFO        [index-management]        idxmgmt/std.go:449        Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.6.2 {now/d}-000001} as ILM is enabled.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.297+0300        INFO        [index-management]        idxmgmt/std.go:453        Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.299+0300        INFO        template/load.go:89        Template filebeat-7.6.2 already exists and will not be overwritten.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.300+0300        INFO        [index-management]        idxmgmt/std.go:295        Loaded index template.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.644+0300        INFO        [index-management]        idxmgmt/std.go:306        Write alias successfully generated.
May 13 12:17:22 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:22.664+0300        INFO        pipeline/output.go:105        Connection to backoff(elasticsearch(http://192.168.2.220:9200)) established
May 13 12:17:48 amdin-virtual-machine filebeat[4920]: 2020-05-13T12:17:48.251+0300        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":{"ms":165}},"total":{"ticks":290,"time":{"ms":298},"value":290},"user":{"ticks":130,"time":{"ms":133}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":24},"info":{"ephemeral_id":"94251c67-a6e0-4d82-9e6a-a39d2d673bcd","uptime":{"ms":30055}},"memstats":{"gc_next":22033904,"memory_alloc":14765896,"memory_total":32108248,"rss":58568704},"runtime":{"goroutines":94}},"filebeat":{"events":{"added":1896,"done":1896},"harvester":{"files":{"0172a8b4-4475-42ee-adc8-db2470ecdb4b":{"last_event_published_time":"2020-05-13T12:17:21.266Z","last_event_timestamp":"2020-05-13T12:17:21.264Z","name":"/var/log/vmware-vmtoolsd-root.log","read_offset":522,"size":522,"start_time":"2020-05-13T12:17:18.264Z"},"32240ca2-74c7-4f54-9f3e-fc1c34304173":{"last_event_published_time":"2020-05-13T12:17:29.277Z","last_event_timestamp":"2020-05-13T12:17:29.277Z","name":"/var/log/auth.log","read_offset":5376,"size":5032,"start_time":"2020-05-13T12:17:18.266Z"},"45c6754f-a096-4f5a-905d-f1357834f969":{"last_event_published_time":"2020-05-13T12:17:21.297Z","last_event_timestamp":"2020-05-13T12:17:21.296Z","name":"/var/log/vmware-vmsvc-root.1.log","read_offset":10155,"size":10155,"start_time":"2020-05-13T12:17:18.246Z"},"877f3d78-e374-44f9-9d21-2a738bc28e5e":{"last_event_published_time":"2020-05-13T12:17:21.292Z","last_event_timestamp":"2020-05-13T12:17:21.292Z","name":"/var/log/fontconfig.log","read_offset":5873,"size":5873,"start_time":"2020-05-13T12:17:18.266Z"},"8c84e524-a0d4-4b34-9b91-637d04838925":{"last_event_published_time":"2020-05-13T12:17:21.250Z","last_event_timestamp":"2020-05-13T12:17:21.250Z","name":"/var/log/vmware-network.log","read_offset":3211,"size":3211,"start_time":"2020-05-13T12:17:18.265Z"},"987901bf-bda9-4c2c-a7ce-db52450b4c54":{"last_event_published_time":"2020-05-13T12:17:21.304Z","last_event_timestamp":"2020-05-13T12:17:21.304Z","name":"/var/log/bootstrap.log","read_offset":56751,"size":56751,"start_time":"2020-05-13T12:17:18.268Z"},"a4a04d7d-ab34-45f5-aa07-3294edd56066":{"last_event_published_time":"2020-05-13T12:17:36.302Z","last_event_timestamp":"2020-05-13T12:17:36.302Z","name":"/var/log/vmware-vmsvc-root.log","read_offset":31095,"size":30941,"start_time":"2020-05-13T12:17:18.261Z"},"a4ca167e-0d07-4952-9c2b-db52d716b6cb":{"last_event_published_time":"2020-05-13T12:17:21.253Z","last_event_timestamp":"2020-05-13T12:17:21.252Z","name":"/var/log/vmware-network.2.log","read_offset":685,"size":685,"start_time":"2020-05-13T12:17:18.264Z"},"a7e321af-51d6-4d17-8a31-f7794121ec4f":{"last_event_published_time":"2020-05-13T12:17:21.294Z","last_event_timestamp":"2020-05-13T12:17:21.294Z","name":"/var/log/vmware-network.1.log","read_offset":3211,"size":3211,"start_time":"2020-05-13T12:17:18.268Z"},"af44e01c-1e0b-4e3f-aaec-7e07a1b639c6":{"last_event_published_time":"2020-05-13T12:17:21.245Z","last_event_timestamp":"2020-05-13T12:17:21.245Z","name":"/var/log/kern.log","read_offset":5636,"size":5636,"start_time":"2020-05-13T12:17:18.266Z"},"b34efc54-ca12-4fd5-b42a-a4301786817b":{"last_event_published_time":"2020-05-13T12:17:21.289Z","last_event_timestamp":"2020-05-13T12:17:21.289Z","name":"/var/log/gpu-manager.log","read_offset":1163,"size":1163,"start_time":"2020-05-13T12:17:18.246Z"},"b4ddd1d8-84a7-4b40-8796-f4cd600364e9":{"last_event_published_time":"2020-05-13T12:17:21.289Z","last_event_timestamp":"2020-05-13T12:17:21.289Z","name":"/var/log/dpkg.log","read_offset":1535,"size":1535,"start_time":"2020-05-13T12:17:18.260Z"},"bcaee23f-de79-4729-878d-b0c049c97e3d":{"last_event_published_time":"2020-05-13T12:17:21.304Z","last_event_timestamp":"2020-05-13T12:17:21.304Z","name":"/var/log/vmware-vmsvc-root.2.log","read_offset":6113,"size":6113,"start_time":"2020-05-13T12:17:18.260Z"},"cded127f-e23c-444b-a074-1c6336b0ada1":{"last_event_published_time":"2020-05-13T12:17:21.284Z","last_event_timestamp":"2020-05-13T12:17:21.282Z","name":"/var/log/alternatives.log","read_offset":3485,"size":3485,"start_time":"2020-05-13T12:17:18.246Z"}},"open_files":14,"running":14,"started":14}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"acked":1818,"batches":39,"total":1818},"read":{"bytes":26755},"type":"elasticsearch","write":{"bytes":1411798}},"pipeline":{"clients":1,"events":{"active":0,"filtered":78,"published":1818,"retry":50,"total":1896},"queue":{"acked":1818}}},"registrar":{"states":{"current":14,"update":1896},"writes":{"success":52,"total":52}},"system":{"cpu":{"cores":1},"load":{"1":0.32,"15":0.04,"5":0.14,"norm":{"1":0.32,"15":0.04,"5":0.14}}}}}}

Your help is appreciated.

Thanks

[jsoriano (Jaime Soriano)]

Hey @ethical20,

In order to use the dashboards provided by the modules, in general you should be using modules.
Are you using the system module or you are collecting these logs directly with an input?

Modules parse logs, providing more structure data, and also add these event.module and event.dataset fields you are missing.

[ethical20]

Hi,

I'm not using modules, I'm collecting these logs directly . As i've read that using modules is not a must.

I used not to use modules before and everything was fine!

[jsoriano (Jaime Soriano)]

@ethical20

I'm not using modules, I'm collecting these logs directly . As i've read that using modules is not a must.

Yes, you are right, using modules is not a must, but they help parsing logs and providing additional metadata in events. For example most of the fields that appear empty in this events view, are provided by modules.
event.module and event.dataset are filled by beats when using modules.
user.name, source.ip and destination.ip are usually parsed from logs. This parsing is usually done with modules, but you can also processors, logstash or ingest pipelines to do it without modules.
Most filebeat dashboards require these fields to be able to display some data, they are usually thought to be used with modules.

Even if you don't use modules or do any parsing, you can still use the discover or logs UI views to visualize the logs, but they won't contain a lot of structured data, that is specially useful in the SIEM app for example.

@ethical20

I used not to use modules before and everything was fine!

What do you mean by fine? Did you have the missing fields? Were you parsing with Logstash or your own ingest pipelines?