Filebeat forward to Kibana ssh auth fail

[mcoa (miguel)]

Hello,
I've filebeat and system module for check ssh auth but in Kibana dont register the ssh action.

My filebeat.yml:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/secure
    - /var/log/messages
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.dashboards.enabled: true
setup.kibana:
  host: "192.168.x.x:5601"
output.elasticsearch:
  hosts: ["192.168.x.x:9200"]

and the system module:

- module: system
  syslog:
    enabled: true
    var.paths: ["/var/log/messages"]
  auth:
    enabled: true
    var.paths: ["/var/log/secure"]

My Index Filebeat-* is fine:

But "Discover" i dont show *ssh* fields .

¿what's wrong?

Thanks.

[Tek_Chand]

@Miguel, Are you getting SSH logs under filebeat index?
If yes then make some activity regarding SSH on your remote machine and check then at kibana end or try to extend the time period for which you are watching the logs on Top Right hand in kibana.

Thanks.

[mcoa (miguel)]

Hello @Tek_Chand,
Yes the filebeat "capture" the ssh login with debug mode:

"@timestamp": "2018-10-10T13:39:05.236Z",
"@metadata": {
  "beat": "filebeat",
  "type": "doc",
  "version": "6.4.1"
},
"offset": 36614,
"message": "Oct 10 10:38:56 test-1 sshd[12178]: Accepted password for mcoa from ::1 port 37236 ssh2",
"prospector": {
  "type": "log"
},
"input": {
  "type": "log"
},
"beat": {
  "version": "6.4.1",
  "name": "test-01.example.com",
  "hostname": "test-01.example.com"
},
"host": {
  "name": "test-01.example.com"
},
"source": "/var/log/secure"
}
2018-10-10T10:39:05.237-0300	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
"@timestamp": "2018-10-10T13:39:05.237Z",
"@metadata": {
  "beat": "filebeat",
  "type": "doc",
  "version": "6.4.1"
},
"beat": {
  "version": "6.4.1",
  "name": "test-01.example.com",
  "hostname": "test-01.example.com"
},
"host": {
  "name": "test-01.example.com"
},
"message": "Oct 10 10:38:56 test-1 sshd[12178]: pam_unix(sshd:session): session opened for user mcoa by (uid=0)",
"source": "/var/log/secure",
"offset": 36702,
"prospector": {
  "type": "log"
},
"input": {
  "type": "log"
}
}
2018-10-10T10:38:41.232-0300	DEBUG	[input]	log/input.go:195	input states cleaned up. Before: 1, After: 1, Pending: 0
2018-10-10T10:38:41.232-0300	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://192.168.x.x:9200))
2018-10-10T10:38:41.232-0300	DEBUG	[elasticsearch]	elasticsearch/client.go:688	ES Ping(url=http://192.168.x.x:9200)
2018-10-10T10:38:41.235-0300	DEBUG	[harvester]	log/log.go:102	End of file reached: /var/log/secure; Backoff now.
2018-10-10T10:38:41.235-0300	DEBUG	[elasticsearch]	elasticsearch/client.go:711	Ping status code: 200
2018-10-10T10:38:41.235-0300	INFO	elasticsearch/client.go:712	Connected to Elasticsearch version 6.4.2
2018-10-10T10:38:41.235-0300	DEBUG	[elasticsearch]	elasticsearch/client.go:730	HEAD http://192.168.x.x:9200/_template/filebeat-6.4.1  <nil>
2018-10-10T10:38:41.237-0300	INFO	template/load.go:129	Template already exists and will not be overwritten.
2018-10-10T10:38:41.237-0300	INFO	pipeline/output.go:105	Connection to backoff(elasticsearch(http://192.168.x.x:9200)) established
2018-10-10T10:38:41.241-0300	DEBUG	[elasticsearch]	elasticsearch/client.go:321	PublishEvents: 15 events have been published to elasticsearch in 4.368835ms.

This registre is shown from Kibana, but not "parser" with system module:

The strange is the next error:

2018-10-10T10:38:40.257-0300	DEBUG	[processors]	processors/processor.go:66	Processors:
2018-10-10T10:38:40.257-0300	DEBUG	[input]	log/config.go:200	recursive glob enabled
2018-10-10T10:38:40.257-0300	DEBUG	[input]	log/input.go:147	exclude_files: [(?-s:.)gz(?-m:$)]. Number of stats: 6
2018-10-10T10:38:40.257-0300	ERROR	fileset/factory.go:105	Error creating input: Can only start an input when all related states are finished: {Id:16871871-64768 Finished:false Fileinfo:0xc42044d6c0 Source:/var/log/secure Offset:34972 Timestamp:2018-10-10 10:38:40.239207208 -0300 -03 m=+28.383377111 TTL:-1ns Type:log Meta:map[] FileStateOS:16871871-64768}
2018-10-10T10:38:40.257-0300	ERROR	[reload]	cfgfile/list.go:104	Error creating runner from config: Can only start an input when all related states are finished: {Id:16871871-64768 Finished:false Fileinfo:0xc42044d6c0 Source:/var/log/secure Offset:34972 Timestamp:2018-10-10 10:38:40.239207208 -0300 -03 m=+28.383377111 TTL:-1ns Type:log Meta:map[] FileStateOS:16871871-64768

any suggestions ?

Thanks.

[Tek_Chand]

@Mcoa, I have checked again your filebeat.yml file config in first post. You have enabled the input prospectors and filebeat config module is set to false. So logs are not going through system module.

So make the below changes in your filebeat.yml config file:

filebeat.inputs:
- type: log
  enabled: false
  paths:
#    - /var/log/secure
#   - /var/log/messages
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
setup.dashboards.enabled: true
setup.kibana:
  host: "192.168.x.x:5601"
output.elasticsearch:
  hosts: ["192.168.x.x:9200"]

Please let me know if you still face any problem.

Thanks.

[mcoa (miguel)]

Hello @Tek_Chand, I changed my config file:

filebeat.inputs:
- type: log
  enabled: false
  paths:
#    - /var/log/secure
#    - /var/log/message
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
setup.dashboards.enabled: true
setup.kibana:
  host: "192.168.x.x:5601"
output.elasticsearch:
  hosts: ["192.168.x.x:9200"]

And my system module

[root@test-1 filebeat]# cat modules.d/system.yml
- module: system
  auth:
    enabled: true
    var.paths: ["/var/log/secure"]

But, i've the same behavior. Is strange, because can see the json by Kibana, but isn't with system module flag.

Regards.

[Tek_Chand]

@Mcoa, Now its really strange. Can you please give me the output of below command:

$filebeat modules list

One more question for you..how you enable the filebeat system module?
I am hope so you have used the below command to enable the filebeat module

$filebeat modules enable system

Thanks.

[mcoa (miguel)]

Hello @Tek_Chand,
The output command:

[root@web-1 ~]# filebeat modules list
Enabled:
system

Disabled:
apache2
auditd
elasticsearch
icinga
iis
kafka
kibana
logstash
mongodb
mysql
nginx
osquery
postgresql
redis
traefik

And yes, for enable system module i used that command.

thanks.

[Tek_Chand]

@mcoa, Thanks for your efforts.

According to your current configuration everything seems fine. Its really weird.

Have you restarted the filebeat service after making the changes in filebeat.yml file?

Thanks.

[mcoa (miguel)]

@Tek_Chand , Yes i restarted the filebeat service after any change.

Thanks.

[steffens (Steffen Siering)]

The system module does not parse old logs, as the file has been visited in the past. Newer logs should be parse though.