SSH and Syslog not reaching on kibana

[Tek_Chand]

Hello Team,

I am using ELK 6.4.0 and beat (Filebeat) 6.4.0. My architecture is Filebeat->Logstash->Elasticsearch->Kibana.

I have installed Filebeat on my 20 servers and sending auth.log, syslog, nginx access.log and application log on kibana using filebeat. I have defined prospector input for paths.

For 15 servers everything is working fine. But on 5 servers nginx access log and application logs are reaching on kibana. But auth.log and syslog are not reaching on kibana dashboard for same 5 servers.

I have verified that the logs are available in auth.log and syslog files on all 5 servers.

Below is my filebeat.yml config:-

#=========================== Filebeat inputs =============================

filebeat.inputs:
- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
#    - /var/log/*.log
    - /var/apps/mobilock/shared/log/production.log
    #- c:\programdata\elasticsearch\logs\*
  exclude_lines: ['^.*health.*$']
  fields_under_root: true
  fields:
    type: application_log
  #multiline.match: after
  multiline.pattern: '[A-Z]{1}, \[[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

- type: log
  enabled: true
  paths:
    - /var/log/syslog
  fields_under_root: true
  fields:
    type: syslog_logs

- type: log
  enabled: true
  paths:
    - /var/log/auth.log
  fields_under_root: true
  fields:
    type: auth_logs

- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  fields_under_root: true
  fields:
    type: nginx_access

I have run the filebeat in debug mode and found that offset is updating whenever we make any activity for auth.log. Its means filebeat is reading the auth.log file properly.
Can you please help me to troubleshoot this strange behavior?
Any assistance will be appreciated.
Thanks.

[Mario_Castro]

Hi @Tek_Chand

It looks like you feel very comfortable with the stack which is very good. I feel it must be something related with permissions on those 5 machines.

I guess that the OS on those machines is automated so that every machine uses exactly the same OS version. I also guess that you're deploying everything using some kind of deployment tool like Ansible or Chef.

It's difficult to guess any error if we can't reproduce the problem, as it's happening to you so I just advise to double check versions, installations, permissions and deployments now. With permissions I also mean to check the user that is reading / writing the logs file as well as the permissions that filebeat user has.

Keep up posted, I'm sorry to not have more answers now.

[Tek_Chand]

@Mario, Thank you for your response.

Finally i found the issue. I did the below steps at my end:-

1. By default Kibana visualize only for 5 servers logs for Filebeat syslog, SSH logs, SUDO commands dashboard. Now i have increase this value in kibana visualize.

2. I was using below filter in my Grok filter for Auth logs:

date {
      match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
   }

Due to this time was changed for logs for these 5 (Because these server in different time zone) servers and show 4 hours in back time and i was searching for 15 minutes (Tried to search in real time).

3. Now i have removed the date filter from my Grok Filter and everything working as expected.

Thanks