Not able to see all the log files which are under filebeat config in kibana

Elastic Stack Beats
1

not able to see all the log files which are under filebeat config in kibana.

In kibana dashboard, choose filebeat-* and in search selected log.file.path.keyword, then selected equals sign i.e : then auto populated list of log files appeared but it is not showing all the log files which are mentioned in filebeat.yml.

Attached is the screen shot from kibana -

We can see tomcat logs and few PingDirectory logs but not others like PingAccess logs etc..

filebeat_logs
filebeat_logs884×395 58.5 KB

filebeat config -

cat /etc/filebeat/filebeat.yml
# Ansible managed


name: app_server1

filebeat.inputs:
    - type: log
      fields_under_root: true
      fields:
         log_type:  developer-portal-api_app_server1
         app_id: node
      paths:
        - /var/log/developer-portal-api/server.log
        - /var/log/developer-portal-api/server-err.log
    - type: log
      fields_under_root: true
      fields:
         log_type:  developer-portal-spa_app_server1
         app_id: node
      paths:
        - /var/log/developer-portal-spa/server.log
        - /var/log/developer-portal-spa/server-err.log
    - type: log
      fields_under_root: true
      fields:
         log_type:  ob-admin-api_app_server1
         app_id: node
      paths:
        - /var/log/ob-admin-api/server.log
        - /var/log/ob-admin-api/server-err.log
    - type: log
      fields_under_root: true
      fields:
         log_type:  ob-admin-spa_app_server1
         app_id: node
      paths:
        - /var/log/ob-admin-spa/server.log
        - /var/log/ob-admin-spa/server-err.log
    - type: log
      fields_under_root: true
      fields:
         log_type:  consent-spa_app_server1
         app_id: node
      paths:
        - /var/log/consent-spa/server.log
        - /var/log/consent-spa/server-err.log
    - type: log
      fields_under_root: true
      fields:
         log_type:  obie-api_app_server1
         app_id: app
      multiline.pattern: ^[[:space:]]+(at|\.{3})\b|^Caused by:|^java|^...|^-
      multiline.negate: false
      multiline.match: after
      paths:
        - /opt/apache-tomcat/logs/catalina.out
    - type: log
      fields_under_root: true
      fields:
         log_type:  pingaccess_app_server1
         app_id: pa
      multiline.pattern: ^[[:space:]]+(at|\.{3})\b|^Caused by:|^java|^...|^-
      multiline.negate: false
      multiline.match: after
      paths:
        - /opt/pingaccess-5.2.0/log/*
    - type: log
      fields_under_root: true
      fields:
         log_type:  pingdirectory_app_server1
         app_id: pd
      multiline.pattern: ^[[:space:]]+(at|\.{3})\b|^Caused by:|^java|^...|^-
      multiline.negate: false
      multiline.match: after
      paths:
        - /opt/PingDirectory/logs/*
    - type: log
      fields_under_root: true
      fields:
         log_type:  pingfederate_app_server1
         app_id: pf
      multiline.pattern: ^[[:space:]]+(at|\.{3})\b|^Caused by:|^java|^...|^-
      multiline.negate: false
      multiline.match: after
      paths:
        - /opt/pingfederate-9.2.0/pingfederate/log/*

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

setup.dashboards.enabled: true
setup.kibana:
  host: "es1_ip:8601"
  username: elastic
  password: xxx


filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true


output.kafka:
  hosts: ['es1_ip:3997', 'es2_ip:3997', 'es3_ip:3997']
  topic: "filebeat"
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  max_message_bytes: 1000000

Can someone point out what can be the problem?

Thanks in Advance.

2

That drop down will only show 10 results and might not show the ones you are specifically looking for. If you continue typing do they start to appear?

Or just put PingAccess in the search bar. Do you get any results?

If not then first I would go back to /opt/pingaccess-5.2.0/log/* and check there are current logs in there within the last 7 days (according to your filter).

3

Thanks @aaron-nimocks for quick help. Its appearing after start typing. It was silly of me to not check first.

1 Like
4

Not silly, I asked the same question before. :slight_smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.