Should the Beat modules only be installed on client?

Mhn · June 26, 2019, 6:30am

Hi everyone,

i'm newbi and i am using elasticsearch , logstash, kibana. I want to send logs from an client to my elastic server (logstash) via FileBeat. Should i install FileBeat modules and activate it only on client or each Beat need to be installed and activate both on client and server ?

Tek_Chand · June 26, 2019, 10:42am

@Mhn,

I am assuming that your architecture is like below:

Filebeat->Logstash->Elasticsaerch->Kibana

Filebeat: Filebeat need to install over all those servers for which you want to send logs over elasticsaerch and can view over kibana dashboard.

Logstash: Logstash is used for extra level parsing of logs.

Elasticsearch: Used to store all the logs for your all servers.

Kibana: Kibana is used to view the logs over GUI dashboard.

So, you need to install Filebeat over only those server for you want send logs over elasticsaerch.

Please let me know if you have any other query.

Thanks.

Mhn · June 26, 2019, 11:54am

@Tek_Chand

thank you for your quick answer. Yes i'm using this method =>
Filebeat->Logstash->Elasticsaerch->Kibana

my question is regards to an issue which I’ve with different OS. I’ve followed this tutorial https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-18-04 and it works everything. Even though after update elastic from
6.4.3 to 7.0.2 was everything OK. But I’ve tried to install on Debian9 and all works but I can see in Dashboards only data from Syslog and not more from Sudo Command and SSH Logins . My config is same as last elastic Installation (Ubuntu 18.04). Do you have any Idea why Dashboard is not working well ?

Tek_Chand · June 26, 2019, 12:08pm

@Mhn,

As per your response you are getting the data from syslog on kibana dashboard not from Sudo and SSH logins.

The SSH logs store in different file. Have you specify that file path in your filebeat.yml file?
Have you define grok filter for auth logs?

Please verify above things.

Thanks.

Mhn · June 26, 2019, 12:21pm

@Tek_Chand

The SSH logs store in different file. Have you specify that file path in your filebeat.yml file?
Have you define grok filter for auth logs?

yes. here is my filebeat.yml =>

 filebeat.inputs:
- type: log
 paths:
    - /var/log/*.log

and here is my filter

filter {
  if [fileset][module] == "system" {
    if [fileset][name] == "auth" {
      grok {
        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
        pattern_definitions => {
          "GREEDYMULTILINE"=> "(.|\n)*"
        }
        remove_field => "message"
      }
      date {
        match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
      geoip {
        source => "[system][auth][ssh][ip]"
        target => "[system][auth][ssh][geoip]"
      }
    }
    else if [fileset][name] == "syslog" {
      grok {
        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
        pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
        remove_field => "message"
      }
      date {
        match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
    }
  }
}

and what i mean with Dashboard you can see in this screenshot :

As i mentioned if I click on another option like Sudo commands or SSH logins, haven't any data.

Tek_Chand · June 26, 2019, 12:30pm

@Mhn,

Are you getting auth log in Index? Are you using filebeat module for auth.log?
Are you storing authlog into filebeat index?
Thanks.

Mhn · June 26, 2019, 12:41pm

@Tek_Chand

Are you getting auth log in Index?

yes =>

Are you using filebeat module for auth.log?
Are you storing authlog into filebeat index?

i'm not sure what do you mean exactly but i think it's so
/etc/filebeat/modules.d/system.yml =>

    - module: system
      # Syslog
      syslog:
        enabled: true
# Authorization logs
  auth:
    enabled: true

Tek_Chand · June 26, 2019, 1:03pm

@Mhn,

You have followed above link and its using filebeat module for system which enable logs for system and auth log.

As per my knowledge when we use logstash we can't use dashboard for auth log due to some pipeline issue. Can you please disable filebeat module once and specify the log file path in filebeat.yml file.

Thanks.

Tek_Chand · June 27, 2019, 5:19am

@Mhn,

Your configuration seems fine. Can you please disable filebeat system module and define in filebeat.yml file like below:

filebeat.inputs:
- type: log
 paths:
    - /var/log/auth.log

as well as replace the grok pattren for auth with below pattern:

["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?", "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}", "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}", "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}", "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}", "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$", "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: (?<system.auth.ssh.event>Accepted) (?<system.auth.ssh.method>publickey) \w+ (?<username>.*)", "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"]

Its working fine at my end.

Thanks.

Mhn · June 27, 2019, 10:54am

it doesn't work if i disable filebeat system module.

system · July 25, 2019, 10:54am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.