Using Filebeat for logging ssh log in

Elastic Stack Beats
1

Hi,
I am using the following configuration,
OS - CentOS Linux release 7.4.1708 (Core)
ES - 6.2
LogStash - 6.2
Kibana - 6.2
FileBeat - 6.2

I was using the filebeat for pushing the logs from the server to the kibana. The enabled modules in filebeat are
system
apache2
auditd

The logs are being shown in the kibana except some logs such as 'ssh logins', 'new user and groups' and 'sudo commands'.
The system module configuration is as follows,

- module: system
  # Syslog
  syslog:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/messages"]

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

  # Authorization logs
  auth:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/secure"]

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

I am not able to view the respective logins and other details as the /var/log/secure is having the data.
Request to advise and thanks in advance.

Regards
Vishnu

2

Are you using Logstash as the output destination for Filebeat? Filebeat modules don't work with Logstash because they require an Elasticsearch Ingest Node pipeline to do the parsing.

3

Hi Andrew,
Thanks for the reply
Yeah... I have done that. I am pushing my logs to the ES Only and I have installed the Required plugins.

Please find the conf for the Filebeat and my ES plugin out.


output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["http://10.10.114.175:9200"]


 ./elasticsearch-plugin list
ingest-geoip
ingest-user-agent

Thanks in advance,
Regards
Vishnu

4

Hi ,
Whether the x-pack installation is required for the same?

Thanks
Vishnu

5

X-Pack is not required by Filebeat modules.

Am I correct in understanding that the logs not being shown are from /var/log/secure?

Can you check to see if you have any logs from that file? In Kibana under Dev Tools -> Console you can run a query like this to see if it returns any hits.

POST /filebeat-*/_search
{
  "size": 10,
  "query": {
    "match": {
      "source": "/var/log/messages"
    }
  }
}

Next make this request in the dev console which will show stats from each ingest pipeline. Look for the system auth pipeline to see how many total events (count) and how many errors (failed).

GET _nodes/stats/ingest

If there's nothing in Elasticsearch then please share your Filebeat logs. Perhaps there some kind of issue on the sending side.

6

Hi Andew,
Thanks for the reply! :slight_smile:

I have gone through the commands. I am getting the results. As I told you before Iam getting the messages log in the kibana except some logs like ssh logins,sudo commands etc. Syslog Dashboard Iam getting and its working fine. The following are the OP.

POST /filebeat-*/_search
{
"size": 10,
"query": {
"match": {
"source": "/var/log/messages"
}
}
}

OutPut:

{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 9,
    "successful": 9,
    "skipped": 0,
    "failed": 0
  },

Ouput for ,

GET _nodes/stats/ingest

{
"_nodes": {
"total": 1,
"successful": 1,
"failed": 0
},

Thanks
Vishnu

7

The JSON output you posted appears to be truncated. The important parts are missing. Can you please share the full responses.

8

Hi,
Please find the OP ,

{
"_nodes": {
"total": 1,
"successful": 1,
"failed": 0
},
"cluster_name": "TEST_GRP",
"nodes": {
"w7vAHqoVSgWDjPa_l1lnZA": {
"timestamp": 1528087416330,
"name": "node-1_175",
"transport_address": "10.10.114.175:9300",
"host": "10.10.114.175",
"ip": "10.10.114.175:9300",
"roles": [
"master",
"data",
"ingest"
],
"ingest": {
"total": {
"count": 811240,
"time_in_millis": 14337,
"current": 0,
"failed": 0
},
"pipelines": {
"filebeat-6.2.4-auditd-log-pipeline": {
"count": 155691,
"time_in_millis": 6781,
"current": 0,
"failed": 0
},
"filebeat-6.2.4-system-syslog-pipeline": {
"count": 446040,
"time_in_millis": 1703,
"current": 0,
"failed": 0
},
"filebeat-6.2.4-apache2-error-pipeline": {
"count": 9,
"time_in_millis": 13,
"current": 0,
"failed": 0
},
"filebeat-6.2.4-system-auth-pipeline": {
"count": 57561,
"time_in_millis": 1263,
"current": 0,
"failed": 0
},
"filebeat-6.2.4-apache2-access-default": {
"count": 62815,
"time_in_millis": 1482,
"current": 0,
"failed": 0
},
"filebeat-6.2.4-postgresql-log-pipeline": {
"count": 89124,
"time_in_millis": 3095,
"current": 0,
"failed": 0
}
}
}
}
}
}

9

It looks like there are no errors in the ingest pipelines for processing the events. And there should be 57k events from the system auth Filebeat module that sends the /var/log/secure data generated on CentOS 7.

Can you confirm that this data is in /var/log/secure on the host?

These dev console queries should return data associated with ssh, sudo, useradd, and groupadd, respectively.

GET filebeat-*/_search?q=_exists_:system.auth.ssh

GET filebeat-*/_search?q=_exists_:system.auth.sudo

GET filebeat-*/_search?q=_exists_:system.auth.useradd

GET filebeat-*/_search?q=_exists_:system.auth.groupadd
1 Like
10

I'm experiencing exactly the same in Centos7.

GET filebeat-*/_search?q=exists:system.auth.sudo

Output correctly parsed hits but the dashboard is empty ( :neutral_face: no results found)

{
"took": 57,
"timed_out": false,
"_shards": {
"total": 6,
"successful": 6,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 113,
"max_score": 1,
"hits": [
{
"_index": "filebeat-6.3.0-2018.06.19",
"_type": "doc",
"_id": "hv3EGGQBEmS5frGURPIU",
"_score": 1,
"_source": {
"offset": 9543,
"prospector": {
"type": "log"
},
"source": "/var/log/secure",
"fileset": {
"module": "system",
"name": "auth"
},
"input": {
"type": "log"
},
"@timestamp": "2018-06-19T17:57:33.000Z",
"system": {
"auth": {
"hostname": "experimental",
"sudo": {
"tty": "pts/1",
"pwd": "/home/luca",
"user": "root",
"command": "/bin/filebeat -e -d publish"
},
"user": "luca",
"timestamp": "Jun 19 17:57:33"
}
},
"beat": {
"hostname": "experimental.box",
"name": "experimental.box",
"version": "6.3.0"
},
"host": {
"name": "experimental.box"
}
}
},
...

11 
{
  "took": 21,
  "timed_out": false,
  "_shards": {
    "total": 102,
    "successful": 102,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 276,
    "max_score": 1,
    "hits": [
      {
        "_index": "filebeat-6.2.4-2018.06.01",
        "_type": "doc",
        "_id": "KKoGumMBctxyZyiWQ7pA",
        "_score": 1,
        "_source": {
          "@timestamp": "2018-05-28T10:25:05.000Z",
          "system": {
            "auth": {
              "hostname": "server01",
              "ssh": {
                "method": "password",
                "port": "49285",
                "ip": "10.212.134.20",
                "event": "Accepted"
              },
              "pid": "13771",
              "user": "root",
              "timestamp": "May 28 10:25:05"
            }
          },
          "offset": 17867,
          "beat": {
            "hostname": "server01",
            "name": "server01",
            "version": "6.2.4"
          },
          "prospector": {
            "type": "log"
          },
          "source": "/var/log/secure",
          "fileset": {
            "module": "system",
            "name": "auth"
          }
        }
      },
      {
        "_index": "filebeat-6.2.4-2018.06.01",
        "_type": "doc",
        "_id": "WKoGumMBctxyZyiWQ7pr",
        "_score": 1,
        "_source": {
          "@timestamp": "2018-05-28T10:40:55.000Z",
          "system": {
            "auth": {
              "hostname": "server01",
              "ssh": {
                "method": "password",
                "port": "63885",
                "ip": "10.212.134.22",
                "event": "Accepted"
              },
              "pid": "13990",
              "user": "root",
              "timestamp": "May 28 10:40:55"
            }
          },
          "offset": 18903,
          "beat": {
            "hostname": "server01",
            "name": "server01",
            "version": "6.2.4"
          },
          "prospector": {
            "type": "log"
          },
          "source": "/var/log/secure",
          "fileset": {
            "module": "system",
            "name": "auth"
          }
        }
      },
      {
        "_index": "filebeat-6.2.4-2018.06.01",
        "_type": "doc",
        "_id": "3aoGumMBctxyZyiWRLu2",
        "_score": 1,
        "_source": {
          "@timestamp": "2018-05-28T12:23:48.000Z",
          "system": {
            "auth": {
              "hostname": "server01",
              "ssh": {
                "method": "password",
                "port": "65438",
                "ip": "10.212.134.20",
                "event": "Accepted"
              },
              "pid": "15072",
              "user": "root",
              "timestamp": "May 28 12:23:48"
            }
          },
          "offset": 23859,
          "beat": {
            "hostname": "server01",
            "name": "server01",
            "version": "6.2.4"
          },
          "prospector": {
            "type": "log"
          },
          "source": "/var/log/secure",
          "fileset": {
            "module": "system",
            "name": "auth"
          }
        }
      },
      {
        "_index": "filebeat-6.2.4-2018.06.01",
        "_type": "doc",
        "_id": "A6oGumMBctxyZyiWRLzk",
        "_score": 1,
        "_source": {
          "@timestamp": "2018-05-28T12:24:26.000Z",
          "system": {
            "auth": {
              "hostname": "server01",
              "ssh": {
                "method": "password",
                "port": "65447",
                "ip": "10.212.134.20",
                "event": "Accepted"
              },
              "pid": "15123",
              "user": "root",
              "timestamp": "May 28 12:24:26"
            }
          },
          "offset": 24493,
          "beat": {
            "hostname": "server01",
            "name": "server01",
            "version": "6.2.4"
          },
          "prospector": {
            "type": "log"
          },
          "source": "/var/log/secure",
          "fileset": {
            "module": "system",
            "name": "auth"
          }
        }
      },
      {
        "_index": "filebeat-6.2.4-2018.06.01",
        "_type": "doc",
        "_id": "OKoGumMBctxyZyiWU8zG",
        "_score": 1,
        "_source": {
          "@timestamp": "2018-05-29T15:27:24.000Z",
          "system": {
            "auth": {
              "hostname": "server01",
              "ssh": {
                "ip": "10.212.134.22",
                "event": "Invalid"
              },
              "pid": "10612",
              "user": "ROOT",
              "timestamp": "May 29 15:27:24"
            }
          },
          "offset": 97697,
          "beat": {
            "hostname": "server01",
            "name": "server01",
            "version": "6.2.4"
          },
          "prospector": {
            "type": "log"
          },
          "source": "/var/log/secure",
          "fileset": {
            "module": "system",
            "name": "auth"
          }
        }
      },
      {
        "_index": "filebeat-6.2.4-2018.06.01",
        "_type": "doc",
        "_id": "SaoGumMBctxyZyiWU8zs",
        "_score": 1,
        "_source": {
          "@timestamp": "2018-05-29T15:27:46.000Z",
          "system": {
            "auth": {
              "hostname": "server01",
              "ssh": {
                "method": "password",
                "port": "61032",
                "ip": "10.212.134.22",
                "event": "Accepted"
              },
              "pid": "10624",
              "user": "root",
              "timestamp": "May 29 15:27:46"
            }
12

Hi @andrewkroh,
I have attached the output for
GET filebeat-*/_search?q=_exists_:system.auth.ssh

Now also Iam not getting any user login details in the dashboard.
Request to advise!
Thanks
Vishnu

13

@vishnuduttpv, I ingested a few of the system.auth.ssh events you posted above to test the [Filebeat System] SSH login attempts dashboard. The board was working for me. The only thing I had to change was to extend the time filter to cover a larger range that included the times when the events occurred. Did you choose a range that included your events?

filebeat-system-auth-ssh-dashboard
filebeat-system-auth-ssh-dashboard.png2694×3016 804 KB

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.