FIlebeat Output Configuration Error

Elastic Stack Beats
21

1st good! Looks like it is working...

2nd please ... please... please do no past images of text they are borderline unreadable... please paste the texxgt with 3 Back ticks ``` before an after the code... see I can not cut / paste. debug search the imagezx...

The _cat indices shows you have filebeat with 1.9M Documents...

sorry typo... please run this and paste a couple results... in text

GET filebeat-*/_search <<<< Note _

So You have data doe sure.
Go to Kibana Discover make sure you set the timepicker correctly ... you probably have a time zone issue... so we will set a wide rang past and future...

Kibana -> Discover -> Data View Filebeat

Please see this post it will show you how

ignore all the cloud meta data stuff...
you can fix that with commenting these out it is just trying to get metadata that is not there...

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
#  - add_cloud_metadata: ~
#  - add_docker_metadata: ~
#  - add_kubernetes_metadata: ~
22

when I run that command I get this error:

"error": "Incorrect  HTTP method for url [/filebeat-*/search?pretty=true] and method [GET], allowed: [POST]". 
"status": 405
23

Please always show the command + results... remember we are debugging this over the internet...

you ran

GET /filebeat-*/search
................^. << No _ underscore..

SHould be

GET filebeat-*/_search
...............^

Go to Discover... you have data... Just "widen" the time picker...

24

Here are the results.

  1. The @timestamp column should show the time of the event correct? It currently shows the ingest time.
  2. I can widen the time picker and see events but the @timestamp is not showing the times that are in the log file. it is showing the ingest time.
{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-filebeat-8.15.1-2024.09.09-000001",
        "_id": "gDug2JEBgF8Y42E8An5m",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-09-09T21:10:40.650Z",
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "hostname": "ec2amaz-368u98e",
            "architecture": "x86_64",
            "os": {
              "kernel": "10.0.20348.1249 (WinBuild.160101.0800)",
              "build": "20348.1249",
              "type": "windows",
              "platform": "windows",
              "version": "10.0",
              "family": "windows",
              "name": "Windows Server 2022 Datacenter"
            },
            "id": "44405d94-c710-497b-8476-4d25391fad78",
            "name": "ec2amaz-368u98e",
            "ip": [
              "fe80::ac1e:27e5:ff97:130d",
              "10.194.107.122"
            ],
            "mac": [
              "0E-A3-A8-5B-8E-F5"
            ]
          },
          "agent": {
            "ephemeral_id": "e88da111-97ff-4116-a446-dcf2dda72088",
            "id": "06a10110-6721-4cb9-bb50-5bd7582a5076",
            "name": "EC2AMAZ-368U98E",
            "type": "filebeat",
            "version": "8.15.1"
          },
          "cloud": {
            "service": {
              "name": "EC2"
            },
            "image": {
              "id": "ami-0e6760ea2851c035a"
            },
            "instance": {
              "id": "i-0600759f4d3851ef9"
            },
            "machine": {
              "type": "r5.4xlarge"
            },
            "region": "us-east-1",
            "availability_zone": "us-east-1b",
            "account": {
              "id": "606565331724"
            },
            "provider": "aws"
          },
          "log": {
            "offset": 192220192,
            "file": {
              "vol": "383979469",
              "path": """D:\01-evidence\ABZ3542\logs\auth.log""",
              "idxhi": "441057280",
              "idxlo": "17849"
            }
          },
          "message": "Jul 14 01:55:00 Director1 sshd[23412]: reprocess config line 31: Deprecated option RSAAuthentication",
          "input": {
            "type": "filestream"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.15.1-2024.09.09-000001",
        "_id": "gTug2JEBgF8Y42E8An5m",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-09-09T21:10:40.706Z",
          "log": {
            "offset": 192220293,
            "file": {
              "path": """D:\01-evidence\ABZ3542\logs\auth.log""",
              "idxhi": "441057280",
              "idxlo": "17849",
              "vol": "383979469"
            }
          },
          "message": "Jul 14 01:55:00 Director1 sshd[23412]: reprocess config line 38: Deprecated option RhostsRSAAuthentication",
          "input": {
            "type": "filestream"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "mac": [
              "0E-A3-A8-5B-8E-F5"
            ],
            "name": "ec2amaz-368u98e",
            "hostname": "ec2amaz-368u98e",
            "architecture": "x86_64",
            "os": {
              "kernel": "10.0.20348.1249 (WinBuild.160101.0800)",
              "build": "20348.1249",
              "type": "windows",
              "platform": "windows",
              "version": "10.0",
              "family": "windows",
              "name": "Windows Server 2022 Datacenter"
            },
            "id": "44405d94-c710-497b-8476-4d25391fad78",
            "ip": [
              "fe80::ac1e:27e5:ff97:130d",
              "10.194.107.122"
            ]
          },
          "agent": {
            "version": "8.15.1",
            "ephemeral_id": "e88da111-97ff-4116-a446-dcf2dda72088",
            "id": "06a10110-6721-4cb9-bb50-5bd7582a5076",
            "name": "EC2AMAZ-368U98E",
            "type": "filebeat"
          },
          "cloud": {
            "machine": {
              "type": "r5.4xlarge"
            },
            "region": "us-east-1",
            "availability_zone": "us-east-1b",
            "account": {
              "id": "606565331724"
            },
            "provider": "aws",
            "service": {
              "name": "EC2"
            },
            "image": {
              "id": "ami-0e6760ea2851c035a"
            },
            "instance": {
              "id": "i-0600759f4d3851ef9"
            }
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.15.1-2024.09.09-000001",
        "_id": "gjug2JEBgF8Y42E8An5m",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-09-09T21:10:40.706Z",
          "log": {
            "offset": 192220400,
            "file": {
              "idxlo": "17849",
              "vol": "383979469",
              "path": """D:\01-evidence\ABZ3542\logs\auth.log""",
              "idxhi": "441057280"
            }
          },
          "message": "Jul 14 01:55:00 Director1 sshd[23412]: Accepted publickey for vnmshauser from 208.69.233.70 port 50254 ssh2: RSA SHA256:kTgnJC4nBLaz7W7nAmIeHaCVdbbfRtQbBc/IgFRMQCM",
          "input": {
            "type": "filestream"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "id": "44405d94-c710-497b-8476-4d25391fad78",
            "ip": [
              "fe80::ac1e:27e5:ff97:130d",
              "10.194.107.122"
            ],
            "name": "ec2amaz-368u98e",
            "mac": [
              "0E-A3-A8-5B-8E-F5"
            ],
            "hostname": "ec2amaz-368u98e",
            "architecture": "x86_64",
            "os": {
              "platform": "windows",
              "version": "10.0",
              "family": "windows",
              "name": "Windows Server 2022 Datacenter",
              "kernel": "10.0.20348.1249 (WinBuild.160101.0800)",
              "build": "20348.1249",
              "type": "windows"
            }
          },
          "agent": {
            "version": "8.15.1",
            "ephemeral_id": "e88da111-97ff-4116-a446-dcf2dda72088",
            "id": "06a10110-6721-4cb9-bb50-5bd7582a5076",
            "name": "EC2AMAZ-368U98E",
            "type": "filebeat"
          },
          "cloud": {
            "service": {
              "name": "EC2"
            },
            "image": {
              "id": "ami-0e6760ea2851c035a"
            },
            "instance": {
              "id": "i-0600759f4d3851ef9"
            },
            "machine": {
              "type": "r5.4xlarge"
            },
            "region": "us-east-1",
            "availability_zone": "us-east-1b",
            "account": {
              "id": "606565331724"
            },
            "provider": "aws"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.15.1-2024.09.09-000001",
        "_id": "gzug2JEBgF8Y42E8An5m",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-09-09T21:10:40.706Z",
          "cloud": {
            "image": {
              "id": "ami-0e6760ea2851c035a"
            },
            "instance": {
              "id": "i-0600759f4d3851ef9"
            },
            "machine": {
              "type": "r5.4xlarge"
            },
            "region": "us-east-1",
            "availability_zone": "us-east-1b",
            "account": {
              "id": "606565331724"
            },
            "provider": "aws",
            "service": {
              "name": "EC2"
            }
          },
          "log": {
            "offset": 192220564,
            "file": {
              "idxlo": "17849",
              "vol": "383979469",
              "path": """D:\01-evidence\ABZ3542\logs\auth.log""",
              "idxhi": "441057280"
            }
          },
          "message": "Jul 14 01:55:00 Director1 sshd[23412]: pam_unix(sshd:session): session opened for user vnmshauser by (uid=0)",
          "input": {
            "type": "filestream"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "os": {
              "kernel": "10.0.20348.1249 (WinBuild.160101.0800)",
              "build": "20348.1249",
              "type": "windows",
              "platform": "windows",
              "version": "10.0",
              "family": "windows",
              "name": "Windows Server 2022 Datacenter"
            },
            "id": "44405d94-c710-497b-8476-4d25391fad78",
            "ip": [
              "fe80::ac1e:27e5:ff97:130d",
              "10.194.107.122"
            ],
            "mac": [
              "0E-A3-A8-5B-8E-F5"
            ],
            "name": "ec2amaz-368u98e",
            "hostname": "ec2amaz-368u98e",
            "architecture": "x86_64"
          },
          "agent": {
            "version": "8.15.1",
            "ephemeral_id": "e88da111-97ff-4116-a446-dcf2dda72088",
            "id": "06a10110-6721-4cb9-bb50-5bd7582a5076",
            "name": "EC2AMAZ-368U98E",
            "type": "filebeat"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.15.1-2024.09.09-000001",
        "_id": "hDug2JEBgF8Y42E8An5m",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-09-09T21:10:40.706Z",
          "log": {
            "file": {
              "vol": "383979469",
              "path": """D:\01-evidence\ABZ3542\logs\auth.log""",
              "idxhi": "441057280",
              "idxlo": "17849"
            },
            "offset": 192220673
          },
          "message": "Jul 14 01:55:00 Director1 systemd-logind[1566]: New session 308567 of user vnmshauser.",
          "input": {
            "type": "filestream"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "name": "ec2amaz-368u98e",
            "id": "44405d94-c710-497b-8476-4d25391fad78",
            "ip": [
              "fe80::ac1e:27e5:ff97:130d",
              "10.194.107.122"
            ],
            "mac": [
              "0E-A3-A8-5B-8E-F5"
            ],
            "hostname": "ec2amaz-368u98e",
            "architecture": "x86_64",
            "os": {
              "name": "Windows Server 2022 Datacenter",
              "kernel": "10.0.20348.1249 (WinBuild.160101.0800)",
              "build": "20348.1249",
              "type": "windows",
              "platform": "windows",
              "version": "10.0",
              "family": "windows"
            }
          },
          "agent": {
            "type": "filebeat",
            "version": "8.15.1",
            "ephemeral_id": "e88da111-97ff-4116-a446-dcf2dda72088",
            "id": "06a10110-6721-4cb9-bb50-5bd7582a5076",
            "name": "EC2AMAZ-368U98E"
          },
          "cloud": {
            "provider": "aws",
            "service": {
              "name": "EC2"
            },
            "image": {
              "id": "ami-0e6760ea2851c035a"
            },
            "instance": {
              "id": "i-0600759f4d3851ef9"
            },
            "machine": {
              "type": "r5.4xlarge"
            },
            "region": "us-east-1",
            "availability_zone": "us-east-1b",
            "account": {
              "id": "606565331724"
            }
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.15.1-2024.09.09-000001",
        "_id": "hTug2JEBgF8Y42E8An5m",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-09-09T21:10:40.706Z",
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "architecture": "x86_64",
            "os": {
              "version": "10.0",
              "family": "windows",
              "name": "Windows Server 2022 Datacenter",
              "kernel": "10.0.20348.1249 (WinBuild.160101.0800)",
              "build": "20348.1249",
              "type": "windows",
              "platform": "windows"
            },
            "id": "44405d94-c710-497b-8476-4d25391fad78",
            "ip": [
              "fe80::ac1e:27e5:ff97:130d",
              "10.194.107.122"
            ],
            "mac": [
              "0E-A3-A8-5B-8E-F5"
            ],
            "name": "ec2amaz-368u98e",
            "hostname": "ec2amaz-368u98e"
          },
          "agent": {
            "ephemeral_id": "e88da111-97ff-4116-a446-dcf2dda72088",
            "id": "06a10110-6721-4cb9-bb50-5bd7582a5076",
            "name": "EC2AMAZ-368U98E",
            "type": "filebeat",
            "version": "8.15.1"
          },
          "cloud": {
            "machine": {
              "type": "r5.4xlarge"
            },
            "region": "us-east-1",
            "availability_zone": "us-east-1b",
            "account": {
              "id": "606565331724"
            },
            "provider": "aws",
            "service": {
              "name": "EC2"
            },
            "image": {
              "id": "ami-0e6760ea2851c035a"
            },
            "instance": {
              "id": "i-0600759f4d3851ef9"
            }
          },
          "log": {
            "offset": 192220760,
            "file": {
              "path": """D:\01-evidence\ABZ3542\logs\auth.log""",
              "idxhi": "441057280",
              "idxlo": "17849",
              "vol": "383979469"
            }
          },
          "message": "Jul 14 01:55:00 Director1 systemd: pam_unix(systemd-user:session): session opened for user vnmshauser by (uid=0)",
          "input": {
            "type": "filestream"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.15.1-2024.09.09-000001",
        "_id": "hjug2JEBgF8Y42E8An5m",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-09-09T21:10:40.706Z",
          "host": {
            "name": "ec2amaz-368u98e",
            "id": "44405d94-c710-497b-8476-4d25391fad78",
            "ip": [
              "fe80::ac1e:27e5:ff97:130d",
              "10.194.107.122"
            ],
            "mac": [
              "0E-A3-A8-5B-8E-F5"
            ],
            "hostname": "ec2amaz-368u98e",
            "architecture": "x86_64",
            "os": {
              "kernel": "10.0.20348.1249 (WinBuild.160101.0800)",
              "build": "20348.1249",
              "type": "windows",
              "platform": "windows",
              "version": "10.0",
              "family": "windows",
              "name": "Windows Server 2022 Datacenter"
            }
          },
          "agent": {
            "name": "EC2AMAZ-368U98E",
            "type": "filebeat",
            "version": "8.15.1",
            "ephemeral_id": "e88da111-97ff-4116-a446-dcf2dda72088",
            "id": "06a10110-6721-4cb9-bb50-5bd7582a5076"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "cloud": {
            "service": {
              "name": "EC2"
            },
            "image": {
              "id": "ami-0e6760ea2851c035a"
            },
            "instance": {
              "id": "i-0600759f4d3851ef9"
            },
            "machine": {
              "type": "r5.4xlarge"
            },
            "region": "us-east-1",
            "availability_zone": "us-east-1b",
            "account": {
              "id": "606565331724"
            },
            "provider": "aws"
          },
          "log": {
            "file": {
              "path": """D:\01-evidence\ABZ3542\logs\auth.log""",
              "idxhi": "441057280",
              "idxlo": "17849",
              "vol": "383979469"
            },
            "offset": 192220873
          },
          "message": "Jul 14 01:55:01 Director1 CRON[23432]: pam_unix(cron:session): session opened for user root by (uid=0)",
          "input": {
            "type": "filestream"
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.15.1-2024.09.09-000001",
        "_id": "hzug2JEBgF8Y42E8An5m",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-09-09T21:10:40.706Z",
          "log": {
            "offset": 192220976,
            "file": {
              "vol": "383979469",
              "path": """D:\01-evidence\ABZ3542\logs\auth.log""",
              "idxhi": "441057280",
              "idxlo": "17849"
            }
          },
          "message": "Jul 14 01:55:01 Director1 CRON[23433]: pam_unix(cron:session): session opened for user root by (uid=0)",
          "input": {
            "type": "filestream"
          },
          "agent": {
            "name": "EC2AMAZ-368U98E",
            "type": "filebeat",
            "version": "8.15.1",
            "ephemeral_id": "e88da111-97ff-4116-a446-dcf2dda72088",
            "id": "06a10110-6721-4cb9-bb50-5bd7582a5076"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "hostname": "ec2amaz-368u98e",
            "architecture": "x86_64",
            "name": "ec2amaz-368u98e",
            "os": {
              "platform": "windows",
              "version": "10.0",
              "family": "windows",
              "name": "Windows Server 2022 Datacenter",
              "kernel": "10.0.20348.1249 (WinBuild.160101.0800)",
              "build": "20348.1249",
              "type": "windows"
            },
            "id": "44405d94-c710-497b-8476-4d25391fad78",
            "ip": [
              "fe80::ac1e:27e5:ff97:130d",
              "10.194.107.122"
            ],
            "mac": [
              "0E-A3-A8-5B-8E-F5"
            ]
          },
          "cloud": {
            "instance": {
              "id": "i-0600759f4d3851ef9"
            },
            "machine": {
              "type": "r5.4xlarge"
            },
            "region": "us-east-1",
            "availability_zone": "us-east-1b",
            "account": {
              "id": "606565331724"
            },
            "provider": "aws",
            "service": {
              "name": "EC2"
            },
            "image": {
              "id": "ami-0e6760ea2851c035a"
            }
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.15.1-2024.09.09-000001",
        "_id": "iDug2JEBgF8Y42E8An5m",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-09-09T21:10:40.706Z",
          "cloud": {
            "account": {
              "id": "606565331724"
            },
            "provider": "aws",
            "service": {
              "name": "EC2"
            },
            "image": {
              "id": "ami-0e6760ea2851c035a"
            },
            "instance": {
              "id": "i-0600759f4d3851ef9"
            },
            "machine": {
              "type": "r5.4xlarge"
            },
            "region": "us-east-1",
            "availability_zone": "us-east-1b"
          },
          "log": {
            "offset": 192221079,
            "file": {
              "idxlo": "17849",
              "vol": "383979469",
              "path": """D:\01-evidence\ABZ3542\logs\auth.log""",
              "idxhi": "441057280"
            }
          },
          "message": "Jul 14 01:55:01 Director1 CRON[23433]: pam_unix(cron:session): session closed for user root",
          "input": {
            "type": "filestream"
          },
          "agent": {
            "ephemeral_id": "e88da111-97ff-4116-a446-dcf2dda72088",
            "id": "06a10110-6721-4cb9-bb50-5bd7582a5076",
            "name": "EC2AMAZ-368U98E",
            "type": "filebeat",
            "version": "8.15.1"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "name": "ec2amaz-368u98e",
            "hostname": "ec2amaz-368u98e",
            "architecture": "x86_64",
            "os": {
              "family": "windows",
              "name": "Windows Server 2022 Datacenter",
              "kernel": "10.0.20348.1249 (WinBuild.160101.0800)",
              "build": "20348.1249",
              "type": "windows",
              "platform": "windows",
              "version": "10.0"
            },
            "id": "44405d94-c710-497b-8476-4d25391fad78",
            "ip": [
              "fe80::ac1e:27e5:ff97:130d",
              "10.194.107.122"
            ],
            "mac": [
              "0E-A3-A8-5B-8E-F5"
            ]
          }
        }
      },
      {
        "_index": ".ds-filebeat-8.15.1-2024.09.09-000001",
        "_id": "iTug2JEBgF8Y42E8An5m",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-09-09T21:10:40.706Z",
          "log": {
            "offset": 192221171,
            "file": {
              "vol": "383979469",
              "path": """D:\01-evidence\ABZ3542\logs\auth.log""",
              "idxhi": "441057280",
              "idxlo": "17849"
            }
          },
          "message": "Jul 14 01:55:01 Director1 CRON[23432]: pam_unix(cron:session): session closed for user root",
          "input": {
            "type": "filestream"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "ip": [
              "fe80::ac1e:27e5:ff97:130d",
              "10.194.107.122"
            ],
            "mac": [
              "0E-A3-A8-5B-8E-F5"
            ],
            "hostname": "ec2amaz-368u98e",
            "name": "ec2amaz-368u98e",
            "architecture": "x86_64",
            "os": {
              "type": "windows",
              "platform": "windows",
              "version": "10.0",
              "family": "windows",
              "name": "Windows Server 2022 Datacenter",
              "kernel": "10.0.20348.1249 (WinBuild.160101.0800)",
              "build": "20348.1249"
            },
            "id": "44405d94-c710-497b-8476-4d25391fad78"
          },
          "agent": {
            "type": "filebeat",
            "version": "8.15.1",
            "ephemeral_id": "e88da111-97ff-4116-a446-dcf2dda72088",
            "id": "06a10110-6721-4cb9-bb50-5bd7582a5076",
            "name": "EC2AMAZ-368U98E"
          },
          "cloud": {
            "machine": {
              "type": "r5.4xlarge"
            },
            "region": "us-east-1",
            "availability_zone": "us-east-1b",
            "account": {
              "id": "606565331724"
            },
            "provider": "aws",
            "service": {
              "name": "EC2"
            }, 
            "image": {
              "id": "ami-0e6760ea2851c035a"
            },
            "instance": {
              "id": "i-0600759f4d3851ef9"
            }
          }
        }
      }
    ]
  }
}
25

OK now we are getting there...

So no... because we just used the common filestream it does not know it is syslog etc so it does not know how to parse......

We did this to minimize variables because of the issues...

Now we will use the system module now to properly parse...

So now we will fix that...

  1. Disable the filestream in filebeat.yml
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: false <<<< HERE

Then let's clean up...

  1. In Kibana Stack Management -> Data Stream
    Delete the filebeat-8.15.1 datastream

  2. Clean up the Filebeat registry so it will re-read the files, to do this, delete the contents of the data directory where you installed filebeat

  3. Enable system module

PS > .\filebeat.exe modules enable system

  1. edit modules.d/system.yml not sure if those are system or audit logs... edit the correct one, I assume you know which they are... Put in the single quotes
- module: system
  # Syslog
  syslog:
    enabled: true <<< HERE

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ['D:\01-evidence\ABZ3542\logs\*']
  1. run setup

PS > .\filebeat.exe setup -e

  1. run filebeat

PS > .\filebeat.exe -e

In the end... this is basically the quickstart...

Go look at your data

1 Like
26

Outstanding. This works now. Thanks for your patience and persistence. I appreciate it. Now the fun begins. :slight_smile:

1 Like
27

And now you've learned a lot and you can apply it going forward!

1 Like