Filebeat not using new pipeline

Elastic Stack Beats
1

Hi, I have created the following pipeline in the dev tools menu

PUT _ingest/pipeline/filebeat-ssologin
{
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": "%{date} [%{log.logger}] %{log.level} %{log.origin.function} - %{message}"
      }
    },
    {
      "date": {
        "field": "date",
        "formats": [
          "HH:mm:ss,SSS"
        ]
      }
    },
    {
      "remove": {
        "field": "date"
      }
    }
  ]
}

This is working fine when I simulate with the following log

POST _ingest/pipeline/filebeat-ssologin/_simulate
{"docs":[{"_source":{"message":"15:55:32,864 [12] INFO SsoBrowserPopUpLib.SsoBrowserPopUp+<>c - Time Out in 290 seconds."}}]}

And I have set this pipleine in my filebeat.yml file, and stopped and restarted filebeat

pipeline: "filebeat-ssologin"

But after this, I'm not getting any logs coming through at all. I hope that I can get some help here getting my logs working with my new pipeline.

Thanks.

2

Some additional info. From the filebeat running on my local machine, I can't see any obvious error on why it's not sending the data

2020-02-07T16:28:45.645Z        DEBUG   [input] input/input.go:152      Run input
2020-02-07T16:28:45.645Z        DEBUG   [input] log/input.go:187        Start next scan
2020-02-07T16:28:45.646Z        DEBUG   [input] log/input.go:417        Check file for harvesting: C:\something\something.log
2020-02-07T16:28:45.646Z        DEBUG   [input] log/input.go:507        Update existing file for harvesting: C:\something\something.log, offset: 164405
2020-02-07T16:28:45.646Z        DEBUG   [input] log/input.go:559        Harvester for file is still running: C:\something\something.log
2020-02-07T16:28:45.646Z        DEBUG   [input] log/input.go:208        input states cleaned up. Before: 1, After: 1, Pending: 0
2020-02-07T16:28:45.675Z        DEBUG   [multiline]     multiline/multiline.go:175      Multiline event flushed because timeout reached.
2020-02-07T16:28:45.675Z        DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-02-07T16:28:40.669Z",
  "@metadata": {
    "beat": "",
    "type": "_doc",
    "version": "",
    "pipeline": "filebeat-ssologin"
  },
  "log": {
    "offset": 164405,
    "file": {
      "path": "C:\\something\\something.log"
    }
  },
  "message": "something ",
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "name": "something"
  },
  "agent": {
    "hostname": "something",
    "id": "47268521-67cf-4b2b-a24c-f17020bc8cb8",
    "version": "7.1.1",
    "type": "filebeat",
    "ephemeral_id": "301f8140-10ce-4716-9ca9-8e4830e2e873"
  }
}
2020-02-07T16:28:46.775Z        DEBUG   [elasticsearch] elasticsearch/client.go:338     PublishEvents: 1 events have been published to elasticsearch in 98.2554ms.
2020-02-07T16:28:46.776Z        DEBUG   [publisher]     memqueue/ackloop.go:160 ackloop: receive ack [211: 0, 1]
2020-02-07T16:28:46.777Z        DEBUG   [publisher]     memqueue/eventloop.go:535       broker ACK events: count=1, start-seq=328, end-seq=328

2020-02-07T16:28:46.777Z        DEBUG   [publisher]     memqueue/ackloop.go:128 ackloop: return ack to broker loop:1
2020-02-07T16:28:46.777Z        DEBUG   [publisher]     memqueue/ackloop.go:131 ackloop:  done send ack
2020-02-07T16:28:46.777Z        DEBUG   [acker] beater/acker.go:64      stateful ack    {"count": 1}
2020-02-07T16:28:46.777Z        DEBUG   [registrar]     registrar/registrar.go:356      Processing 1 events
2020-02-07T16:28:46.777Z        DEBUG   [registrar]     registrar/registrar.go:326      Registrar state updates processed. Count: 1
2020-02-07T16:28:46.777Z        DEBUG   [registrar]     registrar/registrar.go:411      Write registry file: C:\Users\something\Desktop\Elastic\filebeat-7.1.1-windows-x86_64\data\registry\filebeat\data.json
2020-02-07T16:28:46.795Z        DEBUG   [registrar]     registrar/registrar.go:404      Registry file updated. 4 states written.
2020-02-07T16:28:47.677Z        DEBUG   [harvester]     log/log.go:102  End of file reached: C:\something\something.log; Backoff now.
2020-02-07T16:28:55.647Z        DEBUG   [input] input/input.go:152      Run input
3

After some reading I thought the issue may be the location where I've set the pipeline in the yml file, was originally

filebeat.inputs:
pipeline: "filebeat-ssologin"

I've now moved it to

output.elasticsearch:
pipeline: "filebeat-ssologin"

But it doesn't look like that has fixed the issue, or changed anything in the filebeat logs, issue remains.

4

Wasn't getting anywhere using the dissect. So switched to using grok instead. And that works

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.