Filebeat not using new pipeline

Hi, I have created the following pipeline in the dev tools menu

PUT _ingest/pipeline/filebeat-ssologin
  "processors": [
      "dissect": {
        "field": "message",
        "pattern": "%{date} [%{log.logger}] %{log.level} %{log.origin.function} - %{message}"
      "date": {
        "field": "date",
        "formats": [
      "remove": {
        "field": "date"

This is working fine when I simulate with the following log

POST _ingest/pipeline/filebeat-ssologin/_simulate
{"docs":[{"_source":{"message":"15:55:32,864 [12] INFO SsoBrowserPopUpLib.SsoBrowserPopUp+<>c - Time Out in 290 seconds."}}]}

And I have set this pipleine in my filebeat.yml file, and stopped and restarted filebeat

pipeline: "filebeat-ssologin"

But after this, I'm not getting any logs coming through at all. I hope that I can get some help here getting my logs working with my new pipeline.


Some additional info. From the filebeat running on my local machine, I can't see any obvious error on why it's not sending the data

2020-02-07T16:28:45.645Z        DEBUG   [input] input/input.go:152      Run input
2020-02-07T16:28:45.645Z        DEBUG   [input] log/input.go:187        Start next scan
2020-02-07T16:28:45.646Z        DEBUG   [input] log/input.go:417        Check file for harvesting: C:\something\something.log
2020-02-07T16:28:45.646Z        DEBUG   [input] log/input.go:507        Update existing file for harvesting: C:\something\something.log, offset: 164405
2020-02-07T16:28:45.646Z        DEBUG   [input] log/input.go:559        Harvester for file is still running: C:\something\something.log
2020-02-07T16:28:45.646Z        DEBUG   [input] log/input.go:208        input states cleaned up. Before: 1, After: 1, Pending: 0
2020-02-07T16:28:45.675Z        DEBUG   [multiline]     multiline/multiline.go:175      Multiline event flushed because timeout reached.
2020-02-07T16:28:45.675Z        DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-02-07T16:28:40.669Z",
  "@metadata": {
    "beat": "",
    "type": "_doc",
    "version": "",
    "pipeline": "filebeat-ssologin"
  "log": {
    "offset": 164405,
    "file": {
      "path": "C:\\something\\something.log"
  "message": "something ",
  "input": {
    "type": "log"
  "ecs": {
    "version": "1.0.0"
  "host": {
    "name": "something"
  "agent": {
    "hostname": "something",
    "id": "47268521-67cf-4b2b-a24c-f17020bc8cb8",
    "version": "7.1.1",
    "type": "filebeat",
    "ephemeral_id": "301f8140-10ce-4716-9ca9-8e4830e2e873"
2020-02-07T16:28:46.775Z        DEBUG   [elasticsearch] elasticsearch/client.go:338     PublishEvents: 1 events have been published to elasticsearch in 98.2554ms.
2020-02-07T16:28:46.776Z        DEBUG   [publisher]     memqueue/ackloop.go:160 ackloop: receive ack [211: 0, 1]
2020-02-07T16:28:46.777Z        DEBUG   [publisher]     memqueue/eventloop.go:535       broker ACK events: count=1, start-seq=328, end-seq=328

2020-02-07T16:28:46.777Z        DEBUG   [publisher]     memqueue/ackloop.go:128 ackloop: return ack to broker loop:1
2020-02-07T16:28:46.777Z        DEBUG   [publisher]     memqueue/ackloop.go:131 ackloop:  done send ack
2020-02-07T16:28:46.777Z        DEBUG   [acker] beater/acker.go:64      stateful ack    {"count": 1}
2020-02-07T16:28:46.777Z        DEBUG   [registrar]     registrar/registrar.go:356      Processing 1 events
2020-02-07T16:28:46.777Z        DEBUG   [registrar]     registrar/registrar.go:326      Registrar state updates processed. Count: 1
2020-02-07T16:28:46.777Z        DEBUG   [registrar]     registrar/registrar.go:411      Write registry file: C:\Users\something\Desktop\Elastic\filebeat-7.1.1-windows-x86_64\data\registry\filebeat\data.json
2020-02-07T16:28:46.795Z        DEBUG   [registrar]     registrar/registrar.go:404      Registry file updated. 4 states written.
2020-02-07T16:28:47.677Z        DEBUG   [harvester]     log/log.go:102  End of file reached: C:\something\something.log; Backoff now.
2020-02-07T16:28:55.647Z        DEBUG   [input] input/input.go:152      Run input

After some reading I thought the issue may be the location where I've set the pipeline in the yml file, was originally

pipeline: "filebeat-ssologin"

I've now moved it to

pipeline: "filebeat-ssologin"

But it doesn't look like that has fixed the issue, or changed anything in the filebeat logs, issue remains.

Wasn't getting anywhere using the dissect. So switched to using grok instead. And that works

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.