# ❗ Multiline Pattern Not Working in Filebeat 9.0.1 (Filestream Type Input)

**URL:** <https://discuss.elastic.co/t/multiline-pattern-not-working-in-filebeat-9-0-1-filestream-type-input/378977>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [June 8, 2025, 1:27am UTC](https://discuss.elastic.co/t/multiline-pattern-not-working-in-filebeat-9-0-1-filestream-type-input/378977 "2025-06-08T01:27:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ravi\_Chandran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_chandran/32/143560_2.png) [@Ravi\_Chandran](https://discuss.elastic.co/u/Ravi_Chandran)\
**Post date:** [June 8, 2025, 1:27am UTC](https://discuss.elastic.co/t/multiline-pattern-not-working-in-filebeat-9-0-1-filestream-type-input/378977/1 "2025-06-08T01:27:54Z")

</div>

### **Summary**

Despite attempting multiple valid `multiline.pattern` configurations, Filebeat v9.0.1 fails to parse multiline log entries correctly from a plain text log file located inside a container. The logs contain Python stack traces and standard application logs. Filebeat continues to treat each line as a separate log event, breaking multiline messages across documents.

### 🖥 **Environment Details**

- **Filebeat Version** : 9.0.1
- **Input Type** : `filestream`
- **Deployment** : Docker (running inside a container)
- **Log Source** : Application writes to `/app/log/techsavvyrc.log`
- **Multiline Log Format** : Python stack traces + timestamped log entries

📄 **Sample Log File (`techsavvyrc.log`)**

```auto
    self.raise_routing_exception(req)
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 500, in raise_routing_exception
    raise request.routing_exception # type: ignore[misc]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/ctx.py", line 362, in match_request
    result = self.url_adapter.match(return_rule=True) # type: ignore
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/werkzeug/routing/map.py", line 629, in match
    raise NotFound() from None
werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
2025-06-06 18:07:41,725 [INFO] service=techsavvyrc trace_id=fc54dfdec4be2a556bf7c8da0ed54a16 span_id=0d8069e49e6f7a83 Home page accessed
2025-06-06 18:08:12,803 [ERROR] service=techsavvyrc trace_id=2cf92917d8e8d5afe7846b2929e86277 span_id=92afc27e7bedd256 Unhandled exception: 405 Method Not Allowed: The method is not allowed for the requested URL.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 891, in dispatch_request
    self.raise_routing_exception(req)
  File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 500, in raise_routing_exception
    raise request.routing_exception # type: ignore[misc]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flask/ctx.py", line 362, in match_request
    result = self.url_adapter.match(return_rule=True) # type: ignore
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/werkzeug/routing/map.py", line 624, in match
    raise MethodNotAllowed(valid_methods=list(e.have_match_for)) from None
werkzeug.exceptions.MethodNotAllowed: 405 Method Not Allowed: The method is not allowed for the requested URL.

```

⚙ **filebeat.yml Configuration**

```auto
############################# General Filebeat Settings ##############################

name: "filebeat-techsavvyrc"
tags: ["techsavvyrc-app", "docker", "training", "testing"]
fields:
  environment: "testing"

############################### Paths & Idempotence ###############################

# Path to the directory where this filebeat.yml is located:
path.config: "/usr/share/filebeat"

# Path where Filebeat will store its registry (offsets, state, etc.)
# (Defaults to $path.home/data, but we override explicitly here.)
path.data: "/usr/share/filebeat/data"

# Path where Filebeat writes its own logs.
path.logs: "/usr/share/filebeat/logs_filebeat"

########################## Filebeat Input: filestream ##############################

filebeat.inputs:
  - type: filestream
    id: "techsavvyrc-app-log"
    enabled: true
    paths:
      - "/usr/share/filebeat/logs/techsavvyrc.log"
    multiline:
      type: pattern
      pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} '
      #pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \[[A-Z]+\]'
      #pattern: '^(\d{4}-\d{2}-\d{2})|(\s+File)|(\s+Traceback)'
      #pattern: '^\s'
      #pattern: '^\['
      match: after
      negate: false
      max_lines: 500
      timeout: 10s
      ignore_older: 0s
    processors:
      - add_fields:
          target: ""
          fields:
            service: "techsavvyrc-app"
            log_source: "internal"
      - add_host_metadata: {}

########################### Placeholders for Future Modules ########################

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

############################## Output Configuration #################################

#output.logstash:
# hosts: ["logstash:5044"]
# loadbalance: false
# bulk_max_size: 200
# worker: 1

output.console:
  enabled: true
  pretty: true
  
########################### Buffering & Queue Controls ##############################

queue.mem:
  events: 4096           
  flush.min_events: 1024 
  flush.timeout: 1s      

############################ Filebeat’s Own Logging #################################

logging.level: debug
logging.selectors: ["*multiline*"]
logging.to_files: true
logging.files:
  path: "/usr/share/filebeat/logs_filebeat"
  name: "filebeat.log"
  rotateeverybytes: 10485760
  keepfiles: 7
  permissions: 0644
logging.to_syslog: false
logging.json: false
logging.metrics.enabled: false
logging.metrics.period: 30s

```

🐳 **Docker Compose Snippet**

```auto
  filebeat:
    image: docker.elastic.co/beats/filebeat:${ELK_VERSION}
    container_name: filebeat
    user: root
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - ./techsavvyrc/app/log:/usr/share/filebeat/logs:ro 
    depends_on:
      - logstash
    networks:
      - observability

```

### 🧪 **Multiline Patterns Tried (All Failed)**

We tested all the following patterns with appropriate `negate` and `match` logic, but Filebeat still failed to group multiline events correctly:

```auto
1) pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} '
2) pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \[[A-Z]+\]'
3) pattern: '^(\d{4}-\d{2}-\d{2})|(\s+File)|(\s+Traceback)'
4) pattern: '^\s'
5) pattern: '^\['

```

📋 **Observed Filebeat Console Output**

```auto
{
  "@timestamp": "2025-06-08T01:03:12.929Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "host": {
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ]
  },
  "ecs": {
    "version": "8.0.0"
  },
  "message": "2025-06-08 00:47:02,070 [INFO] service=techsavvyrc trace_id=0f0c92999ee3fb4d4cbd675af83568aa span_id=4f1e9ca612d79253 Home page accessed",
  "log_source": "internal",
  "agent": {
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513"
  },
  "log": {
    "file": {
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512"
    },
    "offset": 57514
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "input": {
    "type": "filestream"
  },
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app"
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "message": "2025-06-08 01:04:06,129 [INFO] service=techsavvyrc trace_id=1b81f97de2fcee75787bc37f36608354 span_id=3775fdb29dda10dc Home page accessed",
  "input": {
    "type": "filestream"
  },
  "service": "techsavvyrc-app",
  "host": {
    "architecture": "x86_64",
    "os": {
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc"
  },
  "log": {
    "offset": 57651,
    "file": {
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4"
    }
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "fields": {
    "environment": "testing"
  },
  "log_source": "internal"
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "message": "2025-06-08 01:04:06,657 [ERROR] service=techsavvyrc trace_id=d3ac98f8ebf989fb4d7acd2b39139c98 span_id=71ae04aba190dd39 Unhandled exception: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.",
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "fields": {
    "environment": "testing"
  },
  "input": {
    "type": "filestream"
  },
  "log_source": "internal",
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945"
  },
  "log": {
    "offset": 57788,
    "file": {
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4"
    }
  },
  "service": "techsavvyrc-app",
  "host": {
    "architecture": "x86_64",
    "os": {
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "log": {
    "file": {
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512"
    },
    "offset": 58064
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "host": {
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false
  },
  "message": "Traceback (most recent call last):",
  "input": {
    "type": "filestream"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "fields": {
    "environment": "testing"
  },
  "host": {
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d"
  },
  "agent": {
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1"
  },
  "message": " File \"/usr/local/lib/python3.11/site-packages/flask/app.py\", line 917, in full_dispatch_request",
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "ecs": {
    "version": "8.0.0"
  },
  "log": {
    "offset": 58099,
    "file": {
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4"
    }
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "input": {
    "type": "filestream"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "message": " rv = self.dispatch_request()",
  "log": {
    "offset": 58197,
    "file": {
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4"
    }
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "log_source": "internal",
  "agent": {
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc"
  },
  "input": {
    "type": "filestream"
  },
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "host": {
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc"
  },
  "log": {
    "offset": 58230,
    "file": {
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4"
    }
  },
  "message": " ^^^^^^^^^^^^^^^^^^^^^^^",
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "input": {
    "type": "filestream"
  },
  "host": {
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ]
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "log": {
    "offset": 58263,
    "file": {
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512"
    }
  },
  "input": {
    "type": "filestream"
  },
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "host": {
    "architecture": "x86_64",
    "os": {
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "message": " File \"/usr/local/lib/python3.11/site-packages/flask/app.py\", line 891, in dispatch_request",
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "agent": {
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "message": " self.raise_routing_exception(req)",
  "service": "techsavvyrc-app",
  "agent": {
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513"
  },
  "log": {
    "offset": 58356,
    "file": {
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512"
    }
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "input": {
    "type": "filestream"
  },
  "fields": {
    "environment": "testing"
  },
  "log_source": "internal",
  "host": {
    "os": {
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "log": {
    "offset": 58394,
    "file": {
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000"
    }
  },
  "message": " File \"/usr/local/lib/python3.11/site-packages/flask/app.py\", line 500, in raise_routing_exception",
  "input": {
    "type": "filestream"
  },
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "host": {
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d"
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "log_source": "internal",
  "agent": {
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513"
  },
  "ecs": {
    "version": "8.0.0"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "log": {
    "offset": 58494,
    "file": {
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512"
    }
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "agent": {
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "message": " raise request.routing_exception # type: ignore[misc]",
  "input": {
    "type": "filestream"
  },
  "host": {
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow"
    }
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "log": {
    "offset": 58552,
    "file": {
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log"
    }
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "input": {
    "type": "filestream"
  },
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "host": {
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow"
    }
  },
  "message": " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^",
  "fields": {
    "environment": "testing"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "agent": {
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1"
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "input": {
    "type": "filestream"
  },
  "service": "techsavvyrc-app",
  "log_source": "internal",
  "agent": {
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "message": " File \"/usr/local/lib/python3.11/site-packages/flask/ctx.py\", line 362, in match_request",
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "fields": {
    "environment": "testing"
  },
  "host": {
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64",
    "os": {
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ]
  },
  "log": {
    "offset": 58588,
    "file": {
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log"
    }
  }
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "log": {
    "offset": 58678,
    "file": {
      "device_id": "64512",
      "inode": "21144000",
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log"
    }
  },
  "input": {
    "type": "filestream"
  },
  "log_source": "internal",
  "host": {
    "os": {
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow",
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64"
  },
  "agent": {
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc",
    "type": "filebeat",
    "version": "9.0.1"
  },
  "message": " result = self.url_adapter.match(return_rule=True) # type: ignore",
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app"
}
{
  "@timestamp": "2025-06-08T01:04:06.946Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.0.1"
  },
  "tags": [
    "techsavvyrc-app",
    "docker",
    "training",
    "testing"
  ],
  "input": {
    "type": "filestream"
  },
  "fields": {
    "environment": "testing"
  },
  "service": "techsavvyrc-app",
  "ecs": {
    "version": "8.0.0"
  },
  "log": {
    "offset": 58748,
    "file": {
      "fingerprint": "8197fc07ed22b6ff1b9e22274649b6ebe92af7af2e7bc19fcf075651e3cca7c4",
      "path": "/usr/share/filebeat/logs/techsavvyrc.log",
      "device_id": "64512",
      "inode": "21144000"
    }
  },
  "message": " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^",
  "log_source": "internal",
  "host": {
    "os": {
      "type": "linux",
      "platform": "rhel",
      "version": "9.5 (Plow)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux",
      "kernel": "5.15.0-307.178.5.el9uek.x86_64",
      "codename": "Plow"
    },
    "id": "3d9d42d5c57d9921c87b9df2e384013d",
    "containerized": false,
    "ip": [
      "172.17.0.2"
    ],
    "mac": [
      "E2-62-79-36-C4-3A"
    ],
    "name": "96eecd0e2661",
    "hostname": "96eecd0e2661",
    "architecture": "x86_64"
  },
  "agent": {
    "type": "filebeat",
    "version": "9.0.1",
    "ephemeral_id": "1f94efb3-f6b6-4b45-b427-d963b907f513",
    "id": "03c75959-aa81-4812-b6a6-f2477d26b945",
    "name": "filebeat-techsavvyrc"
  }
}

```

✅ **Steps Tried**

- Verified the log file is accessible inside the container.
- Confirmed format: each exception stack trace is indented, and new entries start with a clear timestamp.
- Cleared registry:  
`rm -rf /usr/share/filebeat/data/registry`
- Enabled debug:  
`filebeat -e -d "multiline"`

### ❓ **Expected Behavior**

Filebeat should group stack traces and related lines with the original timestamped log line as one cohesive event.

### ❌ **Actual Behavior**

Each line, including indented traceback lines, is sent as a separate log event. No multiline grouping occurs.

* * *

### 🙏 **Request for Help**

Please assist with:

- Identifying if this is a regression or behavioral change in Filebeat v9.0.1.
- Confirming whether the registry, encoding, or buffering logic changed.
- Suggesting a working multiline pattern or workaround.

### 🐳 **Docker Commands Used**

```auto
1) docker run --rm -it -v "$(pwd)/filebeat/filebeat.yml":/usr/share/filebeat/filebeat.yml -v "$(pwd)/techsavvyrc/app/log":/usr/share/filebeat/logs docker.elastic.co/beats/filebeat:9.0.1 -e -d "multiline"
2) docker run --rm -it -v "$(pwd)/filebeat/filebeat.yml":/usr/share/filebeat/filebeat.yml -v "$(pwd)/techsavvyrc/app/log":/usr/share/filebeat/logs docker.elastic.co/beats/filebeat:9.0.1 bash
3) docker logs -f filebeat
4) docker exec -it filebeat /bin/bash

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 8, 2025, 2:53am UTC](https://discuss.elastic.co/t/multiline-pattern-not-working-in-filebeat-9-0-1-filestream-type-input/378977/2 "2025-06-08T02:53:01Z")

</div>

Hi @Ravi_Chandran Welxomne to the community ....

You are using the wrong syntax

> [@Ravi\_Chandran](#):
>
> ```auto
> filebeat.inputs:
> - type: filestream <<< FILESTREAM 
> id: "techsavvyrc-app-log"
> enabled: true
> paths:
> - "/usr/share/filebeat/logs/techsavvyrc.log"
> multiline: <<<< WRONG SYNTAX uses parsers sytax 
> type: pattern
> pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} '
> #pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} \[[A-Z]+\]'
> #pattern: '^(\d{4}-\d{2}-\d{2})|(\s+File)|(\s+Traceback)'
> #pattern: '^\s'
> #pattern: '^\['
> match: after
> negate: false << Probably should be true
> max_lines: 500
> timeout: 10s
> ignore_older: 0s
> 
> ```

See [here](https://www.elastic.co/docs/reference/beats/filebeat/multiline-examples#_examples_of_multiline_configuration) should use the `parsers` syntax

> Please note that the example below only works with `filestream` input, and not with `log` input.
> 
> ```auto
> parsers:
> - multiline:
> type: pattern
> pattern: '^\['
> negate: true
> match: after
> 
> ```

Your is probably something like

```auto
parsers:
- multiline:
    type: pattern
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} '
    negate: true
    match: after

```

---

<div class="post-metadata">

**Author:** ![Ravi\_Chandran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_chandran/32/143560_2.png) [@Ravi\_Chandran](https://discuss.elastic.co/u/Ravi_Chandran)\
**Post date:** [June 15, 2025, 8:47am UTC](https://discuss.elastic.co/t/multiline-pattern-not-working-in-filebeat-9-0-1-filestream-type-input/378977/3 "2025-06-15T08:47:09Z")

</div>

Hi @Stephenb,

Thank you so much for the quick response — your suggestion worked like a charm and was truly a **saviour**! 🙌

I had been struggling with various multiline patterns under `filestream` without success, but placing the configuration under the `parsers:` section, as you recommended, **immediately resolved the issue**. This saved me a lot of time and confusion.

For others facing similar challenges, I’m sharing a **working setup** below using **Filebeat 9.0.1** , **Logstash** , and **Docker Compose** , along with a link to my GitHub repository that contains all supporting files used in my local observability test setup.

```auto
⚠️ Disclaimer:
This setup is intended only for testing and learning purposes. It lacks security configurations (such as TLS, authentication, access control, etc.) and must not be used in a production environment without proper hardening.

```

**✅ Working Setup**

📁 `docker-compose.yml`

```auto
volumes:
  es_data:
  portainer_data:

networks:
  observability:
    driver: bridge

services:

  techsavvyrc-app:
    build: ./techsavvyrc
    container_name: techsavvyrc-app
    ports:
      - 8000:8000
    volumes:
      - ./techsavvyrc/app/log:/app/log
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
    depends_on:
      - otel-collector
      - apm-server
      - prometheus
    logging:
      driver: "json-file"
    networks:
      - observability

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ELK_VERSION}
    container_name: elasticsearch
    ports:
      - 9200:9200
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    volumes:
      - es_data:/usr/share/elasticsearch/data
    networks:
      - observability

  kibana:
    image: docker.elastic.co/kibana/kibana:${ELK_VERSION}
    container_name: kibana
    ports:
      - 5601:5601
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch
    networks:
      - observability

  filebeat:
    image: docker.elastic.co/beats/filebeat:${ELK_VERSION}
    container_name: filebeat
    user: root
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - ./techsavvyrc/app/log:/usr/share/filebeat/logs:ro
    depends_on:
      - logstash
    networks:
      - observability

  logstash:
    image: docker.elastic.co/logstash/logstash:${ELK_VERSION}
    container_name: logstash
    volumes:
      - ./logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro
      - ./logstash/logstash.yml:/usr/share/logstash/config/logstash.yml
    ports:
      - 5044:5044
    depends_on:
      - elasticsearch
    networks:
      - observability
    environment:
      - LS_JAVA_OPTS=-Xms2g -Xmx2g

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
    ports:
      - 9090:9090
    networks:
      - observability

  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    networks:
      - observability

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: otel-collector
    volumes:
      - ./otel/otel-config.yaml:/etc/otelcol/config.yaml:ro
    command: ["--config=/etc/otelcol/config.yaml"]
    ports:
      - "4318:4318"
      - "8888:8888"
      - "8889:8889"
    networks:
      - observability

  portainer:
    image: portainer/portainer-ce
    container_name: portainer
    ports:
      - "9000:9000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer_data:/data
    restart: unless-stopped
    networks:
      - observability

```

📁 `filebeat.yml`

```auto
# ===================================================================================
# Filebeat Configuration File - Version 9.0.1
#
# Description:
# This configuration is designed specifically for a containerized observability setup.
# It captures both application logs and Docker container logs using Filebeat inside
# a Docker environment.
#
# Purpose:
# - Educational and training use only.
# - Not suitable for production environments due to the absence of essential security
# features like TLS encryption, user authentication, and hardened logging.
#
# GitHub Reference:
# Full stack and related files are available here:
# https://github.com/TechSavvyRC/techsavvyrc-demo
#
# Official Documentation:
# https://www.elastic.co/guide/en/beats/filebeat/9.0/filebeat-reference-yml.html
# ===================================================================================

# ================================= General Settings ================================
# Basic identification and tagging for this Filebeat instance.
# Sets the name, adds tags for filtering, and includes custom fields in all events.
name: "filebeat-techsavvyrc" # Custom name for this Filebeat instance
tags: ["techsavvyrc-app", "docker", "training", "testing"] # Tags added to every event
fields: # Additional fields in each event
  environment: "testing" # Custom fields added to all events

# ================================== Path Settings ==================================
# Directory paths where Filebeat stores configuration, data, and its own log files.
# These paths ensure proper file organization within the container.
path.config: "/usr/share/filebeat" # Base path for config
path.data: "/usr/share/filebeat/data" # Path to registry/state
path.logs: "/usr/share/filebeat/logs_filebeat" # Path for internal logs

# =================================== Input Settings ================================
# Defines input sources to harvest logs from files, containers, or other systems.
# Each input can include parsing rules (e.g., multiline handling) and preprocessing
# logic (e.g., enrichment via processors).
# Supported input types include: filestream, docker, journald, tcp/udp, etc.
filebeat.inputs:

  # Input to monitor application log file from a mounted path inside the container.
  # Includes multiline parsing for stack traces and adds custom metadata to events.
  - type: filestream # Recommended in Filebeat 9.x
    id: "techsavvyrc-app-log" # Unique ID for the input
    enabled: true # Enable this input
    paths:
      - "/usr/share/filebeat/logs/techsavvyrc-app.log" # Mounted log file path
    # Multiline parsing for Python stack traces
    parsers:
      - multiline:
          type: pattern
          pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} ' # Timestamp pattern
          match: after # Append continuation lines *after* matched lines
          negate: true # Treat non-matching lines as continuation
          max_lines: 500 # Max lines per event
          timeout: 10s # Flush if no new line within 10 seconds
          #ignore_older: 0s # Do not skip old files
    # Event enrichment
    processors:
      - add_fields: # Add custom fields to events
          target: ""
          fields:
            service: "techsavvyrc-app"
            log_source: "internal"
      - add_host_metadata: {} # Adds host metadata (OS, hostname, etc.)

  # Streams logs from all Docker containers (mounted via /var/lib/docker/containers)
  # Docker log files are JSON formatted. We decode JSON here.
  - type: filestream
    id: "docker-containers"
    enabled: true
    paths:
      - /var/lib/docker/containers/*/*.log # All container logs
    symlinks: true # Follow symlinks (just in case)
    parsers:
      - ndjson:
          keys_under_root: true # Place JSON keys at root level
          overwrite_keys: true # Overwrite conflicting keys like "message"
    processors:
      - add_fields:
          target: ""
          fields:
            service: "docker-container" # Mark logs as from docker container
            log_source: "docker"
      - add_docker_metadata: # Adds container ID, name, labels, etc.
          host: "unix:///var/run/docker.sock"
          match_source: true

# ================================== Module Settings ================================
# Configuration for Filebeat modules (nginx, system, etc.).
# Currently disabled but ready for future use - modules provide pre-built configs for common services.
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false # Enable for dynamic module loading
  reload.period: 10s # How often to check for module file changes

# ================================= Output Settings =================================
# Configuration for where to send processed log events.
# Primary output is Logstash, with alternative outputs commented for easy activation.
# Primary output to Logstash
#output.logstash:
# hosts: ["logstash:5044"] # Logstash listening on port 5044
# loadbalance: false # Set true for multi-node logstash
# bulk_max_size: 200 # Max events per bulk request
# worker: 1 # Number of concurrent workers

# Direct output to Elasticsearch, uncomment if used without Logstash.
# output.elasticsearch:
# hosts: ["http://elasticsearch:9200"] # ES URL
# index: "techsavvyrc-%{+yyyy.MM.dd}" # Dynamic index name
# pipeline: "filebeat-%{[agent.version]}-pipeline" # Optional ingest pipeline
# username: "elastic" # Auth user
# password: "changeme" # Auth password
# ssl.enabled: false # Enable for HTTPS
# bulk_max_size: 200
# worker: 1
output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  indices:
    - index: "techsavvyrc-app-%{+yyyy.MM.dd}" # For app logs
      when.equals:
        fields.service: "techsavvyrc-app"
    - index: "docker-containers-%{+yyyy.MM.dd}" # For container logs
      when.equals:
        fields.service: "docker-containers"

# Console output for debugging:
#output.console:
# enabled: true
# pretty: true

# ================================= Queue Settings ==================================
# In-memory queue configuration for buffering events before output.
# Controls memory usage and batching behavior for optimal performance.
queue.mem:
  events: 4096 # maximum events to keep in memory (default ~4096)
  flush.min_events: 1024 # force a flush to output when ≥1024 events are ready
  flush.timeout: 1s # force a flush if no new events after 1s

# =============================== Dashboard Setup ===============================
# Kibana dashboard and index template setup configuration.
# When enabled, automatically creates dashboards and templates during setup phase.
# Kibana dashboard configuration (uncomment to enable)
# setup.dashboards.enabled: true
# setup.dashboards.index: ".filebeat" # where dashboards are stored in ES
# setup.dashboards.overwrite: false # set to true to re-import every setup run
# setup.dashboards.directory: "/path/to/your/custom/dashboards"

# setup.kibana:
# hosts: ["http://kibana:5601"] # Kibana endpoint
# username: "kibana_system" # Optional authentication
# password: "changeme"

# ================================ Logging Settings ===============================
# Configuration for Filebeat's own internal logging.
# Controls log levels, file rotation, and where Filebeat writes its operational logs.
logging.level: debug # Options: debug, info, warning, error, critical
logging.selectors: ["*multiline*"] # Focus on multiline logs
logging.to_files: true
logging.files:
  path: "/usr/share/filebeat/logs_filebeat" # Log file location
  name: "filebeat.log"
  rotateeverybytes: 10485760 # Rotate every 10 MB
  keepfiles: 7 # Keep last 7 files before deletion
  permissions: 0644
logging.to_syslog: false
logging.json: false
logging.metrics.enabled: false
logging.metrics.period: 30s

# ============================== Monitoring Settings ==============================
# Self-monitoring configuration for Filebeat performance metrics.
# When enabled, sends Filebeat's own operational metrics to Elasticsearch for monitoring.
monitoring.enabled: false

# To send to Elasticsearch:
# monitoring.elasticsearch:
# hosts: ["http://elasticsearch:9200"]
# To register monitoring dashboards in Kibana:
# monitoring.kibana:
# hosts: ["http://kibana:5601"]
# username: "kibana_system"
# password: "changeme"
# If you have Elastic Agent, disable Filebeat’s built-in monitoring:
# monitoring.elastic_agent.enabled: true

```

My Git Repo: [TechSavvyRC GIT Repo](https://github.com/TechSavvyRC/techsavvyrc-demo/tree/main)