Thread is missing when i send logs from ECS fargate to Elastic search

Elastic Stack Elasticsearch
,
1

Hi! I was running my application in ECS fargate(AWS Service) and in AWS monitoring i was getting logs like 2023-12-06T15:28:53.745+05:30 06-12-2023 09:58:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -

But when i injest this log in Elasticsearch i was not getting the [main] part. I was running this on filebeat/beats container version 7:12:1
and below i my filebeat.yml file

filebeat.inputs:
  - type: log
    paths:
     - /usr/share/filebeat/logs/
    fields:
      type: "service"
    multiline.type: pattern
    multiline.pattern: '^[0-9]{2}-[0-9]{2}-[0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'
    multiline.negate: true
    multiline.match: after
    processors:
      - dissect:
          tokenizer: '%{+Timestamp|string} %{+Timestamp|string} %{Message|string}'
          field: "message"
          target_prefix: "abc.log"
      - if:
          regexp:
            message: ERROR
        then:
          - add_fields:
              target: 'abc.log'
              fields:
                level: ERROR
        else:
        - if:
            regexp:
              message: WARN
          then:
            - add_fields:
                target: 'abc.log'
                fields:
                  level: WARN
          else:
            - add_fields:
                target: 'abc.log'
                fields:
                  level: INFO 

                  
setup.ilm.enabled: false

filebeat.config.modules:  
  reload.enabled: true
  reload.period: 10s
  
setup.template:
  name: "logs-%{[agent.version]}"
  pattern: "logs-%{[agent.version]}-*"
  overwrite: true
  enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.elasticsearch:
  hosts: ["https://url:443"]
  index: "%{[fields.type]:other}_%{+MMyy}_test"
  username: ""
  password: ""
  
setup.kibana:
  host: "localhost:5601"

logging.level: debug
logging.to_files: true
logging.files:
 path: /usr/share/filebeat/logs/
 name: filebeat
 keepfiles: 7
 permissions: 0644
xpack.security.enabled: false
xpack.monitoring.enabled: false
xpack.graph.enabled: false
xpack.watcher.enabled: false
xpack.reporting.enabled: false
1 Like
2

Hi @NitinKalburgii Welcome to the commuity!

So a couple issues... your multiline pattern is incorrect it does not match the format of the time string
and your dissect is also incorrect it does not match the pattern of the message

But not sure why you need multi-line unless you have exceptions perhaps that is your point.

But most important 7.12 is ANCIENT ... you should upgrade with haste! :slight_smile:

Here is the log file I used

2023-12-06T15:28:53.745+05:30 06-12-2023 09:58:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -
exception....yada 1
exception....yada 2
exception....yada 3
exception....yada 4
exception....yada 5
2023-12-06T15:29:53.745+05:30 06-12-2023 09:59:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Completeing ProtocolHandler ["http-nio-8070"] -
2023-12-06T15:30:53.745+05:30 06-12-2023 09:60:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -
2023-12-06T15:31:53.745+05:30 06-12-2023 09:61:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Completeing ProtocolHandler ["http-nio-8070"] -

Here is the filebeat.yml

filebeat.inputs:
  - type: log
    paths:
     - /Users/sbrown/workspace/sample-data/discuss/discuss-filebeat-fargate.log
    fields:
      type: "service"
    multiline.type: pattern
    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'
    multiline.negate: true
    multiline.match: after
    processors:
      - dissect:
          tokenizer: '%{+Timestamp|string} %{+Date|string} %{+Hour|string} %{Message|string}'
          field: "message"
          target_prefix: "abc.log"
      - if:
          regexp:
            message: ERROR
        then:
          - add_fields:
              target: 'abc.log'
              fields:
                level: ERROR
        else:
        - if:
            regexp:
              message: WARN
          then:
            - add_fields:
                target: 'abc.log'
                fields:
                  level: WARN
          else:
            - add_fields:
                target: 'abc.log'
                fields:
                  level: INFO 

                  
setup.ilm.enabled: false

filebeat.config.modules:  
  reload.enabled: true
  reload.period: 10s
  
setup.template:
  name: "logs-%{[agent.version]}"
  pattern: "logs-%{[agent.version]}-*"
  overwrite: true
  enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "%{[fields.type]:other}_%{+MMyy}_test"
  username: ""
  password: ""
  
setup.kibana:
  host: "localhost:5601"

and the Result

GET service_1223_test/_search

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 4,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "service_1223_test",
        "_id": "RtCZQowBGoMhm_TXNStt",
        "_score": 1,
        "_ignored": [
          "message.keyword"
        ],
        "_source": {
          "@timestamp": "2023-12-07T04:46:25.027Z",
          "fields": {
            "type": "service"
          },
          "abc": {
            "log": {
              "Hour": "09:58:53.745",
              "Message": """[main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -
exception....yada 1
exception....yada 2
exception....yada 3
exception....yada 4
exception....yada 5""",
              "Timestamp": "2023-12-06T15:28:53.745+05:30",
              "Date": "06-12-2023"
            }
          },
          "agent": {
            "version": "8.11.1",
            "ephemeral_id": "890961c6-05b3-4543-9219-545db253a9f1",
            "id": "043ee5cb-c344-4132-98e9-b1b110c39f30",
            "name": "hyperion",
            "type": "filebeat"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "name": "hyperion"
          },
          "log": {
            "offset": 0,
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-filebeat-fargate.log"
            },
            "flags": [
              "multiline"
            ]
          },
          "message": """2023-12-06T15:28:53.745+05:30 06-12-2023 09:58:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -
exception....yada 1
exception....yada 2
exception....yada 3
exception....yada 4
exception....yada 5""",
          "input": {
            "type": "log"
          },
          "abc.log": {
            "level": "INFO"
          }
        }
      },
      {
        "_index": "service_1223_test",
        "_id": "R9CZQowBGoMhm_TXNStt",
        "_score": 1,
        "_source": {
          "@timestamp": "2023-12-07T04:46:25.027Z",
          "abc": {
            "log": {
              "Hour": "09:59:53.745",
              "Message": """[main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Completeing ProtocolHandler ["http-nio-8070"] -""",
              "Timestamp": "2023-12-06T15:29:53.745+05:30",
              "Date": "06-12-2023"
            }
          },
          "ecs": {
            "version": "8.0.0"
          },
          "input": {
            "type": "log"
          },
          "fields": {
            "type": "service"
          },
          "abc.log": {
            "level": "INFO"
          },
          "host": {
            "name": "hyperion"
          },
          "agent": {
            "version": "8.11.1",
            "ephemeral_id": "890961c6-05b3-4543-9219-545db253a9f1",
            "id": "043ee5cb-c344-4132-98e9-b1b110c39f30",
            "name": "hyperion",
            "type": "filebeat"
          },
          "log": {
            "offset": 260,
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-filebeat-fargate.log"
            }
          },
          "message": """2023-12-06T15:29:53.745+05:30 06-12-2023 09:59:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Completeing ProtocolHandler ["http-nio-8070"] -"""
        }
      },
      {
        "_index": "service_1223_test",
        "_id": "SNCZQowBGoMhm_TXNStt",
        "_score": 1,
        "_source": {
          "@timestamp": "2023-12-07T04:46:25.027Z",
          "host": {
            "name": "hyperion"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "message": """2023-12-06T15:30:53.745+05:30 06-12-2023 09:60:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -""",
          "fields": {
            "type": "service"
          },
          "abc": {
            "log": {
              "Date": "06-12-2023",
              "Hour": "09:60:53.745",
              "Message": """[main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -""",
              "Timestamp": "2023-12-06T15:30:53.745+05:30"
            }
          },
          "abc.log": {
            "level": "INFO"
          },
          "log": {
            "offset": 419,
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-filebeat-fargate.log"
            }
          },
          "input": {
            "type": "log"
          },
          "agent": {
            "id": "043ee5cb-c344-4132-98e9-b1b110c39f30",
            "name": "hyperion",
            "type": "filebeat",
            "version": "8.11.1",
            "ephemeral_id": "890961c6-05b3-4543-9219-545db253a9f1"
          }
        }
      },
      {
        "_index": "service_1223_test",
        "_id": "SdCZQowBGoMhm_TXSCt1",
        "_score": 1,
        "_source": {
          "@timestamp": "2023-12-07T04:46:25.027Z",
          "fields": {
            "type": "service"
          },
          "abc": {
            "log": {
              "Date": "06-12-2023",
              "Hour": "09:61:53.745",
              "Message": """[main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Completeing ProtocolHandler ["http-nio-8070"] -""",
              "Timestamp": "2023-12-06T15:31:53.745+05:30"
            }
          },
          "abc.log": {
            "level": "INFO"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "name": "hyperion"
          },
          "agent": {
            "name": "hyperion",
            "type": "filebeat",
            "version": "8.11.1",
            "ephemeral_id": "890961c6-05b3-4543-9219-545db253a9f1",
            "id": "043ee5cb-c344-4132-98e9-b1b110c39f30"
          },
          "log": {
            "offset": 579,
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-filebeat-fargate.log"
            }
          },
          "message": """2023-12-06T15:31:53.745+05:30 06-12-2023 09:61:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Completeing ProtocolHandler ["http-nio-8070"] -""",
          "input": {
            "type": "log"
          }
        }
      }
    ]
  }
}
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.