# Thread is missing when i send logs from ECS fargate to Elastic search

**URL:** <https://discuss.elastic.co/t/thread-is-missing-when-i-send-logs-from-ecs-fargate-to-elastic-search/348774>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-monitoring, docker\
**Created:** [December 7, 2023, 4:15am UTC](https://discuss.elastic.co/t/thread-is-missing-when-i-send-logs-from-ecs-fargate-to-elastic-search/348774 "2023-12-07T04:15:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![NitinKalburgii](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nitinkalburgii/32/128877_2.png) [@NitinKalburgii](https://discuss.elastic.co/u/NitinKalburgii)\
**Post date:** [December 7, 2023, 4:15am UTC](https://discuss.elastic.co/t/thread-is-missing-when-i-send-logs-from-ecs-fargate-to-elastic-search/348774/1 "2023-12-07T04:15:17Z")

</div>

Hi! I was running my application in ECS fargate(AWS Service) and in AWS monitoring i was getting logs like `	2023-12-06T15:28:53.745+05:30	06-12-2023 09:58:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -`

But when i injest this log in Elasticsearch i was not getting the [main] part. I was running this on [filebeat/beats container version 7:12:1](http://docker.elastic.co/beats/filebeat-oss:7.12.1)  
and below i my filebeat.yml file

```auto
filebeat.inputs:
  - type: log
    paths:
     - /usr/share/filebeat/logs/
    fields:
      type: "service"
    multiline.type: pattern
    multiline.pattern: '^[0-9]{2}-[0-9]{2}-[0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'
    multiline.negate: true
    multiline.match: after
    processors:
      - dissect:
          tokenizer: '%{+Timestamp|string} %{+Timestamp|string} %{Message|string}'
          field: "message"
          target_prefix: "abc.log"
      - if:
          regexp:
            message: ERROR
        then:
          - add_fields:
              target: 'abc.log'
              fields:
                level: ERROR
        else:
        - if:
            regexp:
              message: WARN
          then:
            - add_fields:
                target: 'abc.log'
                fields:
                  level: WARN
          else:
            - add_fields:
                target: 'abc.log'
                fields:
                  level: INFO 

                  
setup.ilm.enabled: false

filebeat.config.modules:  
  reload.enabled: true
  reload.period: 10s
  
setup.template:
  name: "logs-%{[agent.version]}"
  pattern: "logs-%{[agent.version]}-*"
  overwrite: true
  enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.elasticsearch:
  hosts: ["https://url:443"]
  index: "%{[fields.type]:other}_%{+MMyy}_test"
  username: ""
  password: ""
  
setup.kibana:
  host: "localhost:5601"

logging.level: debug
logging.to_files: true
logging.files:
 path: /usr/share/filebeat/logs/
 name: filebeat
 keepfiles: 7
 permissions: 0644
xpack.security.enabled: false
xpack.monitoring.enabled: false
xpack.graph.enabled: false
xpack.watcher.enabled: false
xpack.reporting.enabled: false

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 7, 2023, 4:47am UTC](https://discuss.elastic.co/t/thread-is-missing-when-i-send-logs-from-ecs-fargate-to-elastic-search/348774/2 "2023-12-07T04:47:57Z")

</div>

Hi @NitinKalburgii Welcome to the commuity!

So a couple issues... your `multiline` pattern is incorrect it does not match the format of the time string  
and your `dissect` is also incorrect it does not match the pattern of the message

But not sure why you need multi-line unless you have exceptions perhaps that is your point.

But most important 7.12 is ANCIENT ... you should upgrade with haste! 🙂

Here is the log file I used

```auto
2023-12-06T15:28:53.745+05:30 06-12-2023 09:58:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -
exception....yada 1
exception....yada 2
exception....yada 3
exception....yada 4
exception....yada 5
2023-12-06T15:29:53.745+05:30 06-12-2023 09:59:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Completeing ProtocolHandler ["http-nio-8070"] -
2023-12-06T15:30:53.745+05:30 06-12-2023 09:60:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -
2023-12-06T15:31:53.745+05:30 06-12-2023 09:61:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Completeing ProtocolHandler ["http-nio-8070"] -

```

Here is the filebeat.yml

```auto
filebeat.inputs:
  - type: log
    paths:
     - /Users/sbrown/workspace/sample-data/discuss/discuss-filebeat-fargate.log
    fields:
      type: "service"
    multiline.type: pattern
    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}'
    multiline.negate: true
    multiline.match: after
    processors:
      - dissect:
          tokenizer: '%{+Timestamp|string} %{+Date|string} %{+Hour|string} %{Message|string}'
          field: "message"
          target_prefix: "abc.log"
      - if:
          regexp:
            message: ERROR
        then:
          - add_fields:
              target: 'abc.log'
              fields:
                level: ERROR
        else:
        - if:
            regexp:
              message: WARN
          then:
            - add_fields:
                target: 'abc.log'
                fields:
                  level: WARN
          else:
            - add_fields:
                target: 'abc.log'
                fields:
                  level: INFO 

                  
setup.ilm.enabled: false

filebeat.config.modules:  
  reload.enabled: true
  reload.period: 10s
  
setup.template:
  name: "logs-%{[agent.version]}"
  pattern: "logs-%{[agent.version]}-*"
  overwrite: true
  enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "%{[fields.type]:other}_%{+MMyy}_test"
  username: ""
  password: ""
  
setup.kibana:
  host: "localhost:5601"

```

and the Result

```auto
GET service_1223_test/_search

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 4,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "service_1223_test",
        "_id": "RtCZQowBGoMhm_TXNStt",
        "_score": 1,
        "_ignored": [
          "message.keyword"
        ],
        "_source": {
          "@timestamp": "2023-12-07T04:46:25.027Z",
          "fields": {
            "type": "service"
          },
          "abc": {
            "log": {
              "Hour": "09:58:53.745",
              "Message": """[main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -
exception....yada 1
exception....yada 2
exception....yada 3
exception....yada 4
exception....yada 5""",
              "Timestamp": "2023-12-06T15:28:53.745+05:30",
              "Date": "06-12-2023"
            }
          },
          "agent": {
            "version": "8.11.1",
            "ephemeral_id": "890961c6-05b3-4543-9219-545db253a9f1",
            "id": "043ee5cb-c344-4132-98e9-b1b110c39f30",
            "name": "hyperion",
            "type": "filebeat"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "name": "hyperion"
          },
          "log": {
            "offset": 0,
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-filebeat-fargate.log"
            },
            "flags": [
              "multiline"
            ]
          },
          "message": """2023-12-06T15:28:53.745+05:30 06-12-2023 09:58:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -
exception....yada 1
exception....yada 2
exception....yada 3
exception....yada 4
exception....yada 5""",
          "input": {
            "type": "log"
          },
          "abc.log": {
            "level": "INFO"
          }
        }
      },
      {
        "_index": "service_1223_test",
        "_id": "R9CZQowBGoMhm_TXNStt",
        "_score": 1,
        "_source": {
          "@timestamp": "2023-12-07T04:46:25.027Z",
          "abc": {
            "log": {
              "Hour": "09:59:53.745",
              "Message": """[main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Completeing ProtocolHandler ["http-nio-8070"] -""",
              "Timestamp": "2023-12-06T15:29:53.745+05:30",
              "Date": "06-12-2023"
            }
          },
          "ecs": {
            "version": "8.0.0"
          },
          "input": {
            "type": "log"
          },
          "fields": {
            "type": "service"
          },
          "abc.log": {
            "level": "INFO"
          },
          "host": {
            "name": "hyperion"
          },
          "agent": {
            "version": "8.11.1",
            "ephemeral_id": "890961c6-05b3-4543-9219-545db253a9f1",
            "id": "043ee5cb-c344-4132-98e9-b1b110c39f30",
            "name": "hyperion",
            "type": "filebeat"
          },
          "log": {
            "offset": 260,
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-filebeat-fargate.log"
            }
          },
          "message": """2023-12-06T15:29:53.745+05:30 06-12-2023 09:59:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Completeing ProtocolHandler ["http-nio-8070"] -"""
        }
      },
      {
        "_index": "service_1223_test",
        "_id": "SNCZQowBGoMhm_TXNStt",
        "_score": 1,
        "_source": {
          "@timestamp": "2023-12-07T04:46:25.027Z",
          "host": {
            "name": "hyperion"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "message": """2023-12-06T15:30:53.745+05:30 06-12-2023 09:60:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -""",
          "fields": {
            "type": "service"
          },
          "abc": {
            "log": {
              "Date": "06-12-2023",
              "Hour": "09:60:53.745",
              "Message": """[main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-8070"] -""",
              "Timestamp": "2023-12-06T15:30:53.745+05:30"
            }
          },
          "abc.log": {
            "level": "INFO"
          },
          "log": {
            "offset": 419,
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-filebeat-fargate.log"
            }
          },
          "input": {
            "type": "log"
          },
          "agent": {
            "id": "043ee5cb-c344-4132-98e9-b1b110c39f30",
            "name": "hyperion",
            "type": "filebeat",
            "version": "8.11.1",
            "ephemeral_id": "890961c6-05b3-4543-9219-545db253a9f1"
          }
        }
      },
      {
        "_index": "service_1223_test",
        "_id": "SdCZQowBGoMhm_TXSCt1",
        "_score": 1,
        "_source": {
          "@timestamp": "2023-12-07T04:46:25.027Z",
          "fields": {
            "type": "service"
          },
          "abc": {
            "log": {
              "Date": "06-12-2023",
              "Hour": "09:61:53.745",
              "Message": """[main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Completeing ProtocolHandler ["http-nio-8070"] -""",
              "Timestamp": "2023-12-06T15:31:53.745+05:30"
            }
          },
          "abc.log": {
            "level": "INFO"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "name": "hyperion"
          },
          "agent": {
            "name": "hyperion",
            "type": "filebeat",
            "version": "8.11.1",
            "ephemeral_id": "890961c6-05b3-4543-9219-545db253a9f1",
            "id": "043ee5cb-c344-4132-98e9-b1b110c39f30"
          },
          "log": {
            "offset": 579,
            "file": {
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-filebeat-fargate.log"
            }
          },
          "message": """2023-12-06T15:31:53.745+05:30 06-12-2023 09:61:53.745 [main] INFO [] o.a.coyote.http11.Http11NioProtocol.log - Completeing ProtocolHandler ["http-nio-8070"] -""",
          "input": {
            "type": "log"
          }
        }
      }
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 4, 2024, 4:48am UTC](https://discuss.elastic.co/t/thread-is-missing-when-i-send-logs-from-ecs-fargate-to-elastic-search/348774/3 "2024-01-04T04:48:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.