Log ingestion to Elastic Cloud not working with ECS Fargate

[Gabriel_Carvalho]

Hi everyone!

I am trying to send logs of my apps running on an ECS Fargate Cluster to Elastic Cloud. I am using aws firelens logging driver and fluentbit as log router, I followed Elastic Cloud's documentation and everything seemed to be pretty straightforward, but it just doesn’t work...

In the first attempt, I was getting these errors:

2022-03-25 18:52:17[2022/03/25 21:52:17] [ warn] [engine] chunk '1-1648245127.815032857.flb' cannot be retried: task_id=0, input=forward.1 > output=es.1
2022-03-25 18:52:11[2022/03/25 21:52:11] [error] [output:es:es.1] HTTP status=400 URI=/_bulk, response:
2022-03-25 18:52:11{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}
2022-03-25 18:52:11[2022/03/25 21:52:11] [ warn] [engine] failed to flush chunk '1-1648245127.815032857.flb', retry in 6 seconds: task_id=0, input=forward.1 > output=es.1 (out_id=1)
2022-03-25 18:51:36[2022/03/25 21:51:36] [ info] [sp] stream processor started
2022-03-25 18:51:36[2022/03/25 21:51:36] [ info] [input:forward:forward.2] listening on 127.0.0.1:24224
2022-03-25 18:51:36[2022/03/25 21:51:36] [ info] [engine] started (pid=1)
2022-03-25 18:51:36[2022/03/25 21:51:36] [ info] [storage] version=1.1.5, initializing...
2022-03-25 18:51:36[2022/03/25 21:51:36] [ info] [storage] in-memory
2022-03-25 18:51:36[2022/03/25 21:51:36] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
2022-03-25 18:51:36[2022/03/25 21:51:36] [ info] [cmetrics] version=0.2.2
2022-03-25 18:51:36[2022/03/25 21:51:36] [ info] [input:tcp:tcp.0] listening on 127.0.0.1:8877
2022-03-25 18:51:36[2022/03/25 21:51:36] [ info] [input:forward:forward.1] listening on unix:///var/run/fluent.sock
2022-03-25 18:51:36e[1mFluent Bit v1.8.9e[0m
2022-03-25 18:51:36* e[1me[93mCopyright (C) 2019-2021 The Fluent Bit Authorse[0m
2022-03-25 18:51:36* e[1me[93mCopyright (C) 2015-2018 Treasure Datae[0m
2022-03-25 18:51:36* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
2022-03-25 18:51:36* https://fluentbit.io

After doing some search, I decided to enable Suppress_Type_Name option, like this: "Suppress_Type_Name On". After that, the errors above do not appear anymore, but fluentbit just doesn’t route any logs, like if it wasn’t able to read logs from the app:

2022-03-26 11:58:56[2022/03/26 14:58:56] [ info] [sp] stream processor started
2022-03-26 11:58:56[2022/03/26 14:58:56] [ info] [engine] started (pid=1)
2022-03-26 11:58:56[2022/03/26 14:58:56] [ info] [storage] version=1.1.5, initializing...
2022-03-26 11:58:56[2022/03/26 14:58:56] [ info] [storage] in-memory
2022-03-26 11:58:56[2022/03/26 14:58:56] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
2022-03-26 11:58:56[2022/03/26 14:58:56] [ info] [cmetrics] version=0.2.2
2022-03-26 11:58:56[2022/03/26 14:58:56] [ info] [input:tcp:tcp.0] listening on 127.0.0.1:8877
2022-03-26 11:58:56[2022/03/26 14:58:56] [ info] [input:forward:forward.1] listening on unix:///var/run/fluent.sock
2022-03-26 11:58:56[2022/03/26 14:58:56] [ info] [input:forward:forward.2] listening on 127.0.0.1:24224
2022-03-26 11:58:56e[1mFluent Bit v1.8.9e[0m
2022-03-26 11:58:56* e[1me[93mCopyright (C) 2019-2021 The Fluent Bit Authorse[0m
2022-03-26 11:58:56* e[1me[93mCopyright (C) 2015-2018 Treasure Datae[0m
2022-03-26 11:58:56* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
2022-03-26 11:58:56* https://fluentbit.io

It keeps like this forever... This is my output config:

[OUTPUT]
    Name es
    Match app-name-firelens*
    Cloud_Auth ****
    Cloud_ID ****
    Include_Tag_Key true
    Index elastic_firelens
    Port 9243
    Suppress_Type_Name On
    Tag_Key tags
    tls On
    tls.verify Off

[matschaffer]

hi @Gabriel_Carvalho!

Have you tried AWS Fargate module | Filebeat Reference [8.1] | Elastic ?

That might just work without having to traverse Fluent Bit.

Or if Fluent Bit is a requirement for you, I'd recommend also reaching out on their forums: Discussions · fluent/fluentd · GitHub

https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch seems to say there should be some more logging information available if Elasticsearch isn't accepting the logs and the fluent bit community would be the best resource to find out how to get those logs.

[Gabriel_Carvalho]

Hi everyone, I found out that I was using the wrong data view. After creating a new data view filtering with _index: elastic_firelens, the logs came up.