# Filebeat configuration in ECE on-premises cluster does not grok the message field **URL:** \ **Category:** Elastic Cloud Enterprise (ECE)\ **Tags:** elastic-stack-monitoring\ **Created:** [May 5, 2020, 1:35pm UTC](https://discuss.elastic.co/t/filebeat-configuration-in-ece-on-premises-cluster-does-not-grok-the-message-field/231163 "2020-05-05T13:35:29Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![GeorgeGkinis](https://avatars.discourse-cdn.com/v4/letter/g/ecc23a/32.png) [@GeorgeGkinis](https://discuss.elastic.co/u/GeorgeGkinis)\ **Post date:** [May 5, 2020, 1:35pm UTC](https://discuss.elastic.co/t/filebeat-configuration-in-ece-on-premises-cluster-does-not-grok-the-message-field/231163/1 "2020-05-05T13:35:30Z")
Hello, The **ECE** installation comes with a default **logging-and-metrics** cluster which is filled with data by **filebeat** , **metricbeat** and **allocator-metricbeat**. I see in kibana that the message field is not being grokked for various components and the wrong timestamp is being used as a result. **filebeat.yml** is located under : /services/beats-runner/managed/filebeat/filebeat.yml When I open this file I see the following : ```auto # Auto-generated by service.jar on container start # Source /scala-services/beats-runner/src/main/resources/services/filebeat/template ``` I would not mind to create the necessary grok filters myself, but I need to know a way to edit the template used. Could you please explain how this can be achieved? Thanks. ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [May 5, 2020, 2:13pm UTC](https://discuss.elastic.co/t/filebeat-configuration-in-ece-on-premises-cluster-does-not-grok-the-message-field/231163/2 "2020-05-05T14:13:14Z")
Unfortunately it's not possible to edit the filebeat template I think the only thing you can do is to create a separate index with an ingest pipeline and then periodically reindex the raw data into that ☹ I'll have a dig around and try to figure out why we haven't fixed this (I'm sure I remember it being discussed ages ago) Thanks for the feedback, it's much appreciated ---
**Author:** ![GeorgeGkinis](https://avatars.discourse-cdn.com/v4/letter/g/ecc23a/32.png) [@GeorgeGkinis](https://discuss.elastic.co/u/GeorgeGkinis)\ **Post date:** [May 5, 2020, 2:14pm UTC](https://discuss.elastic.co/t/filebeat-configuration-in-ece-on-premises-cluster-does-not-grok-the-message-field/231163/3 "2020-05-05T14:14:40Z")
Hello Alex, Can't I setup the pipeline on the same index? Is the index template fixed and strict? I don't really care at this point for the older data.. ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [May 5, 2020, 2:44pm UTC](https://discuss.elastic.co/t/filebeat-configuration-in-ece-on-premises-cluster-does-not-grok-the-message-field/231163/4 "2020-05-05T14:44:25Z")
> [@GeorgeGkinis](#): > > Can't I setup the pipeline on the same index? We don't _recommend_ making changes to the system data flows because of the risk of breaking downstream dependencies, upgrades etc. (The filebeat and index templates are embedded in the docker image. The templates and mappings that are loaded into the cluster can of course be edited, but may be overwritten during cluster or ECE upgrades) ---
**Author:** ![GeorgeGkinis](https://avatars.discourse-cdn.com/v4/letter/g/ecc23a/32.png) [@GeorgeGkinis](https://discuss.elastic.co/u/GeorgeGkinis)\ **Post date:** [May 5, 2020, 4:08pm UTC](https://discuss.elastic.co/t/filebeat-configuration-in-ece-on-premises-cluster-does-not-grok-the-message-field/231163/5 "2020-05-05T16:08:14Z")
I understand. For now i have altered the index template **service-logs-6.3.2** and added **"default\_pipeline": "ece-logs"** The pipeline looks as follows : ```auto PUT _ingest/pipeline/ece-logs { "description": "parse ECE logs", "processors": [ { "set": { "field": "original_message", "value": "{{message}}" } }, { "grok": { "field": "message", "patterns": [ "\\[%{TIMESTAMP_ISO8601:timestamp}\\]\\[%{LOGLEVEL:level}\\s*\\]\\[%{DATA:logger}\\]?\\s*%{GREEDYDATA:message}", "%{TIMESTAMP_ISO8601:timestamp}?\\s*%{LOGLEVEL:level}?\\s*\\[%{DATA:logger}\\] ?\\s*%{GREEDYDATA:message}", "{\\\"level\\\":\\\"%{LOGLEVEL:level}\\\",\\\"log\\\":\\\"%{DATA:logger}\\\",\\\"@timestamp\\\":\\\"%{TIMESTAMP_ISO8601:timestamp}\\\",\\\"message\\\":\\\"*%{GREEDYDATA:message}\\\"}" ] } }, { "uppercase": { "field": "level" } }, { "date": { "field": "timestamp", "formats": [ "ISO8601", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-ddTHH:mm:ss,SSS" ], "timezone": "Europe/Amsterdam" } }, { "remove": { "field": [ "timestamp", "original_message" ] } } ], "on_failure": [ { "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } } ] } ``` Looks good so far :slight\_smile 🙂 If something goes wrong then **original\_message** is quite helpfull for debugging. If all is OK then **timestamp** and **original\_message** are removed. I suppose we could **PUT** this pipeline and alter the **index template** using **cron** or some other mechanism. It is not super clean but I prefer not parsing some documents than not parsing any at all. I will keep you posted for additions to the other _-logs-_ indices. Cheers! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 19, 2020, 4:08pm UTC](https://discuss.elastic.co/t/filebeat-configuration-in-ece-on-premises-cluster-does-not-grok-the-message-field/231163/6 "2020-05-19T16:08:20Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.