# Filebeat 6.6 system module fields not being exported(or parsed) **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 21, 2019, 6:31pm UTC](https://discuss.elastic.co/t/filebeat-6-6-system-module-fields-not-being-exported-or-parsed/169476 "2019-02-21T18:31:25Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![ChrisOdney](https://avatars.discourse-cdn.com/v4/letter/c/3ab097/32.png) [@ChrisOdney](https://discuss.elastic.co/u/ChrisOdney)\ **Post date:** [February 21, 2019, 6:31pm UTC](https://discuss.elastic.co/t/filebeat-6-6-system-module-fields-not-being-exported-or-parsed/169476/1 "2019-02-21T18:31:26Z")
Hi, I have installed filebeat and enabled the system module, I have also loaded the ingest pipelines manually as the filebeat output is being sent to logstash and to ES from there. The system dashboards were working earlier but have stopped doing so, the message field is not being parsed to JSON. Here are the before(working) and after(not working) JSON messages. When it was working: ``` { "_index": "filebeat-6.6.0-2019.02.19", "_type": "doc", "_id": "AChrBGkBWUhMufoBrD1g", "_version": 1, "_score": null, "_source": { "offset": 67858, "log": { "file": { "path": "/var/log/auth.log" } }, "prospector": { "type": "log" }, "source": "/var/log/auth.log", "fileset": { "module": "system", "name": "auth" }, "input": { "type": "log" }, "@timestamp": "2019-02-19T06:22:28.000Z", "system": { "auth": { "hostname": "ip-xxxxxxxxxxxx", "sudo": { "tty": "pts/1", "pwd": "/home/ubuntu", "user": "root", "command": "/bin/echo Hay! 4.0" }, "user": "ubuntu", "timestamp": "Feb 19 06:22:28" } }, "meta": { "cloud": { "machine_type": "t2.micro", "availability_zone": "ap-south-1a", "instance_id": "i-xxxxxxxxxxxxxx", "provider": "ec2", "region": "ap-south-1" } }, "host": { "os": { "codename": "bionic", "name": "Ubuntu", "family": "debian", "version": "18.04.1 LTS (Bionic Beaver)", "platform": "ubuntu" }, "containerized": false, "name": "ip-xxxxxxxxxxxxxx", "id": "919577b5e29b45cdb2d", "architecture": "x86_64" }, "beat": { "hostname": "ip-xxxxxxxxxxxxxxxxx", "name": "ip-xxxxxxxxxxxxxxxxx", "version": "6.6.0" }, "event": { "dataset": "system.auth" } }, "fields": { "@timestamp": [ "2019-02-19T06:22:28.000Z" ] }, "sort": [ 1550557348000 ] } ``` When it stopped working: ``` { "_index": "filebeat-6.6.0-2019.02.21", "_type": "doc", "_id": "Fd5nD2kBA9gNjQYzof-z", "_version": 1, "_score": null, "_source": { "message": "Feb 21 09:33:55 ip-172-31-29-94 sudo: root : TTY=pts/1 ; PWD=/var/log ; USER=root ; COMMAND=/bin/echo Hay version 8.0!", "input": { "type": "log" }, "log": { "file": { "path": "/var/log/auth.log" } }, "beat": { "name": "ip-xxxxxxxxxxxxx", "version": "6.6.0", "hostname": "ip-xxxxxxxxxxxxx" }, "host": { "os": { "name": "Ubuntu", "version": "18.04.1 LTS (Bionic Beaver)", "codename": "bionic", "platform": "ubuntu", "family": "debian" }, "name": "ip-xxxxxxxxxxxxx", "containerized": false, "id": "919577b5e29b45c360264e490", "architecture": "x86_64" }, "tags": [ "beats_input_codec_plain_applied" ], "@version": "1", "meta": { "cloud": { "instance_id": "i-05014b6e3321", "machine_type": "t2.micro", "region": "ap-south-1", "provider": "ec2", "availability_zone": "ap-south-1a" } }, "fileset": { "module": "system", "name": "auth" }, "event": { "dataset": "system.auth" }, "offset": 103306, "prospector": { "type": "log" }, "@timestamp": "2019-02-21T09:34:02.590Z", "source": "/var/log/auth.log" }, "fields": { "@timestamp": [ "2019-02-21T09:34:02.590Z" ] }, "highlight": { "message": [ "Feb 21 09:33:55 ip-172-31-29-94 sudo: root : TTY=pts/1 ; PWD=/var/log ; USER=root ; COMMAND=/bin/@kibana-highlighted-field@echo@/kibana-highlighted-field@ Hay version 8.0!" ] }, "sort": [ 1550741642590 ] } ``` Apart from the message string not being parsed, one of the other differences I see is that the latter JSON has a tag called "beats\_input\_codec\_plain\_applied". Thanks, Chris ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [February 25, 2019, 11:21am UTC](https://discuss.elastic.co/t/filebeat-6-6-system-module-fields-not-being-exported-or-parsed/169476/2 "2019-02-25T11:21:52Z")
Could you please share your Logstash configuration formatted using ``? Also, a few example logs which can be parsed and a few ones which cannot be parsed might also be helpful. ---
**Author:** ![CoreyDinkens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/coreydinkens/32/40995_2.png) [@CoreyDinkens](https://discuss.elastic.co/u/CoreyDinkens)\ **Post date:** [February 25, 2019, 5:37pm UTC](https://discuss.elastic.co/t/filebeat-6-6-system-module-fields-not-being-exported-or-parsed/169476/3 "2019-02-25T17:37:58Z")
Chris, I just spent a lot of time trying to figure basically the same issue on 6.6. After a lot of fruitless troubleshooting I finally realized the issues plaguing me were only related to a couple of things: 1. Kibana was not to same version as rest of the system (6.4.2 instead of 6.6 like rest) 2. Index templates were not setup/loaded correctly. Make sure your field mappings are present for the corresponding template if you `GET _cat/templates`. You can also view a list of your templates with `GET /_cat/templates?v&s=name` You need to ensure to push the templates to ES with the following Filebeat settings: **Please read up on these settings before you deploy them. You risk overwriting your existing index templates, field mappings, etc** ```auto #setup.kibana: # host: ":5601" # username: "user" # password: "pass" setup.dashboards.enabled: true setup.dashboards.index: syslog-* setup.template.enabled: true #setup.template.overwrite: true setup.template.name: "syslog" setup.template.pattern: "syslog-*" #setup.template.settings: # index.number_of_shards: 1 # index.number_of_replicas: 1 ``` Once resolved, I had to update the Dashboard Indices and I was good to go. ---
**Author:** ![ChrisOdney](https://avatars.discourse-cdn.com/v4/letter/c/3ab097/32.png) [@ChrisOdney](https://discuss.elastic.co/u/ChrisOdney)\ **Post date:** [February 27, 2019, 9:14am UTC](https://discuss.elastic.co/t/filebeat-6-6-system-module-fields-not-being-exported-or-parsed/169476/4 "2019-02-27T09:14:06Z")
@CoreyDinkens Thanks for the input, I tried setting up the dashboards as mentioned, it did not make any difference though. ---
**Author:** ![ChrisOdney](https://avatars.discourse-cdn.com/v4/letter/c/3ab097/32.png) [@ChrisOdney](https://discuss.elastic.co/u/ChrisOdney)\ **Post date:** [February 27, 2019, 9:16am UTC](https://discuss.elastic.co/t/filebeat-6-6-system-module-fields-not-being-exported-or-parsed/169476/5 "2019-02-27T09:16:19Z")
@kvch This is my logstash configuration ``` input { beats { port => 5044 } } output { elasticsearch { manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" } } ``` Unfortunately, I don't have the logs that were being parsed correctly, however, I have the json which I have uploaded in my first post. Please do let me know if there is any other info that you may need. Thanks. ---
**Author:** ![CoreyDinkens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/coreydinkens/32/40995_2.png) [@CoreyDinkens](https://discuss.elastic.co/u/CoreyDinkens)\ **Post date:** [February 27, 2019, 3:30pm UTC](https://discuss.elastic.co/t/filebeat-6-6-system-module-fields-not-being-exported-or-parsed/169476/6 "2019-02-27T15:30:09Z")
@ChrisOdney Apologies if I was not completely clear above; loading the Dashboards is really the last thing to do once everything else is working properly. I was really suggesting that you ensure your _index templates_ + _field mappings_ have been setup properly. I believe that is the key to your fields being expanded and mapped properly. ---
**Author:** ![ChrisOdney](https://avatars.discourse-cdn.com/v4/letter/c/3ab097/32.png) [@ChrisOdney](https://discuss.elastic.co/u/ChrisOdney)\ **Post date:** [March 4, 2019, 5:37am UTC](https://discuss.elastic.co/t/filebeat-6-6-system-module-fields-not-being-exported-or-parsed/169476/7 "2019-03-04T05:37:31Z")
@CoreyDinkens here is my output to the two queries you mentioned in your earlier reply. GET \_cat/templates ``` .watch-history-9 [.watcher-history-9*] 2147483647 metricbeat-6.6.1 [metricbeat-6.6.1-*] 1 .monitoring-kibana [.monitoring-kibana-6-*] 0 6050399 .watches [.watches*] 2147483647 .ml-notifications [.ml-notifications] 0 6060099 apm-6.6.0 [apm-6.6.0-*] 1 .monitoring-alerts [.monitoring-alerts-6] 0 6050399 security-index-template [.security-*] 1000 .triggered_watches [.triggered_watches*] 2147483647 .ml-state [.ml-state] 0 6060099 .monitoring-logstash [.monitoring-logstash-6-*] 0 6050399 security_audit_log [.security_audit_log*] 1000 logstash-index-template [.logstash] 0 kibana_index_template:.kibana [.kibana] 0 .ml-anomalies- [.ml-anomalies-*] 0 6060099 .ml-meta [.ml-meta] 0 6060099 logstash [logstash-*] 0 60001 .monitoring-es [.monitoring-es-6-*] 0 6050399 .ml-config [.ml-config] 0 6060099 filebeat-6.6.0 [filebeat-6.6.0-*] 1 .monitoring-beats [.monitoring-beats-6-*] 0 6050399 ``` Output for GET /\_cat/templates?v&s=name ``` name index_patterns order version .ml-anomalies- [.ml-anomalies-*] 0 6060099 .ml-config [.ml-config] 0 6060099 .ml-meta [.ml-meta] 0 6060099 .ml-notifications [.ml-notifications] 0 6060099 .ml-state [.ml-state] 0 6060099 .monitoring-alerts [.monitoring-alerts-6] 0 6050399 .monitoring-beats [.monitoring-beats-6-*] 0 6050399 .monitoring-es [.monitoring-es-6-*] 0 6050399 .monitoring-kibana [.monitoring-kibana-6-*] 0 6050399 .monitoring-logstash [.monitoring-logstash-6-*] 0 6050399 .triggered_watches [.triggered_watches*] 2147483647 .watch-history-9 [.watcher-history-9*] 2147483647 .watches [.watches*] 2147483647 apm-6.6.0 [apm-6.6.0-*] 1 filebeat-6.6.0 [filebeat-6.6.0-*] 1 kibana_index_template:.kibana [.kibana] 0 logstash [logstash-*] 0 60001 logstash-index-template [.logstash] 0 metricbeat-6.6.1 [metricbeat-6.6.1-*] 1 security-index-template [.security-*] 1000 security_audit_log [.security_audit_log*] 1000 ``` As far as my understanding goes, templates help in settings and mappings for indices.The problem I am facing here is that a message string is not being parsed into JSON which I think is more related to ES ingestion. Thanks, Chris. ---
**Author:** ![ChrisOdney](https://avatars.discourse-cdn.com/v4/letter/c/3ab097/32.png) [@ChrisOdney](https://discuss.elastic.co/u/ChrisOdney)\ **Post date:** [March 4, 2019, 5:40am UTC](https://discuss.elastic.co/t/filebeat-6-6-system-module-fields-not-being-exported-or-parsed/169476/8 "2019-03-04T05:40:52Z")
@kvch Any ideas what I am doing wrong here? I am out of options and I am stuck on this for a while now. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 1, 2019, 5:40am UTC](https://discuss.elastic.co/t/filebeat-6-6-system-module-fields-not-being-exported-or-parsed/169476/9 "2019-04-01T05:40:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.