# When using logstash output sends only limited information. Any other output, no problem **URL:** \ **Category:** Beats\ **Tags:** auditbeat\ **Created:** [September 8, 2021, 3:33pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692 "2021-09-08T15:33:19Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![syunusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syunusic/32/4131_2.png) [@syunusic](https://discuss.elastic.co/u/syunusic)\ **Post date:** [September 8, 2021, 3:33pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692/1 "2021-09-08T15:33:19Z")
If I use auditbeat (I guess is the same with other beats) and configure elasticsearch or file output I see all information I need. In this example I modify /etc/sudoers' attributes, then I modify it, and then I modify its attributes again. This is what I see if file output is configured: ```auto {"@timestamp":"2021-09-08T15:11:28.412Z","@metadata":{"beat":"auditbeat","type":"_doc","version":"7.14.0"},"event":{"dataset":"file","category":["file"],"type":["change"],"action":["attributes_modified"],"kind":"event","module":"file_integrity"},"service":{"type":"file_integrity"},"file":{"ctime":"2021-09-08T15:11:28.408Z","mode":"0640","group":"root","path":"/etc/sudoers","inode":"3163160","size":3269,"type":"file","uid":"0","owner":"root","hash":{"sha1":"b5ec859e00290bc0ff3ca12d6c5828382f22b5dd"},"mtime":"2021-09-08T15:09:56.780Z","gid":"0"},"hash":{"sha1":"b5ec859e00290bc0ff3ca12d6c5828382f22b5dd"},"tags":["akainix","linux"],"ecs":{"version":"1.10.0"},"host":{"name":"pruebasuse.novalocal"},"agent":{"type":"auditbeat","version":"7.14.0","hostname":"pruebasuse.novalocal","ephemeral_id":"7023cb1c-274f-46d0-8d7c-96a5a8e5b6b7","id":"ecc5d2d3-e5a2-4c8f-ad6c-9e97599ab0b0","name":"pruebasuse.novalocal"}} {"@timestamp":"2021-09-08T15:11:28.412Z","@metadata":{"beat":"auditbeat","type":"_doc","version":"7.14.0"},"file":{"gid":"0","mode":"0640","group":"root","uid":"0","owner":"root","path":"/etc/sudoers","mtime":"2021-09-08T15:11:28.408Z","type":"file","inode":"3163160","hash":{"sha1":"b5ec859e00290bc0ff3ca12d6c5828382f22b5dd"},"ctime":"2021-09-08T15:11:28.408Z","size":3269},"hash":{"sha1":"b5ec859e00290bc0ff3ca12d6c5828382f22b5dd"},"tags":["akainix","linux"],"ecs":{"version":"1.10.0"},"host":{"name":"pruebasuse.novalocal"},"agent":{"ephemeral_id":"7023cb1c-274f-46d0-8d7c-96a5a8e5b6b7","id":"ecc5d2d3-e5a2-4c8f-ad6c-9e97599ab0b0","name":"pruebasuse.novalocal","type":"auditbeat","version":"7.14.0","hostname":"pruebasuse.novalocal"},"event":{"kind":"event","category":["file"],"type":["change"],"module":"file_integrity","dataset":"file","action":["attributes_modified"]},"service":{"type":"file_integrity"}} {"@timestamp":"2021-09-08T15:11:28.413Z","@metadata":{"beat":"auditbeat","type":"_doc","version":"7.14.0"},"host":{"name":"pruebasuse.novalocal"},"agent":{"hostname":"pruebasuse.novalocal","ephemeral_id":"7023cb1c-274f-46d0-8d7c-96a5a8e5b6b7","id":"ecc5d2d3-e5a2-4c8f-ad6c-9e97599ab0b0","name":"pruebasuse.novalocal","type":"auditbeat","version":"7.14.0"},"hash":{"sha1":"b5ec859e00290bc0ff3ca12d6c5828382f22b5dd"},"event":{"kind":"event","module":"file_integrity","dataset":"file","category":["file"],"type":["change"],"action":["attributes_modified"]},"service":{"type":"file_integrity"},"file":{"type":"file","uid":"0","owner":"root","path":"/etc/sudoers","mode":"0440","group":"root","hash":{"sha1":"b5ec859e00290bc0ff3ca12d6c5828382f22b5dd"},"mtime":"2021-09-08T15:11:28.408Z","ctime":"2021-09-08T15:11:28.412Z","size":3269,"gid":"0","inode":"3163160"},"tags":["akainix","linux"],"ecs":{"version":"1.10.0"}} ``` If I configure logstash as output this is what I get: ```auto 2021-09-08T15:09:56.780Z {name=pruebasuse.novalocal} %{message} 2021-09-08T15:09:56.780Z {name=pruebasuse.novalocal} %{message} 2021-09-08T15:09:56.781Z {name=pruebasuse.novalocal} %{message} ``` Configuration in auditbeat (relevant part): ```auto output.logstash: hosts: ["172.16.233.64:5045"] ``` And the input part of logstash's: ```auto input { beats { port => 5045 } } ``` I'm using logstash and auditbeat 7.14. ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [September 8, 2021, 5:22pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692/2 "2021-09-08T17:22:38Z")
> [@syunusic](#): > > `2021-09-08T15:09:56.781Z {name=pruebasuse.novalocal} %{message}` What's the output config of your Logstash that gives this? ---
**Author:** ![syunusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syunusic/32/4131_2.png) [@syunusic](https://discuss.elastic.co/u/syunusic)\ **Post date:** [September 10, 2021, 12:40pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692/3 "2021-09-10T12:40:04Z")
This is it: ```auto output { if [agent][type] == "auditbeat" or [agent][type] == "winlogbeat" or [agent][type] == "metricbeat" { --- else if "XXXXXXXX" in [tags] { file { path => "/...path.../%{[agent][type]}/%{[host][name]}.json" codec => "line" } } } } ``` Maybe you need the whole configuration of auditbeat: ```auto auditbeat.modules: - module: auditd audit_rule_files: ['${path.config}/audit.rules.d/*.conf'] audit_rules: | - module: file_integrity paths: - /bin - /usr/bin - /sbin - /usr/sbin - /etc - module: system datasets: - host - login - process - user state.period: 12h user.detect_password_changes: true login.wtmp_file_pattern: /var/log/wtmp* login.btmp_file_pattern: /var/log/btmp* tags: ["XXXXXXX"] output.logstash: hosts: ["_logstash_ip_:5045"] ``` And also I've to note this: some events are logged correctly, like logons and so on. This is what the file written by logstash looks like: ```auto 2021-09-10T12:37:43.994Z {name=pruebasuse.novalocal} Login by user root (UID: 0) on pts/1 (PID: 16694) from 172.16.233.64 (IP: 172.16.233.64) 2021-09-10T12:37:45.252Z {name=pruebasuse.novalocal} Process bash (PID: 16694) by user root STARTED 2021-09-10T12:37:45.252Z {name=pruebasuse.novalocal} Process sshd (PID: 16692) by user root STARTED 2021-09-10T12:40:35.088Z {name=pruebasuse.novalocal} %{message} 2021-09-10T12:40:35.088Z {name=pruebasuse.novalocal} %{message} ``` The ones that contain stuff like an administrator running an elevated privileges command appears with "%{message}". ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [September 10, 2021, 2:33pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692/4 "2021-09-10T14:33:17Z")
> [@syunusic](#): > > `codec => "line" }` Try using the `json_lines` codec here and see if that makes a difference. ---
**Author:** ![syunusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syunusic/32/4131_2.png) [@syunusic](https://discuss.elastic.co/u/syunusic)\ **Post date:** [September 10, 2021, 2:59pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692/5 "2021-09-10T14:59:59Z")
Nope.. same thing. I also tried with no explicit codec... no difference. ```auto 2021-09-10T14:58:55.877Z {name=pruebasuse.novalocal} %{message} 2021-09-10T14:58:55.878Z {name=pruebasuse.novalocal} %{message} 2021-09-10T14:58:55.877Z {name=pruebasuse.novalocal} %{message} ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 10, 2021, 4:29pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692/6 "2021-09-10T16:29:29Z")
Try to remove the `codec` from the `file` output, this will make logstash write the entire event in a `json` format. This way it will be possible to see if the event has the `message` field or not, as the presence of `%{message}` could indicate that the message field does not exist in that event. ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [September 10, 2021, 5:10pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692/7 "2021-09-10T17:10:12Z")
Many of Auditbeat's events do not have a `message`. Only a few of the `system` module datasets create a message describing what happened, but the auditd and FIM datasets have not `message`. ---
**Author:** ![syunusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syunusic/32/4131_2.png) [@syunusic](https://discuss.elastic.co/u/syunusic)\ **Post date:** [September 13, 2021, 1:24pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692/8 "2021-09-13T13:24:31Z")
I'd agree on this, but as you can see in the original post, I show the same event with two outputs: with file (and the same with elasticsearch as output) you can see the file attribute's modification. With logstash, doing the same (modifying /etc/sudoers) it doesn't show anything, just the "%{message}" thing. ---
**Author:** ![syunusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syunusic/32/4131_2.png) [@syunusic](https://discuss.elastic.co/u/syunusic)\ **Post date:** [September 13, 2021, 1:24pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692/9 "2021-09-13T13:24:56Z")
I already try that. Same result. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 13, 2021, 1:45pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692/10 "2021-09-13T13:45:13Z")
If you removed the `codec` option from the `file` output, then your output will be a `json` document, if it is not a `json` document then something could be wrong in your configuration. How are you running logstash? Are you running it as a service or using the command line? Are you pointing to config files or directory with multiple config files? Can you share your full pipeline configuration? ---
**Author:** ![syunusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syunusic/32/4131_2.png) [@syunusic](https://discuss.elastic.co/u/syunusic)\ **Post date:** [September 13, 2021, 2:17pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692/11 "2021-09-13T14:17:12Z")
You are totally right. I just did it again.. and worked fine. I would swear I did it without the codec but I guess I was wrong. I took off the codec part and that did the trick. Thank you very much for your help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 11, 2021, 4:17pm UTC](https://discuss.elastic.co/t/when-using-logstash-output-sends-only-limited-information-any-other-output-no-problem/283692/12 "2021-10-11T16:17:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.