# Sanitizing Logs Before Shipping **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 26, 2017, 11:32am UTC](https://discuss.elastic.co/t/sanitizing-logs-before-shipping/109134 "2017-11-26T11:32:55Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Ossie](https://avatars.discourse-cdn.com/v4/letter/o/8491ac/32.png) [@Ossie](https://discuss.elastic.co/u/Ossie)\ **Post date:** [November 26, 2017, 11:32am UTC](https://discuss.elastic.co/t/sanitizing-logs-before-shipping/109134/1 "2017-11-26T11:32:55Z")
Hi, I currently have a situation where an application sometimes logs sensitive data to a file. I want to ship these logs using Filebeat but would prefer to sanitize the sensitive fields before they reach Elasticsearch. I can't modify the application. Is there any way to do this with Filebeat? If it isn't currently possible, will this feature be added any time soon? Thanks a lot. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 27, 2017, 12:14am UTC](https://discuss.elastic.co/t/sanitizing-logs-before-shipping/109134/2 "2017-11-27T00:14:30Z")
This is currently not possible in Filebeat. For these use cases we recommend to add Logstash and add do the sanitizing of events in LS before forwarding it to ES. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 28, 2017, 1:53pm UTC](https://discuss.elastic.co/t/sanitizing-logs-before-shipping/109134/3 "2017-11-28T13:53:31Z")
Out of curiosity: What exactly do you mean by sanitize fields? What exactly is a field? you have JSON or you thinking of some regular expression to much some sub-string in your log? By sanitizing you want to: a) remove the field b) replace with custom string (e.g. \*\*\*\*\*\*\* of same or constant length?) c) compute and replace with hash (or randomly generated value) such that events with same sensitive fields can still be correlated? ---
**Author:** ![Ossie](https://avatars.discourse-cdn.com/v4/letter/o/8491ac/32.png) [@Ossie](https://discuss.elastic.co/u/Ossie)\ **Post date:** [November 28, 2017, 7:43pm UTC](https://discuss.elastic.co/t/sanitizing-logs-before-shipping/109134/4 "2017-11-28T19:43:13Z")
The output is currently JSON and I wanted to replace it with a custom string (\*\*\*\*\*). But the other option of replacing it with a hash look interesting as well. Thanks a lot for your feedback. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 29, 2017, 2:01pm UTC](https://discuss.elastic.co/t/sanitizing-logs-before-shipping/109134/5 "2017-11-29T14:01:45Z")
If you already have JSON, you can use the `drop_fields` processor to remove fields. Otherwise you will have to use an Ingest Node pipeline or Logstash to redact contents (ensure you have TLS configured, such that events are encrypted when being send). ---
**Author:** ![Ossie](https://avatars.discourse-cdn.com/v4/letter/o/8491ac/32.png) [@Ossie](https://discuss.elastic.co/u/Ossie)\ **Post date:** [December 6, 2017, 7:08am UTC](https://discuss.elastic.co/t/sanitizing-logs-before-shipping/109134/6 "2017-12-06T07:08:46Z")
Thanks steffens. Is there any way to also delete a file after processing it using filebeat (in my case, a single transaction is stored per file)? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [December 6, 2017, 2:08pm UTC](https://discuss.elastic.co/t/sanitizing-logs-before-shipping/109134/7 "2017-12-06T14:08:23Z")
No. Filebeats main purpose is to tail log files, that is it relies on external tools doing log-rotation and deleting files. ---
**Author:** ![Ossie](https://avatars.discourse-cdn.com/v4/letter/o/8491ac/32.png) [@Ossie](https://discuss.elastic.co/u/Ossie)\ **Post date:** [December 6, 2017, 4:55pm UTC](https://discuss.elastic.co/t/sanitizing-logs-before-shipping/109134/8 "2017-12-06T16:55:06Z")
Great. Thanks for your help steffens. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 3, 2018, 4:55pm UTC](https://discuss.elastic.co/t/sanitizing-logs-before-shipping/109134/9 "2018-01-03T16:55:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.