# Encoded data in message field using filebeat filestream input
**URL:** https://discuss.elastic.co/t/encoded-data-in-message-field-using-filebeat-filestream-input/330907
**Category:** Beats
**Tags:** filebeat
**Created:** [April 27, 2023, 7:01am UTC](https://discuss.elastic.co/t/encoded-data-in-message-field-using-filebeat-filestream-input/330907 "2023-04-27T07:01:24Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Ryan\_Clark](https://discuss.elastic.co/u/Ryan_Clark)
#### Post date: [April 27, 2023, 7:01am UTC](https://discuss.elastic.co/t/encoded-data-in-message-field-using-filebeat-filestream-input/330907/1 "2023-04-27T07:01:25Z")
I'm using filebeat to read in a multiline log. I'm able to get the data into elasticsearch with the multiline event stored into the message field.
Log Sample:
```auto
Date: Wed Apr 19 09:57:45 2023
Computer Name: SystemX
User Name: SystemX.User
Project includes 1 folder(s) and 4 file(s).
============================================================================================
encrypt mode:
AS_ENCRYPT_MODE_AES256_SHA2
set a password for this encryption:
using a user supplied password
set up a group and master password:
unencrypted
no encrypted with groupinfo
============================================================================================
C:\Users\User\Desktop\Test files\File1.txt 8b6ccb43dca2040c3cfbcd7bfff0b387d4538c33 15bytes 2023/4/6 19:49:45
C:\Users\User\Desktop\Test files\File2.docx a3dcef559e04628b1c71a1d87d353e070bd5d40a 11853bytes 2023/4/6 19:49:45
C:\Users\User\Desktop\Test files\File3.pptx 2ca33d9f81a91d2648971f5a12d03ec0ef9fc408 31579bytes 2023/4/6 19:49:45
C:\Users\User\Desktop\Test files\File4.xlsx f4e15a60f7313fae60b9f05b0dc016ab6c68f031 8426bytes 2023/4/6 19:49:45
END OF FILE
```
Filebeat.yml excerpt:
```auto
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
# filestream is an input for collecting log messages from files.
- type: filestream
close_timeout: 5m
# Unique ID among all inputs, an ID is required.
id: "WinZip Safe Media"
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- "C:\\ProgramData\\WinZip Log Files\\*"
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
prospector.scanner.exclude_files: ['.zip$']
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
parsers:
- multiline:
type: pattern
pattern: '^Date\:.*'
negate: true
match: after
```
Visualize in Discover:
This issue is that I tried to create an ingest pipeline to parse out the data into custom fields. My grok processor does not match because the data is coming in encoded. I can see this when viewing the data in JSON.
Here is my grok processor match statement:
```auto
{
"grok": {
"field": "message",
"patterns": [
"(?m).*Date: %{DATA:event_timestamp}\\n\\n.*User Name: .*\\.%{DATA:user_name}\\n\\n.*Project.*encrypt mode:\\n\\n%{DATA:encrypt_algo}\\n\\n.*\\=\\n\\n%{GREEDYDATA:file_list}.*END OF FILE"
],
"ignore_failure": true
}
},
{
"set": {
"field": "user.name",
"value": "{{user_name}}",
"ignore_failure": true
}
}
]
```
I've never had issues with the log input type in the past, not sure if there is something I'm missing with this filestream input.
I tested the grok statement in Dev Tools Grok Debugger and it works fine.
---
### Author:  [@marc.guasch](https://discuss.elastic.co/u/marc.guasch)
#### Post date: [April 27, 2023, 2:54pm UTC](https://discuss.elastic.co/t/encoded-data-in-message-field-using-filebeat-filestream-input/330907/2 "2023-04-27T14:54:33Z")
Hello! It seems like the encoding might not be correctly detected, maybe you could try to explicitly set the `encoding` ([filestream input | Filebeat Reference [8.7] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#_encoding_2)) and see if it makes a difference.
---
### Author:  [@Ryan\_Clark](https://discuss.elastic.co/u/Ryan_Clark)
#### Post date: [April 27, 2023, 5:11pm UTC](https://discuss.elastic.co/t/encoded-data-in-message-field-using-filebeat-filestream-input/330907/3 "2023-04-27T17:11:20Z")
I did see that list of encodings but didn't see an example or direction on where that gets set. Is there a setting for the filestream in filebeat.yml ? Normally the documentation shows an example but in this case it does not.
I did try adding under filestream in the filebeat.yml:
```auto
encoding: plain
```
I also tried:
```auto
encoding: utf-8
```
That didn't seem to change anything.
Is the encoding setting nested right under the filestream definition or does it go somewhere else in the filebeat.yml?
---
### Author:  [@Ryan\_Clark](https://discuss.elastic.co/u/Ryan_Clark)
#### Post date: [May 3, 2023, 8:16pm UTC](https://discuss.elastic.co/t/encoded-data-in-message-field-using-filebeat-filestream-input/330907/4 "2023-05-03T20:16:35Z")
It was encoding. The file had a weird encoding (utf-16le-bom). I was able to see what the file encoding was in Notepad++ and through trial and error on placement of the setting, I was able to get it to work. In my case I put "encoding: utf-16le-bom" right under "- type: filestream" in the filebeat.yml.
It would be helpful for others to have an example of this like all the other configuration options for encoding on the filestream page.
> **[filestream input | Filebeat Reference \[8.7\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#_encoding_2)**
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 31, 2023, 10:17pm UTC](https://discuss.elastic.co/t/encoded-data-in-message-field-using-filebeat-filestream-input/330907/5 "2023-05-31T22:17:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.