# Filebeat 7.14.0 filestream input field log.offset is character count of the line
**URL:** https://discuss.elastic.co/t/filebeat-7-14-0-filestream-input-field-log-offset-is-character-count-of-the-line/282956
**Category:** Beats
**Tags:** filebeat
**Created:** [August 31, 2021, 7:01pm UTC](https://discuss.elastic.co/t/filebeat-7-14-0-filestream-input-field-log-offset-is-character-count-of-the-line/282956 "2021-08-31T19:01:46Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Michael\_Savettiere](https://discuss.elastic.co/u/Michael_Savettiere)
#### Post date: [August 31, 2021, 7:01pm UTC](https://discuss.elastic.co/t/filebeat-7-14-0-filestream-input-field-log-offset-is-character-count-of-the-line/282956/1 "2021-08-31T19:01:46Z")
Using Filebeat 7.14.0 input: filestream and output.console: pretty: true
I see that the value of log.offset is the size of the event.message and not the offset from the beginning of the file. I used to use input: log and here the log.offset was from the beginning of the file. Please confirm that input filestream is supposed to function this way and different than input log. I ask because the filestream input is touted as a better log file processor than log input. I need the offset from the beginning of the file as a unique identifier of the record.
My filebeat.yml file look like the following:
filebeat.inputs:
- type: filestream
paths:
- /tmp/mikes/\*.dat
file\_identity.path: ~
output.console:
pretty: true
I have a data file with simple data:
/tmp/mikes/data.dat
123456789012345
12345678901234567890
123456789012345678901234567890
The output shows each line as an event and the offset reflective of the size of each line:
{
"@timestamp": "2021-08-31T18:04:35.580Z",
"@metadata": {
"beat": "filebeat",
"type": "\_doc",
"version": "7.14.0"
},
"input": {
"type": "filestream"
},
"ecs": {
"version": "1.10.0"
},
"host": {
"name": "[xxx.dev.xxx.net](http://xxx.dev.xxx.net)"
},
"agent": {
"version": "7.14.0",
"hostname": "xxx",
"ephemeral\_id": "5761f0a1-279c-403d-971a-23ecc9c3e52c",
"id": "bfa07733-0278-4c9d-a8bc-6ff1255bf6a5",
"name": "[xxx.dev.xxx.net](http://xxx.dev.xxx.net)",
"type": "filebeat"
},
"environment": [
"remote-filebeat"
],
"log": {
"path": "/tmp/mikes/data.dat",
"offset": 16
},
"message": "123456789012345",
"tags": [
"xxx-log",
"[xxx.dev.xxx.net](http://xxx.dev.xxx.net)"
]
}
{
"@timestamp": "2021-08-31T18:04:35.580Z",
"@metadata": {
"beat": "filebeat",
"type": "\_doc",
"version": "7.14.0"
},
"log": {
"offset": 21,
"path": "/tmp/mikes/data.dat"
},
"message": "12345678901234567890",
"tags": [
"xxx-log",
"[xxx.dev.xxx.net](http://xxx.dev.xxx.net)"
],
"input": {
"type": "filestream"
},
"ecs": {
"version": "1.10.0"
},
"host": {
"name": "[xxx.dev.xxx.net](http://xxx.dev.xxx.net)"
},
"agent": {
"type": "filebeat",
"version": "7.14.0",
"hostname": "xxx",
"ephemeral\_id": "5761f0a1-279c-403d-971a-23ecc9c3e52c",
"id": "bfa07733-0278-4c9d-a8bc-6ff1255bf6a5",
"name": "[xxx.dev.xxx.net](http://xxx.dev.xxx.net)"
},
"environment": [
"remote-filebeat"
]
}
{
"@timestamp": "2021-08-31T18:04:35.581Z",
"@metadata": {
"beat": "filebeat",
"type": "\_doc",
"version": "7.14.0"
},
"host": {
"name": "[xxx.dev.xxx.net](http://xxx.dev.xxx.net)"
},
"agent": {
"version": "7.14.0",
"hostname": "xxx",
"ephemeral\_id": "5761f0a1-279c-403d-971a-23ecc9c3e52c",
"id": "bfa07733-0278-4c9d-a8bc-6ff1255bf6a5",
"name": "[xxx.dev.xxx.net](http://xxx.dev.xxx.net)",
"type": "filebeat"
},
"environment": [
"remote-filebeat"
],
"message": "123456789012345678901234567890",
"log": {
"offset": 31,
"path": "/tmp/mikes/data.dat"
},
"input": {
"type": "filestream"
},
"ecs": {
"version": "1.10.0"
}
}
The registry file log.json does show the correct offset at the end of the file.
log.json
{"op":"set","id":1}
{"k":"filestream::.global::path::/tmp/mikes/data.dat","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/tmp/mikes/data.dat","identifier\_name":"path"}}}
{"op":"set","id":2}
{"k":"filestream::.global::path::/tmp/mikes/data.dat","v":{"ttl":1800000000000,"updated":[280444765226302,1630433075],"cursor":null,"meta":{"source":"/tmp/mikes/data.dat","identifier\_name":"path"}}}
{"op":"set","id":3}
{"k":"filestream::.global::path::/tmp/mikes/data.dat","v":{"ttl":1800000000000,"updated":[280444765628790,1630433075],"cursor":{"offset":68},"meta":{"source":"/tmp/mikes/data.dat","identifier\_name":"path"}}}
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [September 1, 2021, 11:09am UTC](https://discuss.elastic.co/t/filebeat-7-14-0-filestream-input-field-log-offset-is-character-count-of-the-line/282956/2 "2021-09-01T11:09:16Z")
You are right! I will open a PR soon with the fix. Thank you for your report and trying the filestream input!
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [September 1, 2021, 12:13pm UTC](https://discuss.elastic.co/t/filebeat-7-14-0-filestream-input-field-log-offset-is-character-count-of-the-line/282956/3 "2021-09-01T12:13:35Z")
PR: [Store offset in log.offset field of events from the filestream input by kvch · Pull Request #27688 · elastic/bea](https://github.com/elastic/beats/pull/27688)
---
### Author:  [@Michael\_Savettiere](https://discuss.elastic.co/u/Michael_Savettiere)
#### Post date: [September 1, 2021, 1:55pm UTC](https://discuss.elastic.co/t/filebeat-7-14-0-filestream-input-field-log-offset-is-character-count-of-the-line/282956/4 "2021-09-01T13:55:42Z")
Thank you for reviewing this and making the correction. I also have another topic related to this regarding the event metadata schema for a difference in input plugins log and filestream. The topic brings to light the difference in schema for the file path which in input log is log.file.path while in input filestream it is log.path. Here is a link to that topic [Filebeat input types log and filestream metadata for file path/name differ](https://discuss.elastic.co/t/filebeat-input-types-log-and-filestream-metadata-for-file-path-name-differ/281713)
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [September 1, 2021, 2:22pm UTC](https://discuss.elastic.co/t/filebeat-7-14-0-filestream-input-field-log-offset-is-character-count-of-the-line/282956/5 "2021-09-01T14:22:10Z")
Thanks, I will look at that one as well.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 29, 2021, 4:23pm UTC](https://discuss.elastic.co/t/filebeat-7-14-0-filestream-input-field-log-offset-is-character-count-of-the-line/282956/6 "2021-09-29T16:23:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.