# Retrieve log line given offset
**URL:** https://discuss.elastic.co/t/retrieve-log-line-given-offset/91308
**Category:** Beats
**Created:** [June 29, 2017, 3:05pm UTC](https://discuss.elastic.co/t/retrieve-log-line-given-offset/91308 "2017-06-29T15:05:14Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@hding](https://discuss.elastic.co/u/hding)
#### Post date: [June 29, 2017, 3:05pm UTC](https://discuss.elastic.co/t/retrieve-log-line-given-offset/91308/1 "2017-06-29T15:05:14Z")
Since the size of our log files is huge, I choose not to store the original log lines, but instead store the filepath and the offset of the log line so as to retrieve the log lines later.
It says [here](https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-log.html#_offset) that "(The exported field `offset`) is the file offset the reported line starts at." However, when I try to retrieve the log line by providing `offset`, I am actually getting the next log line. It seems that `offset` is pointing at the end of the log line instead of the beginning.
Do I need to read the log line backwards from `offset`? Is there a better solution?
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [June 30, 2017, 11:05am UTC](https://discuss.elastic.co/t/retrieve-log-line-given-offset/91308/2 "2017-06-30T11:05:12Z")
Which filebeat version are you using?
Looks like a very unfortunate bug to me. You can follow the issue on github here: [https://github.com/elastic/beats/issues/4587](https://github.com/elastic/beats/issues/4587)
As workaround: The string (in JSON) should be utf-8 encoded. Using offset - byte size of string - 1 (for newline character) should give you the lines start offset. In case you do some event processing in logstash/Elastic ingest pipeline, you can adjust the offset.
---
### Author:  [@hding](https://discuss.elastic.co/u/hding)
#### Post date: [June 30, 2017, 12:42pm UTC](https://discuss.elastic.co/t/retrieve-log-line-given-offset/91308/3 "2017-06-30T12:42:22Z")
Hi Steffen,
I'm using Filebeat 5.4.1.
In the [source code](https://github.com/elastic/beats/blob/master/filebeat/prospector/log/harvester.go), these are lines 239-277:
```auto
// Get copy of state to work on
// This is important in case sending is not successful so on shutdown
// the old offset is reported
state := h.getState()
state.Offset += int64(message.Bytes)
// Create state event
data := util.NewData()
if h.source.HasState() {
data.SetState(state)
}
text := string(message.Content)
// Check if data should be added to event. Only export non empty events.
if !message.IsEmpty() && h.shouldExportLine(text) {
data.Event = common.MapStr{
"@timestamp": common.Time(message.Ts),
"source": state.Source,
"offset": state.Offset, // Offset here is the offset before the starting char.
}
data.Event.DeepUpdate(message.Fields)
// Check if json fields exist
var jsonFields common.MapStr
if fields, ok := data.Event["json"]; ok {
jsonFields = fields.(common.MapStr)
}
if h.config.JSON != nil && len(jsonFields) > 0 {
reader.MergeJSONFields(data.Event, jsonFields, &text, *h.config.JSON)
} else if &text != nil {
if data.Event == nil {
data.Event = common.MapStr{}
}
data.Event["message"] = text
}
}
```
Line 243 first adds the current message length to state.offset:
```auto
state.Offset += int64(message.Bytes)
```
Then line 259 sets the exported field "offset" to state.offset:
```auto
"offset": state.Offset, // Offset here is the offset before the starting char.
```
If we change the order of these two sections of code, would it solve the issue?
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [June 30, 2017, 8:49pm UTC](https://discuss.elastic.co/t/retrieve-log-line-given-offset/91308/4 "2017-06-30T20:49:06Z")
Thanks. We're still wondering when this bug was introduced 😕
Feel free to open a PR: [https://github.com/elastic/beats/pulls](https://github.com/elastic/beats/pulls)
Also see [contribution guide](https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html) if you want to provide a PR with fix and system test.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 20, 2017, 3:05pm UTC](https://discuss.elastic.co/t/retrieve-log-line-given-offset/91308/5 "2017-07-20T15:05:23Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.