# Denormalize data within a log file

**URL:** <https://discuss.elastic.co/t/denormalize-data-within-a-log-file/141277>\
**Category:** Logstash\
**Created:** [July 24, 2018, 2:40am UTC](https://discuss.elastic.co/t/denormalize-data-within-a-log-file/141277 "2018-07-24T02:40:46Z")\
**Posts on this page:** 1\
**Showing post:** 17

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [July 24, 2018, 5:45pm UTC](https://discuss.elastic.co/t/denormalize-data-within-a-log-file/141277/17 "2018-07-24T17:45:48Z")

</div>

Sure. A few caveats. I used a preview release of Logstash v 6.3.2 (its being released now, as I type)  
In the logstash file `config/logstash.yml`

```auto
config.support_escapes: true
pipeline.batch.size: 1
pipeline.workers: 1

```

one worker and one event per batch - because the file with 20 000 QSO lines (due to the split) will explode the batch to 20 000. There will be a lot of duplicated data. You might be advised to do some surgery on the very big files - copy and remove the top half of the QSO lines from one and the bottom half from the other.

You will need to install the latest version or the file input  
`bin/logstash-plugin install logstash-input-file --version 4.1.5`

Also we should only read one file at a time too.  
File input:

```auto
  file {
    path => "/path/to/radio/sample.txt" # replace
    sincedb_path => "/dev/null" # replace with real path to a sincedb file when ready
    delimiter => "§¶¶§" # improbable delimiter, all data is accumulated until EOF
    mode => "read"
    max_open_files => 1
    file_completed_action => "log"
    file_completed_log_path => "/path/to/radio/completed.txt" # replace
  }

```

Good luck.

---

_[View the full topic](https://discuss.elastic.co/t/denormalize-data-within-a-log-file/141277)._