# Parse log file with two formats in it
**URL:** https://discuss.elastic.co/t/parse-log-file-with-two-formats-in-it/166876
**Category:** Logstash
**Created:** [February 3, 2019, 5:54pm UTC](https://discuss.elastic.co/t/parse-log-file-with-two-formats-in-it/166876 "2019-02-03T17:54:29Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)
#### Post date: [February 3, 2019, 5:54pm UTC](https://discuss.elastic.co/t/parse-log-file-with-two-formats-in-it/166876/1 "2019-02-03T17:54:29Z")
I have a log file with two formats in it. The metadata is more like a key value format where the delimiter is '=', the actual data is in a csv format. Both these formats have been bundled into a single file. Can I generate each event as individual document with metadata as part of every document?
The log file is [like this](https://pastebin.com/Pt2budXv).
Appreciate any help on this.
Thanks
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 3, 2019, 6:29pm UTC](https://discuss.elastic.co/t/parse-log-file-with-two-formats-in-it/166876/2 "2019-02-03T18:29:13Z")
This should give you an idea of how to do it. It drops comments and lines that are just whitespace. Then it parses key=value and stashes it in a class variable. Then it parses anything with multiple commas as a csv. That leaves you with odds and ends to handle, such as
```auto
The RTC is running 0 hours, 0 mins and 2 secs behind real time.
The flash last updated 1/3/2018 at 2:04
The last .tab file placed 1/3/2018 at 14:01
```
If you need to get data out of those lines use grok. Make sure you anchor your patterns using ^.
```
if [message] =~ /^#/ {
drop {}
} else if [message] =~ /^\s*$/ {
drop {}
} else if [message] =~ /^[A-Za-z0-9]+=/ {
ruby {
init => '
@@metadata = {}
'
code => '
msg = event.get("message")
matches = msg.scan(/^([A-Za-z0-9]+)=(.*)/)
m = matches[0]
@@metadata[m[0]] = m[1]
'
}
drop {}
} else if [message] =~ /,.*,.*,/ {
csv {
autodetect_column_names => true
}
ruby {
code => '
event.set("metadata", @@metadata)
'
}
}
```
---
### Author:  [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)
#### Post date: [February 3, 2019, 10:05pm UTC](https://discuss.elastic.co/t/parse-log-file-with-two-formats-in-it/166876/3 "2019-02-03T22:05:42Z")
@Badger, Any ideas on what my input plugin should be?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 3, 2019, 10:32pm UTC](https://discuss.elastic.co/t/parse-log-file-with-two-formats-in-it/166876/4 "2019-02-03T22:32:10Z")
You probably want either a file input or filebeat connecting to a beats input.
---
### Author:  [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)
#### Post date: [February 5, 2019, 6:53pm UTC](https://discuss.elastic.co/t/parse-log-file-with-two-formats-in-it/166876/5 "2019-02-05T18:53:54Z")
@Badger, My metadata looks [like this](https://pastebin.com/sdRjgd6q). Any idea how should I remove the trailing "\r" in the values of each key.
Thanks.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 5, 2019, 7:05pm UTC](https://discuss.elastic.co/t/parse-log-file-with-two-formats-in-it/166876/6 "2019-02-05T19:05:11Z")
\r counts as whitespace, so
```
mutate { strip => ["message"] }
```
will remove it and the space before it.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 5, 2019, 7:05pm UTC](https://discuss.elastic.co/t/parse-log-file-with-two-formats-in-it/166876/7 "2019-03-05T19:05:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.