# Differentiate between sections within a single log file
**URL:** https://discuss.elastic.co/t/differentiate-between-sections-within-a-single-log-file/168478
**Category:** Logstash
**Created:** [February 14, 2019, 7:30pm UTC](https://discuss.elastic.co/t/differentiate-between-sections-within-a-single-log-file/168478 "2019-02-14T19:30:10Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)
#### Post date: [February 14, 2019, 7:30pm UTC](https://discuss.elastic.co/t/differentiate-between-sections-within-a-single-log-file/168478/1 "2019-02-14T19:30:10Z")
Hey,
I have a log file which looks [like this](https://pastebin.com/j4ez3m4r). Its basically a tab separated text file with a few lines of metadata text. I would want to extract "TIME" only as part of metadata field. For other tab separated lines, I would want to ignore data under "DOTS HEARD FROM BUT NOT IN CONFIGURATION" and "REPEATERS". I am using the csv filter with separator as "tab space", but I am not able to distinguish between the sections.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 14, 2019, 8:21pm UTC](https://discuss.elastic.co/t/differentiate-between-sections-within-a-single-log-file/168478/2 "2019-02-14T20:21:32Z")
I would do something very similar to the solution I [proposed](https://discuss.elastic.co/t/parse-log-file-with-two-formats-in-it/166876/2?u=badger) for the other format you had...
```
if [message] =~ /^(\s*$|:::::)/ {
drop {}
} else if [message] =~ /^TIME/ {
# parse it and stash it in a ruby class variable
} else if [message] =~ " .* .* " {
csv {separator => " " autodetect_column_names => true }
# and append the metadata
} else {
drop {}
}
```
---
### Author:  [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)
#### Post date: [February 15, 2019, 6:47am UTC](https://discuss.elastic.co/t/differentiate-between-sections-within-a-single-log-file/168478/3 "2019-02-15T06:47:02Z")
Hey @Badger, I did try a similar solution before posting this question. The issue I am facing is, there will be 3 column header lines matching the condition `" .* .* "`. I want to drop the other two because they have no data rows under them. I want to parse only the ones which have data rows under the column headers. Hope you got my concern.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 15, 2019, 2:38pm UTC](https://discuss.elastic.co/t/differentiate-between-sections-within-a-single-log-file/168478/4 "2019-02-15T14:38:23Z")
I can only suggest an ad-hoc solution like dropping lines when one of the csv fields is text rather than numeric.
---
### Author:  [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)
#### Post date: [February 15, 2019, 6:45pm UTC](https://discuss.elastic.co/t/differentiate-between-sections-within-a-single-log-file/168478/5 "2019-02-15T18:45:37Z")
Any idea how to do the same in logstash. I am not able to drop the lines within the csv filter.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 15, 2019, 7:40pm UTC](https://discuss.elastic.co/t/differentiate-between-sections-within-a-single-log-file/168478/6 "2019-02-15T19:40:50Z")
Let them go through the csv filter, and them drop them. Something like
```
if [count] =~ /[a-z]/ { drop {} }
```
should discard the lines starting with rpId and dotId
---
### Author:  [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)
#### Post date: [February 16, 2019, 7:26pm UTC](https://discuss.elastic.co/t/differentiate-between-sections-within-a-single-log-file/168478/7 "2019-02-16T19:26:39Z")
Does this look correct, it isn't dropping the lines starting with rpld and dotId.
```auto
filter{
if [message] =~ /^\s*$/ {
drop {}
} else if [message] =~ /^TIME/ {
mutate { strip => ["message"] }
ruby {
init => '
@@metadata = {}
'
code => '
msg = event.get("message")
matches = msg.scan(/^([A-Za-z0-9]+)::(.*)/)
m = matches[0]
@@metadata[m[0]] = m[1]
'
}
drop {}
} else if [message] =~ " .* .* " {
csv {
separator => " "
columns => ["laneID","dotID","count","occ","reboots","batLvl","stuckHi","dwnTime","blips","channel","mode","t_slot","rssiAvg","rssiStd","lqiAvg","lqiStd","latAvg","latMed","spdAvg","spdMed","sdifAvg","sdif95","confidence","discover"]
}
if [count] =~ /[a-z]/ { drop {} }
ruby {
code => '
event.set("metadata", @@metadata)
'
}
}
}
```
Am I missing something here?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 16, 2019, 7:34pm UTC](https://discuss.elastic.co/t/differentiate-between-sections-within-a-single-log-file/168478/8 "2019-02-16T19:34:42Z")
%LOSS does not match /a-z/
```
if [count] =~ /[a-zA-Z]/ { drop {} }
```
plus you will need a final branch to process lines that do not contain tabs
} else { drop {} }
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 16, 2019, 7:34pm UTC](https://discuss.elastic.co/t/differentiate-between-sections-within-a-single-log-file/168478/9 "2019-03-16T19:34:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.