# How to extract rows with logstash

**URL:** https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236
**Category:** Logstash
**Created:** [November 2, 2017, 6:32pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236 "2017-11-02T18:32:44Z")
**Posts on this page:** 13
**Page:** 1

<div class="post-metadata">

### Author: ![citanionarrei](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@citanionarrei](https://discuss.elastic.co/u/citanionarrei)
#### Post date: [November 2, 2017, 6:32pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/1 "2017-11-02T18:32:44Z")

</div>

hi  
i have this log file

2017-06-08 12:10:12,924 | INFO | nager-plugin-out | ExtraDataServices | 302 -  
2017-06-08 12:10:12,925 | DEBUG | nager-plugin-out | deleteByExample | 285 -  
2017-06-08 12:10:12,925 | DEBUG | nager-plugin-out | deleteByExample | 285 - o  
2017-06-08 12:10:12,926 | DEBUG | nager-plugin-out |deleteByExample | 285 - o  
2017-06-08 12:10:12,926 | INFO | nager-plugin-out | ExtraDataServices | 302

I want to extract only the lines where the field INFO appears

i'm using this configuration on logstash

```
input{
file { path => "C:\Users\Lock\Desktop\info.3"
start_position => "beginning"	
  sincedb_path => "/dev/null"
  ignore_older => 0
}
} 

filter {
grok {		 
	match => ["message", "(?<info>^$.*\| INFO \|.*\n)"]
	break_on_match => true
	add_field => { "type" => "log_info"} 		
}
}

output {

file {
  path => "C:\Users\Lock\Desktop\Output_info.log"
} 
		
elasticsearch {
index => "info-%{+YYYY.MM.dd}"
document_type => "log_info"

}
stdout {codec=> rubydebug}

}

```

but this conf doesn't work becouse it take all rows !!

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 2, 2017, 7:49pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/2 "2017-11-02T19:49:23Z")

</div>

Replace

```
match => ["message", "(?<info>^$.*\| INFO \|.*\n)"]
```

with

```
match => ["message", "(?<info>.*\| INFO \|.*)"]
```

(single space, not two spaces after INFO). Then you can

```
if ! ( "" in [info] ) {drop{}}
```

---

<div class="post-metadata">

### Author: ![citanionarrei](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@citanionarrei](https://discuss.elastic.co/u/citanionarrei)
#### Post date: [November 2, 2017, 8:53pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/3 "2017-11-02T20:53:22Z")

</div>

ok perfect many thanks...now it work  
but there are two spaces after INFO 🙂  
right,Now I have this

```
{"path":"C:\\Users\\Lock\\Desktop\\as.3","@timestamp":"2017-11-02T20:13:03.325Z","@version":"1","host":"Lock","message":"2017-06-08 12:10:12,906 | INFO | nager-plugin-out |

```

ok, now I would like to delete fields of my path, my timestamp, host..  
Than i thinked to use "mutate filter" but....  
when i use it for the host field and for the path field it work properly,  
but if i use it also for @timestamp and @version fields , system give me an error

```
input{
file 
  { 	
 path => "C:\Users\Lock\Desktop\as.3"
   start_position => "beginning"	
   sincedb_path => "/dev/null"
 ignore_older => 0
 }
} 

filter {
grok {		 
	match => ["message", "(?<info>.*\| INFO \|.*)"]
	break_on_match => true
	add_field => {"type" => "log_info"}
	 }
	
if !( "" in [info]){drop{}}

mutate {remove_field => ["path","host"] }
#mutate {remove_field => ["path","@timestamp","@version","host"] }# if i use this row i have errors
	}

output {

file { path => "C:\Users\Lock\Desktop\Output_as.log"	} 
		
elasticsearch {
index => "error-%{+YYYY.MM.dd}"
document_type => "log_info"
			  }
stdout {codec=> rubydebug}

}

```

where is the error?  
where I'm wronging

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 3, 2017, 3:32pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/4 "2017-11-03T15:32:57Z")

</div>

What version are you running and what error message do you get?

In V6.0.0-rc1, which is what I am running, using remove\_field to remove @version and @timestamp works just fine. It may be a limitation in an earlier release.

Not sure why you would not want a timestamp. If you want to set the timestamp to the time in the message you can do that using

```
filter {
  grok {
    match => ["message", "(?<ts>[^|]*) \| %{WORD:level}%{SPACE} \| .*"]
  }
  if "DEBUG" == [level] {drop{}}
  date {
    match => ["ts", "yyyy-mm-DD HH:mm:ss,SSS"]
    timezone => "Europe/Rome"
  }
  mutate {remove_field => ["path", "host", "ts"] }
}
```

---

<div class="post-metadata">

### Author: ![citanionarrei](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@citanionarrei](https://discuss.elastic.co/u/citanionarrei)
#### Post date: [November 3, 2017, 3:41pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/5 "2017-11-03T15:41:59Z")

</div>

hi many thanks ...  
but i dont understand perfectly how work this line

> [@Badger](#):
>
> match =\> ["message", "(?\<ts\>[^|]_) | %{WORD:level}%{SPACE} | ._"]

ps: i don't want @timestamp becouse it is timestamp of my operation and i want only timestamp of the my line log

---

<div class="post-metadata">

### Author: ![citanionarrei](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@citanionarrei](https://discuss.elastic.co/u/citanionarrei)
#### Post date: [November 3, 2017, 4:40pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/6 "2017-11-03T16:40:24Z")

</div>

your grok filter is ok...thanks!!! 😄

> [@Badger](#):
>
> What version are you running and what error message do you get?
> 
> In V6.0.0-rc1, which is what I am running, using remove\_field to remove @version and @timestamp works just fine. It may be a limitation in an earlier release.

my version is 5.5.3  
the error is

```
[2017-11-03T17:37:53,610][FATAL][logstash.runner] An unexpected error occurred! {:error=>#<LogStash::Error: timestamp field is missing>, :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:202:in `sprintf'", "C:/Users/Lock/Desktop/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.7-java/lib/logstash/outputs/elasticsearch/common.rb:173:in `event_action_params'", "C:/Users/Lock/Desktop/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.7-java/lib/logstash/outputs/elasticsearch/common.rb:48:in `event_action_tuple'", "C:/Users/Lock/Desktop/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.7-java/lib/logstash/outputs/elasticsearch/common.rb:42:in `multi_receive'", "org/jruby/RubyArray.java:2414:in `map'", "C:/Users/Lock/Desktop/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.7-java/lib/logstash/outputs/elasticsearch/common.rb:42:in `multi_receive'", "C:/Users/Lock/Desktop/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in `multi_receive'", "C:/Users/Lock/Desktop/logstash/logstash-core/lib/logstash/output_delegator.rb:47:in `multi_receive'", "C:/Users/Lock/Desktop/logstash/logstash-core/lib/logstash/pipeline.rb:420:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "C:/Users/Lock/Desktop/logstash/logstash-core/lib/logstash/pipeline.rb:419:in `output_batch'", "C:/Users/Lock/Desktop/logstash/logstash-core/lib/logstash/pipeline.rb:365:in `worker_loop'", "C:/Users/Lock/Desktop/logstash/logstash-core/lib/logstash/pipeline.rb:330:in `start_workers'"]}

```

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 3, 2017, 4:47pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/7 "2017-11-03T16:47:47Z")

</div>

I see. It worked for me because I am only using a stdout / rubydebug output. So the remove\_field mutation worked just fine. The problem is the elasticsearch output requires every event to have a timestamp.

---

<div class="post-metadata">

### Author: ![citanionarrei](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@citanionarrei](https://discuss.elastic.co/u/citanionarrei)
#### Post date: [November 4, 2017, 10:44am UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/8 "2017-11-04T10:44:12Z")

</div>

> [@Badger](#):
>
> can do that using

hi friends  
i have another problem  
i want estract from this rows only where INFO and nager-plugin-out are present

```
2017-06-04 07:22:39,307 | INFO | ActiveMQ Task-1 | FailoverTransport | 128 - 
2017-06-05 09:04:36,082 | INFO | nager-plugin-out | ManagedManagementStrategy | 
2017-06-05 09:04:36,086 | INFO | nager-plugin-out | DefaultTypeConverter | 99 - 
2017-06-05 09:04:36,089 | INFO | nager-plugin-out | DefaultRuntimeEndpointRegistry |  
2017-06-05 09:04:36,090 | INFO | nager-plugin-out | DefaultCamelContext | 99 - 

```

i try to use this configuration but grok don't match

```
 filter {
 grok {
match => ["message", "(?<ts>[^|]*) \| %{WORD:level}%{SPACE} \|.*| %{WORD:level1}%{SPACE} \|.*"]

 }
 if ( "INFO" != [level] AND "nager-plugin-out" != [level1] ) {drop{}}
 date {
    match => ["ts", "yyyy-mm-DD HH:mm:ss,SSS"]
    timezone => "Europe/Rome"
   }

```

where am i wrong?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 4, 2017, 12:13pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/9 "2017-11-04T12:13:40Z")

</div>

There are four issues.

Firstly, boolean operators have to be lowercase. You should see a message like 'Expected one of #, and, or, xor, nand, ) at line 16, column 29 (byte 314) after filter ...'. So if you fold that AND down to and the configuration will at least compile,

Secondly, you want the condition to be or, not and.

Thirdly, you have an unescaped | in your regex, which is used for alternation (pattern1 or pattern2), and an extra .\*.

Fourthly, you have used WORD, which in my locale does not allow the word to contain -. It is a sequence of \w, which for me is [A-Za-z0-9\_]

Putting it all together, try this:

```
grok {
    match => ["message", "(?<ts>[^|]*) \| %{WORD:level}%{SPACE} \| (?<msg>[^|]*)%{SPACE} \|.*"]
  }
  if ( "INFO" != [level] or "nager-plugin-out" != [msg] ) {drop{}}
```

---

<div class="post-metadata">

### Author: ![citanionarrei](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@citanionarrei](https://discuss.elastic.co/u/citanionarrei)
#### Post date: [November 6, 2017, 12:40pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/10 "2017-11-06T12:40:24Z")

</div>

ok good...many thanks for help  
i have another problem

```
2017-07-13 13:32:32,562 | WARN | ...other words.. {"request":{"cks":31155,"terminal"....}

```

I can extract a line where there is a WARM but I also want to extract and process the Json that is there after {"request":{"cks":31155,"terminal"........}  
I'm trying with this conf.

```
filter {
grok {		 
	match => ["message", "(?<check>).*\| WARN \|.*(?<check_json>){\"request\".*}"]
	break_on_match => true
	add_field => {"type" => "log_warn"}
    }
if !( "" in [check]){drop{}}

mutate { remove_field => ["path","host","check"]
      add_tag => ["WARNING"]	}
	
	
if "_grokparsefailure" in [tags] {drop {}}	
else {
	json {
		source => "check_json"
	}}
}

```

what do you think?

---

<div class="post-metadata">

### Author: ![citanionarrei](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@citanionarrei](https://discuss.elastic.co/u/citanionarrei)
#### Post date: [November 6, 2017, 1:09pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/11 "2017-11-06T13:09:38Z")

</div>

ok it work in this way

```
grok {		 
match => ["message", "(?<check>.*\| WARN \|.*)(?<check_json>{\"request\".*})"]
break_on_match => true
add_field => {"type" => "log_warn"}
}
if !( "" in [check]){drop{}}

mutate { remove_field => ["path","host","check_json"]
  add_tag => ["WARNING"]	}

```

if i want send a mail with body with "message" and the "json" how i can make it?

```
email {
			 codec => "plain"
			 contenttype => "text/html; charset=UTF-8"

             ............

			 via => "smtp"
			 body => "%{message}",%{check_json}

```

in this way i see correctly only "message"

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 6, 2017, 6:06pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/12 "2017-11-06T18:06:09Z")

</div>

You used mutate/remove\_field to delete check\_json, so by the time it gets to the output filter it no longer exists.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 4, 2017, 6:06pm UTC](https://discuss.elastic.co/t/how-to-extract-rows-with-logstash/106236/13 "2017-12-04T18:06:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.