# \[ERROR\]\[logstash.instrument.periodicpoller.jvm\] Periodi cPoller: exception

**URL:** <https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281>\
**Category:** Logstash\
**Created:** [July 7, 2017, 2:08pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281 "2017-07-07T14:08:01Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 7, 2017, 2:08pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/1 "2017-07-07T14:08:01Z")

</div>

We are running filebeat,logstash, kibana and elasticsearach in Master machine(1 machine) and from slave machines(6 machines). We are just running Filebeat and all the data from those machines is transferred to master machine.

We are facing below two error in Logstash console  
We are thinking that the error occurred because we are loading data from 7 server (almost around 300 log files) and CPU utilization is reaching upto 100%

After the **timeout error** logstash stop working and data parsing and inserting in elasticsearch stop.

Errors:

1. 

2017-06-28T03:04:53,176][ERROR][logstash.filters.grok] Error while attempti  
g to check/cancel excessively long grok patterns {:message=\>"Mutex relocking by  
same thread", :class=\>"ThreadError", :backtrace=\>["org/jruby/ext/thread/Mutex.j  
va:90:in `lock'", "org/jruby/ext/thread/Mutex.java:147:in`synchronize'", "G:/E  
K\_Softwares/exe/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/logstash-filter-gro  
-3.2.3/lib/logstash/filters/grok/timeout\_enforcer.rb:38:in `stop_thread_groking ", "G:/ELK_Softwares/exe/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/logstash-f lter-grok-3.2.3/lib/logstash/filters/grok/timeout_enforcer.rb:53:in`cancel\_tim  
d\_out!'", "org/jruby/RubyHash.java:1342:in `each'", "G:/ELK_Softwares/exe/logst sh-5.0.0/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.2.3/lib/logstash/f lters/grok/timeout_enforcer.rb:45:in`cancel\_timed\_out!'", "org/jruby/ext/threa  
/Mutex.java:149:in `synchronize'", "G:/ELK_Softwares/exe/logstash-5.0.0/vendor/ undle/jruby/1.9/gems/logstash-filter-grok-3.2.3/lib/logstash/filters/grok/timeo t_enforcer.rb:44:in`cancel\_timed\_out!'", "G:/ELK\_Softwares/exe/logstash-5.0.0/  
endor/bundle/jruby/1.9/gems/logstash-filter-grok-3.2.3/lib/logstash/filters/gro  
/timeout\_enforcer.rb:63:in `start!'"]}

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [July 7, 2017, 2:13pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/2 "2017-07-07T14:13:26Z")

</div>

The error mentions "excessively long grok patterns".  
Do you mind sharing your config?

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 7, 2017, 2:16pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/3 "2017-07-07T14:16:11Z")

</div>

input  
{  
beats {  
port =\> 5042  
codec =\> multiline  
{  
pattern =\> "^#2.0"  
what =\> "previous"  
negate =\>"true"  
charset =\> "ISO-8859-1"  
}  
}  
}  
filter  
{  
if "trace" in [type]{  
grok {  
match =\> {"message" =\> "(?m)#%{DATA:m1}#%{DATA:DateTime}#%{DATA:Timezone}#%{DATA:Severity}#%{DATA:Category}#%{DATA:m6}#%{DATA:CustomerMessageComponent}#%{DATA:RuntimeComponent}#%{DATA:LogID}#%{DATA:CorrelationID}#%{DATA:Application}#%{DATA:Location}#%{DATA:User}#%{DATA:Session}#%{DATA:m2}#%{DATA:PassportSession}#%{DATA:PassportUserActivityID}#%{DATA:PassportConnection}#%{DATA:PassportConnectionCounter}#%{DATA:Thread}#%{DATA:m4}#%{DATA:m5}#%{GREEDYDATA:ErrorMessage}#"}  
}   
mutate  
{  
#remove\_tag =\> ["multiline"],  
remove\_field =\> ["m1","m4","m6"]  
strip =\> ["DateTime"]  
}  
date {  
match =\> ["DateTime", "YYYY MM dd HH:mm:ss:SSS"]  
timezone =\> "EST"  
target =\> "DateTime"  
}  
}

```
if "trc-prd" in [type]{
	grok {
		match => {"message" => "(?m)#%{DATA:prd_m1}#%{DATA:prd_DateTime}#%{DATA:prd_Timezone}#%{DATA:prd_Severity}#%{DATA:prd_Category}#%{DATA:prd_m6}#%{DATA:prd_CustomerMessageComponent}#%{DATA:prd_RuntimeComponent}#%{DATA:prd_LogID}#%{DATA:prd_CorrelationID}#%{DATA:prd_Application}#%{DATA:prd_Location}#%{DATA:prd_User}#%{DATA:prd_Session}#%{DATA:prd_m2}#%{DATA:prd_PassportSession}#%{DATA:prd_PassportUserActivityID}#%{DATA:prd_PassportConnection}#%{DATA:prd_PassportConnectionCounter}#%{DATA:prd_Thread}#%{DATA:prd_m4}#%{DATA:prd_m5}#%{GREEDYDATA:prd_ErrorMessage}#"}
	}		
	mutate
	{
		#remove_tag => ["multiline"],
		remove_field => ["prd_m1","prd_m4","prd_m6"]
		strip => ["prd_DateTime"]
	}
	date {
		match => ["prd_DateTime", "YYYY MM dd HH:mm:ss:SSS"]
		timezone => "EST"
		target => "prd_DateTime"
	}
}
	
if "application" in [type]{
	grok {
		match => {"message" => "(?m)#%{DATA:m1}#%{DATA:DateTime}#%{DATA:Timezone}#%{DATA:Severity}#%{DATA:Category}#%{DATA:m6}#%{DATA:CustomerMessageComponent}#%{DATA:RuntimeComponent}#%{DATA:LogID}#%{DATA:CorrelationID}#%{DATA:Application}#%{DATA:Location}#%{DATA:User}#%{DATA:Session}#%{DATA:m2}#%{DATA:PassportSession}#%{DATA:PassportUserActivityID}#%{DATA:PassportConnection}#%{DATA:PassportConnectionCounter}#%{DATA:Thread}#%{DATA:m4}#%{DATA:m5}#%{GREEDYDATA:ErrorMessage}#"}
	}		
	mutate
	{
		#remove_tag => ["multiline"],
		remove_field => ["m1","m4","m6"]
		strip => ["DateTime"]			
	}
	date {
		match => ["DateTime", "YYYY MM dd HH:mm:ss:SSS"]
		timezone => "EST"
		target => "DateTime"
	}
}
if "appl-prd" in [type]{
	grok {
		match => {"message" => "(?m)#%{DATA:prd_m1}#%{DATA:prd_DateTime}#%{DATA:prd_Timezone}#%{DATA:prd_Severity}#%{DATA:prd_Category}#%{DATA:prd_m6}#%{DATA:prd_CustomerMessageComponent}#%{DATA:prd_RuntimeComponent}#%{DATA:prd_LogID}#%{DATA:prd_CorrelationID}#%{DATA:prd_Application}#%{DATA:prd_Location}#%{DATA:prd_User}#%{DATA:prd_Session}#%{DATA:prd_m2}#%{DATA:prd_PassportSession}#%{DATA:prd_PassportUserActivityID}#%{DATA:prd_PassportConnection}#%{DATA:prd_PassportConnectionCounter}#%{DATA:prd_Thread}#%{DATA:prd_m4}#%{DATA:prd_m5}#%{GREEDYDATA:prd_ErrorMessage}#"}
	}		
	mutate
	{
		#remove_tag => ["multiline"],
		remove_field => ["prd_m1","prd_m4","prd_m6"]
		strip => ["prd_DateTime"]			
	}
	date {
		match => ["prd_DateTime", "YYYY MM dd HH:mm:ss:SSS"]
		timezone => "EST"
		target => "prd_DateTime"
	}
}
	
if "security" in [type]{
	grok {
		match => {"message" => "(?m)#%{DATA:security_m1}#%{DATA:security_DateTime}#%{DATA:security_Timezone}#%{DATA:security_Severity}#%{DATA:security_Category}#%{DATA:security_m6}#%{DATA:security_CustomerMessageComponent}#%{DATA:security_RuntimeComponent}#%{DATA:security_LogID}#%{DATA:security_CorrelationID}#%{DATA:security_Application}#%{DATA:security_Location}#%{DATA:security_User}#%{DATA:security_Session}#%{DATA:security_m2}#%{DATA:security_PassportSession}#%{DATA:security_PassportUserActivityID}#%{DATA:security_PassportConnection}#%{DATA:security_PassportConnectionCounter}#%{DATA:security_Thread}#%{DATA:security_m4}#%{DATA:security_m5}#%{GREEDYDATA:security_RemainingMessage}"}
		}

	if "LOGIN." in [security_RemainingMessage] {
	
		grok {
			match => {"security_RemainingMessage" => "%{DATA:security_ErrorMessage}\nUser: %{DATA:security_LoginUser}\nIP Address: %{DATA:security_IPAddress}\n%{GREEDYDATA:security_FinalRemainingMessage}"}
		}		
	}
	else {
		grok {
			match => {"security_RemainingMessage" => "%{DATA:security_ErrorMessage}#%{GREEDYDATA:security_FinalRemainingMessage}"}
		}	
	}
	
mutate
{
	#remove_tag => ["multiline"],
	remove_field => ["security_m1","security_m4","security_m6","security_FinalRemainingMessage","security_RemainingMessage"]
	rename => {"host" => "hostname"}
	strip => ["security_DateTime"]
}
date {
		match => ["security_DateTime", "YYYY MM dd HH:mm:ss:SSS"]
		timezone => "EST"
		target => "security_DateTime"
	}
}
if "sec-prd" in [type]{
	grok {
		match => {"message" => "(?m)#%{DATA:prd_security_m1}#%{DATA:prd_security_DateTime}#%{DATA:prd_security_Timezone}#%{DATA:prd_security_Severity}#%{DATA:prd_security_Category}#%{DATA:prd_security_m6}#%{DATA:prd_security_CustomerMessageComponent}#%{DATA:prd_security_RuntimeComponent}#%{DATA:prd_security_LogID}#%{DATA:prd_security_CorrelationID}#%{DATA:prd_security_Application}#%{DATA:prd_security_Location}#%{DATA:prd_security_User}#%{DATA:prd_security_Session}#%{DATA:prd_security_m2}#%{DATA:prd_security_PassportSession}#%{DATA:prd_security_PassportUserActivityID}#%{DATA:prd_security_PassportConnection}#%{DATA:prd_security_PassportConnectionCounter}#%{DATA:prd_security_Thread}#%{DATA:prd_security_m4}#%{DATA:prd_security_m5}#%{GREEDYDATA:prd_security_RemainingMessage}"}
		}

	if "LOGIN." in [prd_security_RemainingMessage] {
	
		grok {
			match => {"prd_security_RemainingMessage" => "%{DATA:prd_security_ErrorMessage}\nUser: %{DATA:prd_security_LoginUser}\nIP Address: %{DATA:prd_security_IPAddress}\n%{GREEDYDATA:prd_security_FinalRemainingMessage}"}
		}		
	}
	else {
		grok {
			match => {"prd_security_RemainingMessage" => "%{DATA:prd_security_ErrorMessage}#%{GREEDYDATA:prd_security_FinalRemainingMessage}"}
		}	
	}
	
mutate
{
	#remove_tag => ["multiline"],
	remove_field => ["prd_security_m1","prd_security_m4","prd_security_m6","prd_security_FinalRemainingMessage","prd_security_RemainingMessage"]
	rename => {"host" => "hostname"}
	strip => ["prd_security_DateTime"]
}
date {
		match => ["prd_security_DateTime", "YYYY MM dd HH:mm:ss:SSS"]
		timezone => "EST"
		target => "prd_security_DateTime"
	}
}

```

}

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 7, 2017, 2:17pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/4 "2017-07-07T14:17:05Z")

</div>

due to size constraint we are sending output seperately

output {  
if "trace" in [type]{  
elasticsearch {  
hosts =\> ["143.22.209.122"]  
index =\> "sap-trace-app-logs-nprd-qa"  
}  
}  
if "trc-prd" in [type]{  
elasticsearch {  
hosts =\> ["143.22.209.122"]  
index =\> "sap-trace-app-logs-prd"  
}  
}  
if "application" in [type]{  
elasticsearch {  
hosts =\> ["143.22.209.122"]  
index =\> "sap-trace-app-logs-nprd-qa"  
}  
}  
if "appl-prd" in [type]{  
elasticsearch {  
hosts =\> ["143.22.209.122"]  
index =\> "sap-trace-app-logs-prd"  
}  
}

```
if "security" in [type]{
	elasticsearch {
		hosts => ["143.22.209.122"]
		index => "sap-security-logs-nprd-qa"  
	}
}
if "sec-prd" in [type]{
	elasticsearch {
		hosts => ["143.22.209.122"]
		index => "sap-security-logs-prd"  
	}
}

```

}

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 7, 2017, 2:19pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/5 "2017-07-07T14:19:33Z")

</div>

Filebeat yml file

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common

# options. The filebeat.full.yml file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

# 

# You can find the full configuration reference here:

# [https://www.elastic.co/guide/en/beats/filebeat/index.html](https://www.elastic.co/guide/en/beats/filebeat/index.html)

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so

# you can use different prospectors for various configurations.

# Below are the prospector specific configurations.

- input\_type: log

- input\_type: log  
paths:

- input\_type: log  
paths:

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.

# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------  
#output.elasticsearch:

# Array of hosts to connect to.

#hosts: ["10.103.21.112:9200"]

# Optional protocol and basic auth credentials.

#protocol: "https"  
#username: "elastic"  
#password: "changeme"

#----------------------------- Logstash output --------------------------------  
output.logstash:

# The Logstash hosts

hosts: ["10.103.21.64:5042"]  
bulk\_max\_size: 1024  
loadbalance: true  
timeout: 120

# Optional SSL. By default is off.

# List of root certificates for HTTPS server verifications

#ssl.certificate\_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.

# Available log levels are: critical, error, warning, info, debug

#logging.level: debug

# At debug level, you can selectively enable logging only for some components.

# To enable all selectors use ["\*"]. Examples of other selectors are "beat",

# "publish", "service".

#logging.selectors: ["\*"]

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 7, 2017, 2:21pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/6 "2017-07-07T14:21:05Z")

</div>

ELK Version

**Version**  
elasticsearch-2.3.5  
filebeat-5.2.2  
kibana-4.5.4  
logstash-5.0.0

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 7, 2017, 2:25pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/7 "2017-07-07T14:25:03Z")

</div>

Please suggest some solution for this problem.

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 7, 2017, 2:27pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/8 "2017-07-07T14:27:26Z")

</div>

[ERROR][logstash.instrument.periodicpoller.jvm] Periodi  
cPoller: exception {:poller=\>#\<LogStash::Instrument::PeriodicPoller::JVM:0x2b744  
2a0 @task=#\<Concurrent::TimerTask:0x5e412c77 @observers=#\<Concurrent::Collection  
::CopyOnNotifyObserverSet:0x2d23ee71 @observers={#\<LogStash::Instrument::Periodi  
cPoller::JVM:0x2b7442a0 ...\>=\>:update}\>, @timeout\_interval=60.0

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 7, 2017, 2:28pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/9 "2017-07-07T14:28:02Z")

</div>

@StopEvent=#\<Concurrent::Event:0x3adecdc  
d @set=false, @iteration=0\>, @value=false, @copy\_on\_deref=nil, @dup\_on\_deref=nil

> > > , @options={:polling\_interval=\>1, :polling\_timeout=\>60}\>, :result=\>nil, :exce  
> > > ption=\>#\<Concurrent::TimeoutError: Concurrent::TimeoutError\>, :executed\_at=\>2017  
> > > -06-28 03:04:55 -0500}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 11, 2017, 5:14pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/10 "2017-07-11T17:14:46Z")

</div>

> match =\> {"message" =\> "(?m)#%{DATA:m1}#%{DATA:DateTime}#%{DATA:Timezone}#%{DATA:Severity}#%{DATA:Category}#%{DATA:m6}#%{DATA:CustomerMessageComponent}#%{DATA:RuntimeComponent}#%{DATA:LogID}#%{DATA:CorrelationID}#%{DATA:Application}#%{DATA:Location}#%{DATA:User}#%{DATA:Session}#%{DATA:m2}#%{DATA:PassportSession}#%{DATA:PassportUserActivityID}#%{DATA:PassportConnection}#%{DATA:PassportConnectionCounter}#%{DATA:Thread}#%{DATA:m4}#%{DATA:m5}#%{GREEDYDATA:ErrorMessage}#"}

This grok expression is _insanely_ inefficient. You _really_ need to reduce the number of DATA and GREEDYDATA patterns. I'm positive that it's possible to construct more exact expressions that aren't so expensive to process.

Or, since this appears to be a log with #-separated records, just use a csv filter.

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 12, 2017, 1:24pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/11 "2017-07-12T13:24:23Z")

</div>

Thank you for reply.

1. only {GREEDYDATA:ErrorMessage} this field is loading too much [data.It](http://data.It) gives me error:  
create"=\>{"\_index"=\>"sap-trace-app-logs-k8p", "\_type"=\>"trc-prd", "  
\_id"=\>"AV02Zi5YHAmgJsrjBUjd", "status"=\>400, "error"=\>{"type"=\>"illegal\_argument  
_exception", "reason"=\>"Document contains at least one immense term in field="p  
rd\_ErrorMessage" (whose UTF8 encoding is longer than the max length 32766), all  
of which were skipped. Please correct the analyzer to not produce such terms.  
The prefix of the first immense term is: '[10, 83, 105, 110, 103, 108, 101, 32,  
67, 108, 105, 99, 107, 32, 65, 112, 112, 114, 111, 118, 97, 108, 32, 58, 32, 10  
9, 80, 111, 115, 116]...', original message: bytes can be at most 32766 in lengt  
h; got 106157", "caused\_by"=\>{"type"=\>"max\_bytes\_length\_exceeded\_exception", "re  
ason"=\>"bytes can be at most 32766 in length; got 106157"}}}}}  
2.I applied "ignore\_above": 256 to this field in the index which resolves my timeout error but I don't want to loose data.  
ignore\_above": 256 will ignore the whole record.  
3.I can see my CPU % is also reaches to 100%.  
4.And while loading dashboard ion kibana I am facing below error.  
RemoteTransportException[[Corona][143.22.209.122:9300][indices:data/read/search[  
phase/query]]]; nested: CircuitBreakingException[[request] Data too large, data  
for [] would be larger than limit of [381891379/364.1mb]];  
Caused by: CircuitBreakingException[[request] Data too large, data for [\<reused_  
arrays\>] would be larger than limit of [381891379/364.1mb]]  
5.Getting Visualize Request Timeout after 30000ms error while loading dashboard.  
I think all these are related to that field .Is there way I can resolve this without loosing data.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 12, 2017, 1:36pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/12 "2017-07-12T13:36:00Z")

</div>

> only {GREEDYDATA:ErrorMessage} this field is loading too much data

What do you mean? Show your configuration, don't describe it.

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 12, 2017, 2:20pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/13 "2017-07-12T14:20:03Z")

</div>

input  
{  
beats {  
port =\> 5111  
codec =\> multiline  
{  
pattern =\> "^#2.0"  
what =\> "previous"  
negate =\>"true"  
charset =\> "ISO-8859-1"  
}  
}

}  
filter  
{  
if "trace" in [type]{  
grok {  
match =\> {"message" =\> "(?m)#%{DATA:m1}#%{DATA:DateTime}#%{DATA:Timezone}#%{DATA:Severity}#%{DATA:Category}#%{DATA:m6}#%{DATA:CustomerMessageComponent}#%{DATA:RuntimeComponent}#%{DATA:LogID}#%{DATA:CorrelationID}#%{DATA:Application}#%{DATA:Location}#%{DATA:User}#%{DATA:Session}#%{DATA:m2}#%{DATA:PassportSession}#%{DATA:PassportUserActivityID}#%{DATA:PassportConnection}#%{DATA:PassportConnectionCounter}#%{DATA:Thread}#%{DATA:m4}#%{DATA:m5}#%{GREEDYDATA:ErrorMessage}#"}  
}   
mutate  
{  
#remove\_tag =\> ["multiline"],  
remove\_field =\> ["m1","m4","m6"]  
strip =\> ["DateTime"]  
}  
date {  
match =\> ["DateTime", "YYYY MM dd HH:mm:ss:SSS"]  
timezone =\> "EST"  
target =\> "DateTime"  
}  
}

if "application" in [type]{  
grok {  
match =\> {"message" =\> "(?m)#%{DATA:m1}#%{DATA:DateTime}#%{DATA:Timezone}#%{DATA:Severity}#%{DATA:Category}#%{DATA:m6}#%{DATA:CustomerMessageComponent}#%{DATA:RuntimeComponent}#%{DATA:LogID}#%{DATA:CorrelationID}#%{DATA:Application}#%{DATA:Location}#%{DATA:User}#%{DATA:Session}#%{DATA:m2}#%{DATA:PassportSession}#%{DATA:PassportUserActivityID}#%{DATA:PassportConnection}#%{DATA:PassportConnectionCounter}#%{DATA:Thread}#%{DATA:m4}#%{DATA:m5}#%{GREEDYDATA:ErrorMessage}#"}  
}   
mutate  
{  
#remove\_tag =\> ["multiline"],  
remove\_field =\> ["m1","m4","m6"]  
strip =\> ["DateTime"]   
}  
date {  
match =\> ["DateTime", "YYYY MM dd HH:mm:ss:SSS"]  
timezone =\> "EST"  
target =\> "DateTime"  
}  
}  
if "security" in [type]{  
grok {  
match =\> {"message" =\> "(?m)#%{DATA:security\_m1}#%{DATA:security\_DateTime}#%{DATA:security\_Timezone}#%{DATA:security\_Severity}#%{DATA:security\_Category}#%{DATA:security\_m6}#%{DATA:security\_CustomerMessageComponent}#%{DATA:security\_RuntimeComponent}#%{DATA:security\_LogID}#%{DATA:security\_CorrelationID}#%{DATA:security\_Application}#%{DATA:security\_Location}#%{DATA:security\_User}#%{DATA:security\_Session}#%{DATA:security\_m2}#%{DATA:security\_PassportSession}#%{DATA:security\_PassportUserActivityID}#%{DATA:security\_PassportConnection}#%{DATA:security\_PassportConnectionCounter}#%{DATA:security\_Thread}#%{DATA:security\_m4}#%{DATA:security\_m5}#%{GREEDYDATA:security\_RemainingMessage}"}  
}

```
	if "LOGIN." in [security_RemainingMessage] {
	
		grok {
			match => {"security_RemainingMessage" => "%{DATA:security_ErrorMessage}\nUser: %{DATA:security_LoginUser}\nIP Address: %{DATA:security_IPAddress}\n%{GREEDYDATA:security_FinalRemainingMessage}"}
		}		
	}
	else {
		grok {
			match => {"security_RemainingMessage" => "%{DATA:security_ErrorMessage}#%{GREEDYDATA:security_FinalRemainingMessage}"}
		}	
	}
mutate
{
	#remove_tag => ["multiline"],
	remove_field => ["security_m1","security_m4","security_m6","security_FinalRemainingMessage","security_RemainingMessage"]
	rename => {"host" => "hostname"}
	strip => ["security_DateTime"]
}
date {
		match => ["security_DateTime", "YYYY MM dd HH:mm:ss:SSS"]
		timezone => "EST"
		target => "security_DateTime"
	}
}

```

}

output {  
if "trace" in [type]{  
elasticsearch {  
hosts =\> ["143.22.209.122"]  
index =\> "test"  
}  
}  
if "application" in [type]{  
elasticsearch {  
hosts =\> ["143.22.209.122"]  
index =\> "test"  
}  
}

```
if "security" in [type]{
	elasticsearch {
		hosts => ["143.22.209.122"]
		index => "security-test"  
	}
}
```

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 12, 2017, 2:22pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/14 "2017-07-12T14:22:33Z")

</div>

Please check configuration file

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 12, 2017, 2:24pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/15 "2017-07-12T14:24:26Z")

</div>

Did you understand anything of what I wrote last time? You still have too many DATA and GREEDYDATA patterns in your grok filters. You need to fix that. Over and out.

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 12, 2017, 2:27pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/16 "2017-07-12T14:27:21Z")

</div>

True .I have too many DATA and GREEDYDATA pattern but it is required to read different log files.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 12, 2017, 2:29pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/17 "2017-07-12T14:29:26Z")

</div>

> but it is required to read different log files.

No, it's not.

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 13, 2017, 1:04pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/18 "2017-07-13T13:04:38Z")

</div>

Hello,

Now I am using CSV filter.

Config file  
input {  
beats {  
port =\> 5042  
codec =\> multiline  
{  
pattern =\> "^#2.0"  
what =\> "previous"  
negate =\>"true"  
charset =\> "ISO-8859-1"  
}  
}  
}  
filter {  
csv {  
columns =\> ["m1","DateTime","Severity","Category","m6","CustomerMessageComponent", "RuntimeComponent" ,"LogID","CorrelationID","Application","Location","User","Session","m2","PassportSession","PassportUserActivityID","PassportConnection","PassportConnectionCounter","Thread","m4","m5","ErrorMessage"]  
separator =\> ","  
}  
}  
output {  
if "trace" in [type]{  
elasticsearch {  
hosts =\> ["10.107.80.64"]  
index =\> "testcsv"  
}  
}

I am not able to see all the fields in my index testcsv.

 ![](https://us1.discourse-cdn.com/elastic/original/3X/5/7/57991f14cd619c565bc991cfbbc6c6ca0756a156.png)  
Also data is not collecting in to the correct field.

Sample data:

> 2.0#2017 07 12 00:49:15:943#0-500#Error#com.sap.engine.services.ejb3.runtime#  
> com.sap.ASJ.ejb.005017#BC-JAS-EJB#ejb#C0008F16D17A09C50000000300001D48#29014150000000005#[sap.com/com.sap.xi.mdt.beans#com.sap.engine.services.ejb3.runtime#SM\_COLL\_GTA#2129##EE2669CB66C411E7911C000001BAB886#ee2669cb66c411e7911c000001bab886##0#Thread](http://sap.com/com.sap.xi.mdt.beans#com.sap.engine.services.ejb3.runtime#SM_COLL_GTA#2129##EE2669CB66C411E7911C000001BAB886#ee2669cb66c411e7911c000001bab886##0#Thread)[HTTP Worker [@924290866],5,Dedicated\_Application\_Thread]#Plain##  
> System exception
> 
> #

Only m1 field shows #2.0#2017 07 12 00:48:15:851#0-500#Error#com.sap.engine.services.ejb3.runtime# this much data.

Please let me know how i will get correct data in all the fields.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 13, 2017, 2:15pm UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/19 "2017-07-13T14:15:07Z")

</div>

Your separator character is #, not a comma.

---

<div class="post-metadata">

**Author:** ![swatititame](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@swatititame](https://discuss.elastic.co/u/swatititame)\
**Post date:** [July 14, 2017, 10:02am UTC](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281/20 "2017-07-14T10:02:27Z")

</div>

I am using below config file  
input {  
beats {  
port =\> 5042  
codec =\> multiline  
{  
pattern =\> "^#2.0"  
what =\> "previous"  
negate =\>"true"  
charset =\> "ISO-8859-1"  
}  
}  
}  
filter {  
csv {  
separator =\> "#"

#m1#DateTime#Timezone#Severity#Category#m6#CustomerMessageComponent#RuntimeComponent#LogID#CorrelationID#Application#Location#User#Session#m2#PassportSession#PassportUserActivityID#PassportConnection#PassportConnectionCounter#Thread#m4#m5#ErrorMessage  
columns =\> ["m1","DateTime","Timezone","Severity","Category","m6","CustomerMessageComponent", "RuntimeComponent" ,"LogID","CorrelationID","Application","Location","User","Session","m2","PassportSession","PassportUserActivityID","PassportConnection","PassportConnectionCounter","Thread","m4","m5","ErrorMessage"]

}  
}

output {  
if "trace" in [type]{  
elasticsearch {  
hosts =\> ["10.103.20.64"]  
index =\> "csvconfigfour"  
} } }  
but still I am not getting data in correct field.

 ![](https://us1.discourse-cdn.com/elastic/original/3X/8/c/8c77f21893c1aa4ef6a6612131d980a2c42fc7b5.png)

[Next page](https://discuss.elastic.co/t/error-logstash-instrument-periodicpoller-jvm-periodi-cpoller-exception/92281.md?page=2)