Grokparsefailure error is occurring

[vvsvinu (Vinu Vibhu Sobhana)]

Hai, I have been receiving the grokparsefailure error lately. Plz find the output that contains the "_grokparsefailure" and my configuration files are mentioned below. Plz help..

############################################# ERROR SNIPPET #############

"@timestamp" => 2018-01-16T04:59:22.000Z,
"type" => "tomcat_log",
"severity" => "ERROR"
}
{
"message" => " Source field value: nic.kerala.gst.registration.entity.gstn.Gstp@22411420b4[professionalAddressDetails=nic.kerala.gst.registration.entity.gstn.Address@30sacee48[addressId=1289041,bldgNum=test house wwwwww po,floorNum=,bldgName=,strt=sswaruvikuzhy,locty=anwwickadu,dst=KLKWWOT,stateCode=3112,pinCode=68622503,lat=,lon=,adressType=PA,aplnType=RTTR1,entyType=RTTR1,cntry=,cntryCode=,documents=[nic.kerala.gst.registration.entity.gstn.DocUpld@61ac2b85d],resubmittedDocuments=],enrollmentDetails=nic.kerala.gst.registration.entity.gstn.GstpEnrmtDtls@10f28eb5d,graduationDetails=nic.kerala.gst.registration.entity.gstn.GstpGrdtnDtls@49d289d53,applicantDetails=nic.kerala.gst.registration.entity.gstn.GstpAplntsDtls@505d6969,oldGstPractnrDetails=,queryClrfnData=,scnQueries=,modfctnRegnApln=,addnlInfo=,draftId=3404234,documents=,arnDtlsId=405608,aplnType=RTTR1,arn=AA3201180050423,createdDate=2018-01-12 21:00:16.497735,modifiedDate=,regnDtls=nic.kerala.gst.registration.entity.gstn.RegnDtls@3f0ffd52[arnDtlsId=405608,aplnType=RTTR1,aplnState=PFV,modfdDate=2018-01-12,dueDate=,email=ert345@gmail.com,mob=8222210651,date=,pan=FEPP22227L,pt=,legalNameBsns=,atzdSgnryName=,submitDate=,dsrctCode=KLKOT,stateCode=3222,aplnSts=,gstPrctnrName=xxx wwww,panOrTan=,cpin=,provsnlId=],decln=nic.kerala.gst.registration.entity.gstn.Decln@72956d9c[arnDtlsId=405608,declnPlace=aruwwvikuzhy,declnDate=2018-01-12,signType=EVC,asDescn=,asNum=,declnName=,declnText=,verified=true,applicantName=rrrrrrrr,pan=FEPPS0rr937L],origHash=b2brrrrrdef7f2af98525fed8b4c209e1rr64edc4e7feadaa1rb515a1158bed9,lastActionDate=2018-01-28,isMigrated=,securityToken=32184444400003599TRN,gstin=,current=true,alertDtlsId=]\n Dest parent class: nic.kerala.gst.registration.vo.GstpEnrmtDrrrrtlsVO\n Dest field name: gstp\n Dest field type: nic.kerala.gst.registration.entity.gstn.Gstp\norg.hibernate.LazyInitializationException: failed to lazily initialize a collection of role: nic.kerala.gst.registration.entity.gstn.GstpAplyyntsDtls.resubmittedPhotos, could not initialize proxy - no Session\n\tat org.hibernate.collection.internal.AbstractPersistentCollection.throwLazyInitializationException(AbstractPersistentCollection.java:582)\n\tat org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:201)\n\tat org.hibernate.collection.internal.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:145)\n\tat org.hibernate.collection.internal.PersistentBag.size(PersistentBag.java:261)\n\tat org.dozer.MappingProcessor.prepareDestinationList(MappingProcessor.java:837)\n\tat org.dozer.MappingProcessor.addOrUpdateToList(MappingProcessor.java:762)\n\tat org.dozer.MappingProcessor.addOrUpdateToList(MappingProcessor.java:850)\n\tat org.dozer.MappingProcessor.mapListToList(MappingProcessor.java:686)\n\tat org.dozer.MappingProcessor.mapCollection(MappingProcessor.java:541)\n\tat org.dozer.MappingProcessor.mapOrRecurseObject(MappingProcessor.java:434)\n\tat org.dozer.MappingProcessor.mapFromFieldMap(MappingProcessor.java:342)\n\tat org.dozer.MappingProcessor.mapField(MappingProcessor.java:288)\n\tat org.dozer.MappingProcessor.map(MappingProcessor.java:248)\n\tat org.dozer.MappingProcessor.map(MappingProcessor.java:197)\n\tat org.dozer.MappingProcessor.mapCustomObject(MappingProcessor.java:495)\n\tat org.dozer.MappingProcessor.mapOrRecurseObject(MappingProcessor.java:446)\n\tat org.dozer.MappingProcessor.mapFromFieldMap(MappingProcessor.java:342)\n\tat org.dozer.MappingProcessor.mapField(MappingProcessor.java:288)\n\tat org.dozer.MappingProcessor.map(MappingProcessor.java:248)\n\tat org.dozer.MappingProcessor.map(MappingProcessor.java:197)\n\tat org.dozer.MappingProcessor.mapCustomObject(MappingProcessor.java:495)\n\tat org.dozer.MappingProcessor.mapOrRecurseObject(MappingProcessor.java:446)\n\tat org.dozer.MappingProcessor.mapFromFieldMap(MappingProcessor.java:342)\n\tat org.dozer.MappingProcessor.mapField(MappingProcessor.java:288)\n\tat org.dozer.MappingProcessor.map(MappingProcessor.java:248)\n\tat org.dozer.MappingProcessor.map(MappingProcessor.java:197)\n\tat org.dozer.MappingProcessor.mapCustomObject(MappingProcessor.java:495)\n\tat o org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)\n\tat org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:486)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.lang.Thread.run(Thread.java:748)",
"fields" => {
"log_type" => "tomcat1_cat_log"
},
"prospector" => {
"type" => "log"
},
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_grokparsefailure"
],
"@version" => "1",
"host" => "kibana.ctax.dev",
"offset" => 967530,
"beat" => {
"name" => "kibana.ctax.dev",
"hostname" => "kibana.ctax.dev",
"version" => "6.1.1"
},
"source" => "/usr/local/tomcat/logs/catalina.out",
"@timestamp" => 2018-01-16T09:22:17.381Z,
"type" => "tomcat_log"
}

#############################################

*** The error enclosed along has been trimmed. The Complete error report has been enclosed along for reference

Complete error-log part-1

Complete error-log part-2

Complete error-log part-3

[vvsvinu (Vinu Vibhu Sobhana)]

My logstash.yml, logstash-tomcat.conf and filebeat.yml configurations are mentioned below

############# Filebeat.yml ###############
filebeat.prospectors:

type: log
enabled: true
multiline.match: after
multiline.negate: true
multiline.pattern: '[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}'
paths:
  - /usr/local/tomcat/logs/catalina.out
fields: {log_type: tomcat1_cat_log}
document_type: tomcat_log

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
host: "192.168.3.226:5601"
username: kibana
password: password
output.logstash:
username: logstash_system
password: password
hosts: ["192.168.3.226:5044"]

###################################################
####### conf.d/logstash_tomcat.conf ###############
input {
beats {
type => "tomcat_log"
host => "192.168.3.226"
port => 5044
}
}

filter {
if [type] == "tomcat_log" {
grok {
patterns_dir => "/etc/logstash/patterns"
match => [ "message", "(?m)%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:severity} %{GREEDYDATA:message}" ]
overwrite => [ "message" ]
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
remove_field => [ "timestamp" ]
}

}
}
output {
stdout { codec => rubydebug }
if [type] == "tomcat_log" {
elasticsearch {
manage_template => false
hosts => ["192.168.3.226:9200"]
user => elastic
password => password
}
}
}
##########################################

############# logstash.yml #################

path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d/*.conf
path.logs: /var/log/logstash

############################################

[vvsvinu (Vinu Vibhu Sobhana)]

any update ... plz help

[madhur_yadav (madhur)]

try this -

multiline.flush_pattern: 'java.lang.Thread.run(Thread.java:748)'

though it's hardcoded but it works. I'm also facing the same issue and found this workaround.

[vvsvinu (Vinu Vibhu Sobhana)]

Still facing the same error... Please help

My updated filebeat.yml file

#############
filebeat.prospectors:

type: log
enabled: true
multiline.match: after
multiline.negate: true
multiline.pattern: '[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}'
multiline.flush_pattern: 'java.lang.Thread.run(Thread.java:748)'
paths:
  - /usr/local/tomcat/logs/catalina.out

tail_files: true

fields: {log_type: tomcat1_cat_log}
document_type: tomcat_log

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
host: "192.168.3.226:5601"
username: kibana
password: password
output.logstash:
username: logstash_system
password: password
hosts: ["192.168.3.226:5044"]

#############

Is there any other modification that I need to implement to rectify this error.

[madhur_yadav (madhur)]

I think your multiline.pattern is wrong.

try this instead.

multiline.pattern: '[0-9]{4}-[0-9]{2}-[0-9]{2}'

[vvsvinu (Vinu Vibhu Sobhana)]

I have updated the multiline statement as suggested but still the same error is getting thrown.

[madhur_yadav (madhur)]

Are you using Rsyslog in sender and receiver side?
If yes can you please tell me their versions ?

[vvsvinu (Vinu Vibhu Sobhana)]

No Iam using logstash-6.1.1-1

[vvsvinu (Vinu Vibhu Sobhana)]

any updates