GROK filtering problem

robegome · September 11, 2018, 3:05pm

Hello everyone!

I am creating this topic because I am not sure why I have a different behavior between a GROK debugger web site VS Kibana (Dev Tools / Grok Debugger section), in fact I am not sure if this thread should be here or in Kibana... PART 1

robegome · September 11, 2018, 3:06pm

PART2 (first part of the HTML log):

Let me explain the strange behavior... I have this HTML log:

</a></p><h1><a name="/30700311">Job 30700311</a></h1><a name="/30700311">
<table>
<tbody><tr>
<th>Domain</th><td>a427077</td>
</tr>
<tr>
<th>Service</th><td>clone1</td>
</tr>
<tr>
<th>Namespace</th><td>dbaas</td>
</tr>
<tr>
<th>Service Type</th><td>dbaas</td>
</tr>
<tr>
<th>Compute Site</th><td>US006_Z51</td>
</tr>
<tr>
<th>Username</th><td>gaurav.d.agarwal@oracle.com</td>
</tr>
<tr>
<th>Operation</th><td>create-dbaas-service</td>
</tr>
<tr>
<th>Status</th><td>Failed</td>
</tr>
<tr>
<th>Sub Status</th><td></td>
</tr>
<tr>
<th>Create Time</th><td>2018-06-30T06:18:37.181+00:00</td>
</tr>
<tr>
<th>Start Time</th><td>2018-06-30T07:03:01.708+00:00</td>
</tr>
<tr>
<th>End Time</th><td>2018-06-30T07:46:35.466+00:00</td>
</tr>
<tr>
<th>Update Time</th><td>2018-06-30T07:46:35.477+00:00</td>
</tr>
<tr>
<th>Job Info</th><td></td>
</tr>
<tr>
<th>Request Parameters</th><td>{{{identity_domain_id=a427077, noRetry=false, createStorageContainerIfMissing=false, assignPublicIP=true, serviceType=dbaas, version=12.1.0.2, ibkup=no, usableStorage=25, snapshotName=snap1, backupDestination=NONE, notificationEmail=gaurav.d.agarwal@oracle.com, useOAuthForStorage=false, namespace=dbaas, serviceInstance=clone1, charset=AL32UTF8, dbConsolePort=1158, subscriptionType=HOURLY, isBYOL=false, disasterRecovery=false, enableNotification=true, goldenGate=false, sid=ORCL, shape=oc3, emExpressPort=5500, ncharset=AL16UTF16, noRollback=false, edition=EE_EP, listenerPort=1521, serviceEntitlementId=550877656, tenant=a427077, serviceVersion=12.1.0.2, hdg=false, failoverDatabase=false, operationName=create-dbaas-service, sla=NONE, isRac=false, provisioningTimeout=180, ibkupOnPremise=false, level=PAAS, tags=[], useHighPerformanceStorage=false, pdbName=PDB1, enableListenerPort=false, count=2, timezone=UTC, description=clone1, computeSiteName=US006_Z51, server_base_uri=https://paassvcmngrinternal-us2-sm02.oraclecloud.com:8888/paas/service/dbcs/, vmPublicKeyText=ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxkq7rxugT7xgHEujJzFlUKRXaMMuKWJX1d3EpXvCSOZd6L2eqsNhEZE9Wztzo0mZBREKJjvf5PlvMWRVl9XvI91eHDXcvTsjL2aPGApkY0ZQzA3AXckWwrJhfPVLJVfOj1/dh02dxlf7G6kr5CPeh26UO6bvxVAG0NHstwDluYzfz/KXiOEjI0jJy1UJMLaRZhEVLix4NLK/ypT6gcGj6v+6XkvMZgvOE+Jtq4rQ4juBxd5KIsGwe2YkTFyM4jdKHrXFP3TUKolM54dOzBsiLMrxdVgKMiw0/e7qz4Wt/8nbfzzQTg+4WJ2VguGjNzksRXvKeSkPli7jZzxYwm2HPQ== rchennoj@adc2190484, serviceName=clone1, sourceServiceName=rman1, trial=false}}}</td>
</tr>
<tr>
<th>Supplemental Logs</th><td>none</td>
</tr>
<tr>
<th>Summary</th><td>
<pre>Job &lt;30700311&gt; v47, action=handleFailure, Failed, namespace=dbaas, version=12.1.0.2, operation=create-dbaas-service, cleanupActionIndex=1, retryCount:0, jobRetryCount:1, jobRetryWaitTime:0, created: 2018-06-30T06:18:37.181+0000, started: 2018-06-30T07:03:01.708+0000, to retry: 2018-06-30T07:02:58.960+0000, failingStartTime: 2018-06-30T07:41:32.549+0000, domain:a427077, instance:clone1, wm:SM-MS-chr302ru26.usdc2.oracleclo, owner:gaurav.d.agarwal@oracle.com

FAILED CURRENT JOB 30700311: action: dbServiceStarted

FAILED CHILD JOB 30687363: action: checkDatabaseConfig
    code: PSM-DB-015, message: Failed to configure Oracle Database Server...[
SCRIPT execution errors in [129.144.146.41]: 
Reloading listener after enable_vncr for LISTENER
listener reload DONE for LISTENER
DONE
INFO : Running dbca
ERROR : There was an error while untar the DB bits.
ERROR : Assistant dbda has failed, please check ocde logfile /var/opt/oracle/log/ORCL/ocde/ocde_2018-06-30_06:23:53.log
INFO: Total time taken by ocde is 1985 seconds 

#### Completed OCDE with errors, please check logs ####
### End: /var/opt/oracle/ocde/ocde.out


]
FAILED CHILD JOB 30688343: action: checkDatabaseConfig
    code: PSM-DB-015, message: Failed to configure Oracle Database Server...[
SCRIPT execution errors in [129.144.146.41]: 
Reloading listener after enable_vncr for LISTENER
listener reload DONE for LISTENER
DONE
INFO : Running dbca
ERROR : There was an error while untar the DB bits.
ERROR : Assistant dbda has failed, please check ocde logfile /var/opt/oracle/log/ORCL/ocde/ocde_2018-06-30_07:08:27.log
INFO: Total time taken by ocde is 1965 seconds 

#### Completed OCDE with errors, please check logs ####
### End: /var/opt/oracle/ocde/ocde.out


]</pre>
</td>
</tr>
</tbody></table>

robegome · September 11, 2018, 3:07pm

PART 3 (second part of the HTML log & my GROK FILTER):

<h1>Logs</h1>
<h2>setTags</h2>
<hr>
<pre>v5, SM-MS-chr302ru26.usdc2.oracleclo, SM-MS-chr302ru26.usdc2.oracleclo-1530301096849, appsmser, chr302ru26.usdc2.oraclecloud.com, sm version=18.2.6-551; SHA=91a8de2e9fe43e93e240a0a861209429d19a3827; build.date=2018-06-22 02:13
2018-06-30T06:18:40.398+00:00 [INFO]:executing action: setTags
2018-06-30T06:18:40.398+00:00 [INFO]:Job Attributes: 
2018-06-30T06:18:40.405+00:00 [INFO]:finished action: setTags
2018-06-30T06:18:40.406+00:00 [INFO]:Action returned status: SUCCESS
</pre>
<h2>setComputeNamespace</h2>
<hr>
<pre>v6, SM-MS-chr302ru26.usdc2.oracleclo, SM-MS-chr302ru26.usdc2.oracleclo-1530301096849, appsmser, chr302ru26.usdc2.oraclecloud.com, sm version=18.2.6-551; SHA=91a8de2e9fe43e93e240a0a861209429d19a3827; build.date=2018-06-22 02:13
2018-06-30T06:18:40.420+00:00 [INFO]:executing action: setComputeNamespace
2018-06-30T06:18:40.420+00:00 [INFO]:Job Attributes: 
2018-06-30T06:18:40.420+00:00 [INFO]:Security context = SecurityContext: user= gaurav.d.agarwal@oracle.com, domain=a427077, roles=[]
2018-06-30T06:18:40.420+00:00 [INFO]:Job Retry count = 0
2018-06-30T06:18:40.462+00:00 [INFO]:finished action: setComputeNamespace
2018-06-30T06:18:40.463+00:00 [INFO]:Action returned status: SUCCESS
</pre>
<h2>createDBService</h2>
<hr>
<pre>v7, SM-MS-chr302ru26.usdc2.oracleclo, SM-MS-chr302ru26.usdc2.oracleclo-1530301096849, appsmser, chr302ru26.usdc2.oraclecloud.com, sm version=18.2.6-551; SHA=91a8de2e9fe43e93e240a0a861209429d19a3827; build.date=2018-06-22 02:13
2018-06-30T06:18:40.476+00:00 [INFO]:executing action: createDBService
2018-06-30T06:18:40.476+00:00 [INFO]:Job Attributes: 
2018-06-30T06:18:40.477+00:00 [INFO]:Started action [create-dbaas-service#createDBService] for service [clone1] in identity domain [a427077]
2018-06-30T06:18:40.485+00:00 [INFO]:Child job 30687297 submitted by parent 30700311 with request: {edition=EE_EP, vmPublicKeyText=ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxkq7rxugT7xgHEujJzFlUKRXaMMuKWJX1d3EpXvCSOZd6L2eqsNhEZE9Wztzo0mZBREKJjvf5PlvMWRVl9XvI91eHDXcvTsjL2aPGApkY0ZQzA3AXckWwrJhfPVLJVfOj1/dh02dxlf7G6kr5CPeh26UO6bvxVAG0NHstwDluYzfz/KXiOEjI0jJy1UJMLaRZhEVLix4NLK/ypT6gcGj6v+6XkvMZgvOE+Jtq4rQ4juBxd5KIsGwe2YkTFyM4jdKHrXFP3TUKolM54dOzBsiLMrxdVgKMiw0/e7qz4Wt/8nbfzzQTg+4WJ2VguGjNzksRXvKeSkPli7jZzxYwm2HPQ== rchennoj@adc2190484, count=2, root_provisioning_job_id=30700311, assignPublicIP=true, subscriptionType=HOURLY, dbConsolePort=1158, namespace=dbaas, enableListenerPort=false, timezone=UTC, level=PAAS, pdbName=PDB1, tenant=a427077, snapshotName=snap1, description=clone1, sla=NONE, ncharset=AL16UTF16, useOAuthForStorage=false, enableNotification=true, charset=AL32UTF8, serviceVersion=12.1.0.2, shape=oc3, identity_domain_id=a427077, serviceType=dbaas, usableStorage=25, server_base_uri=https://paassvcmngrinternal-us2-sm02.oraclecloud.com:8888/paas/service/dbcs/, computeSiteName=US006_Z51, isBYOL=false, noRetry=false, isRac=false, provisioningTimeout=180, notificationEmail=gaurav.d.agarwal@oracle.com, createStorageContainerIfMissing=false, listenerPort=1521, version=12.1.0.2, serviceName=clone1, useHighPerformanceStorage=false, serviceInstance=clone1, failoverDatabase=false, emExpressPort=5500, serviceEntitlementId=550877656, trial=false, tags=[], sid=ORCL, noRollback=false, operationName=create-db-service, goldenGate=false, backupDestination=NONE, ibkup=no, disasterRecovery=false, sourceServiceName=rman1, hdg=false, ibkupOnPremise=false}
2018-06-30T06:18:40.485+00:00 [INFO]:Finished action [create-dbaas-service#createDBService] for service [clone1] in identity domain [a427077], status [WAIT]
2018-06-30T06:18:40.491+00:00 [INFO]:finished action: createDBService
2018-06-30T06:18:40.491+00:00 [INFO]:Action returned status: WAIT
</pre>

and this is the Grok filtering that I am using:

<h1><a name=%{QS}>%{DATA:Job_word}%{SPACE}%{NUMBER:job_id}</a></h1><a name=%{QS}>
<table>
<tbody><tr>
<th>%{DATA:Domain_word}</th><td>%{DATA:Domain_id}</td>
</tr>
<tr>
<th>%{DATA:Service_word}</th><td>%{DATA:Service_id}</td>
</tr>
<tr>
<th>%{DATA:NameSpace_word}</th><td>%{DATA:NameSpace_id}</td>
</tr>
<tr>
<th>%{DATA:ServiceType_word}</th><td>%{DATA:ServiceType_id}</td>
</tr>
<tr>
<th>%{DATA:ComputeSite_word}</th><td>%{DATA:ComputeSite_id}</td>
</tr>
<tr>
<th>%{DATA:Username_word}</th><td>%{DATA:Username_value}</td>
</tr>
<tr>
<th>%{DATA:Operation_word}</th><td>%{DATA:Operation_value}</td>
</tr>
<tr>
<th>%{DATA:Status_word}</th><td>%{DATA:Status_value}</td>
</tr>
<tr>
<th>%{DATA:SbSts_word}</th><td>%{DATA:SbSts_id}</td>
</tr>
<tr>
<th>%{DATA:CreateTime_word}</th><td>%{DATA:CreateTime_time}</td>
</tr>
<tr>
<th>%{DATA:StartTime_word}</th><td>%{DATA:StartTime_time}</td>
</tr>
<tr>
<th>%{DATA:EndTime_word}</th><td>%{DATA:EndTime_time}</td>
</tr>
<tr>
<th>%{DATA:UpdateTime_word}</th><td>%{DATA:UpdateTime_time}</td>
</tr>
<tr>
<th>%{DATA:JobInfo_word}</th><td>%{DATA:JobInfo_text}</td>
</tr>
<tr>
<th>%{DATA:ReqParam_word}</th><td>%{GREEDYDATA:ReqParam_value}</td>
</tr>
<tr>
<th>%{DATA:SupplementalLogs_word}</th><td>%{DATA:SupplementalLogs_value}</td>
</tr>
<tr>
<th>%{DATA:Summary_word}</th><td>
<pre>(?<message>(.|\r|\n)*)
</td>
</tr>
</tbody></table>

robegome · September 11, 2018, 3:08pm

PART 4 (strange behavior):

now using images, the first one is the Grok Debugger Site output (and it seems OK):

... but the second image is the Grok Debugger (DevTools Kibana) and here i am receiving an error:

...so my doubt here is if I have some error in my filtering (parsing) or not... could you please help me?

Thanks!

system · October 9, 2018, 3:08pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
HTML log file / PARSE FAILURE Logstash	2	1125	October 3, 2018
Grok parse failure in LogStash, but works in Kibana debugger Logstash	6	1804	January 23, 2019
Probleme de synchronisation entre logstash et kibana Discussions en français	11	988	March 12, 2021
_grokparsefailure but it work on grok debugger Logstash	19	666	July 30, 2018
[Logstash] Grok Filter does not work _grokparsefailure Logstash	16	652	March 22, 2022

GROK filtering problem

Related topics