Logstash output not the same as grok debugger

Elastic Stack Logstash
1

hello guys,
my grok debugger pattern output doesn't look like the logstash output in kibana!! any idea !

Selection_007
Selection_007.png954×579 25.9 KB

here is in kibana :

Selection_009
Selection_009.jpg913×220 111 KB

2

any help!

3

Please do not post pictures of text, just post the text itself. What does your pattern look like and what do the lines of text you are processing look like?

4 
2019-03-11 11:12:40,670 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-28144: Echec de l'exécution du gestionnaire d'audit détaillé
ORA-20417: ERROR SECURITY DATA
ORA-06512: à "PRODUCTION_AUDIT", ligne 39
ORA-06512: à ligne 1

and this is the pattern used that correctly match in grok debugger:

%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<classname>[^\]]+)\] %{GREEDYDATA:Error}(?<msg>[^\)]+)
5

and also it regroup all other log events in one record!!

{
  "_index": "filebeat-2019.04.08",
  "_type": "doc",
  "_id": "HQJe_GkBGmfQmneQJD3j",
  "_version": 1,
  "_score": null,
  "_source": {
    "number": "0",
    "n": "null",
    "prospector": {},
    "classname": "impl.SensusAnalyzeLogger",
    "Url": "business.service.interfaces.ICroneJobsService",
    "beat": {},
    "@version": "1",
    "Loglevel": "INFO",
    "timestamp": "2019-03-11 00:00:00,013",
    "msg": "getMapAllCronedJobsParams | null | >> entry >>  | null\n2019-03-11 00:00:00,014 INFO  [impl.SensusAnalyzeLogger] null | 0 | null | persistence.manager.interfaces.ICroneEntityManager | getMapJobsCronPramas | null | >> entry >>  | null\n2019-03-11 00:00:00,016 INFO  [com.sensus.persistence.commons.query.impl.SensusHibernateQueryFacade] @PARAMETER_BY_ID@|query time :2 ms |result size : 6|{paramIds=[7, 8, 6, 11, 9, 10]}\n2019-03-11 00:00:00,016 INFO  [log.impl.SensusAnalyzeLogger] null | 0 | null | persistence.manager.interfaces.ICroneEntityManager | getMapJobsCronPramas | null | << exit  <<  | 2 ms\n2019-03-11 00:00:00,016 INFO  [impl.SensusAnalyzeLogger] null | 0 | null | service.interfaces.ICroneJobsService | getMapAllCronedJobsParams | null | << exit  <<  | 3 ms\n2019-03-11 00:01:00,014 INFO  [impl.SensusAnalyzeLogger] null | 0 | null | service.interfaces.ICroneJobsService | getMapAllCronedJobsParams | null | >> entry >>  | null\n
6

OK, so your pattern says to collect everything into Error, followed by one of more characters that are not close-parenthesis grouped into msg. And that's what you get -- one character in msg.

If you want to store everything up to the first newline in Error and all the rest in msg, then tell grok that.

    grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<classname>[^\]]+)\] (?<Error>[^
]+)%{GREEDYDATA:msg}" } }
8

2019-03-11 09:11:42,233 ERROR [impl.CommonModificationBudgetaireBusinessServiceImpl] DISPONIBLE_MC_NON com.sensus.common.exception.SensusFunctionalException: DISPONIBLE_MC_NON at verifierDisponibleGlobaleMC(CommonModificationBudgetaireBusinessServiceImpl.java:8474)

my pattern is working with java log too so i have to stop on ")" character for that i put (?<msg>[^)]+) at the end, and i guess %{GREEDYDATA:msg} can't do this also i tried the pattern you suggest in grok debugger and it shows Compile ERROR

9

what do you see :thinking:

10

With the grok filter I showed I get

       "msg" => "\nORA-20417: ERROR SECURITY DATA\nORA-06512: à \"PRODUCTION_AUDIT\", ligne 39\nORA-06512: à ligne 1",
     "Error" => "ORA-28144: Echec de l\\'exécution du gestionnaire d\\'audit détaillé",
11

and i get this !!!

[parse_exception] [patterns] Invalid regex pattern found in: [%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<classname>[^\]]+)\] (?<Error>[^]+)%{GREEDYDATA:msg}]. empty char-class, with { header={ processor_type="grok" & property_name="patterns" } }

12

That would suggest you are trying to use

 grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<classname>[^\]]+)\] (?<Error>[^]+)%{GREEDYDATA:msg}" } }

rather than

        grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<classname>[^\]]+)\] (?<Error>[^
]+)%{GREEDYDATA:msg}" } }

No need to quote the newline, just put a literal newline in the char class.

13

sorry i didn't get it unfortunately i have to post a picture again so i can understand you.
this is what i'm getting

Selection_011
Selection_011.png1092×508 31.4 KB

14

I cannot speak to the grok debugger, only to the grok filter.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.