Logstash output not the same as grok debugger

[markov (galay)]

hello guys,
my grok debugger pattern output doesn't look like the logstash output in kibana!! any idea !

here is in kibana :

[markov (galay)]

any help!

[Badger]

Please do not post pictures of text, just post the text itself. What does your pattern look like and what do the lines of text you are processing look like?

[markov (galay)]

2019-03-11 11:12:40,670 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-28144: Echec de l'exécution du gestionnaire d'audit détaillé
ORA-20417: ERROR SECURITY DATA
ORA-06512: à "PRODUCTION_AUDIT", ligne 39
ORA-06512: à ligne 1

and this is the pattern used that correctly match in grok debugger:

%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<classname>[^\]]+)\] %{GREEDYDATA:Error}(?<msg>[^\)]+)

[markov (galay)]

and also it regroup all other log events in one record!!

{
  "_index": "filebeat-2019.04.08",
  "_type": "doc",
  "_id": "HQJe_GkBGmfQmneQJD3j",
  "_version": 1,
  "_score": null,
  "_source": {
    "number": "0",
    "n": "null",
    "prospector": {},
    "classname": "impl.SensusAnalyzeLogger",
    "Url": "business.service.interfaces.ICroneJobsService",
    "beat": {},
    "@version": "1",
    "Loglevel": "INFO",
    "timestamp": "2019-03-11 00:00:00,013",
    "msg": "getMapAllCronedJobsParams | null | >> entry >>  | null\n2019-03-11 00:00:00,014 INFO  [impl.SensusAnalyzeLogger] null | 0 | null | persistence.manager.interfaces.ICroneEntityManager | getMapJobsCronPramas | null | >> entry >>  | null\n2019-03-11 00:00:00,016 INFO  [com.sensus.persistence.commons.query.impl.SensusHibernateQueryFacade] @PARAMETER_BY_ID@|query time :2 ms |result size : 6|{paramIds=[7, 8, 6, 11, 9, 10]}\n2019-03-11 00:00:00,016 INFO  [log.impl.SensusAnalyzeLogger] null | 0 | null | persistence.manager.interfaces.ICroneEntityManager | getMapJobsCronPramas | null | << exit  <<  | 2 ms\n2019-03-11 00:00:00,016 INFO  [impl.SensusAnalyzeLogger] null | 0 | null | service.interfaces.ICroneJobsService | getMapAllCronedJobsParams | null | << exit  <<  | 3 ms\n2019-03-11 00:01:00,014 INFO  [impl.SensusAnalyzeLogger] null | 0 | null | service.interfaces.ICroneJobsService | getMapAllCronedJobsParams | null | >> entry >>  | null\n

[Badger]

@markov

%{GREEDYDATA:Error}(?<msg>[^)]+)

OK, so your pattern says to collect everything into Error, followed by one of more characters that are not close-parenthesis grouped into msg. And that's what you get -- one character in msg.

If you want to store everything up to the first newline in Error and all the rest in msg, then tell grok that.

    grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<classname>[^\]]+)\] (?<Error>[^
]+)%{GREEDYDATA:msg}" } }

[markov (galay)]

2019-03-11 09:11:42,233 ERROR [impl.CommonModificationBudgetaireBusinessServiceImpl] DISPONIBLE_MC_NON com.sensus.common.exception.SensusFunctionalException: DISPONIBLE_MC_NON at verifierDisponibleGlobaleMC(CommonModificationBudgetaireBusinessServiceImpl.java:8474)

my pattern is working with java log too so i have to stop on ")" character for that i put (?<msg>[^)]+) at the end, and i guess %{GREEDYDATA:msg} can't do this also i tried the pattern you suggest in grok debugger and it shows Compile ERROR

[markov (galay)]

what do you see

[Badger]

With the grok filter I showed I get

       "msg" => "\nORA-20417: ERROR SECURITY DATA\nORA-06512: à \"PRODUCTION_AUDIT\", ligne 39\nORA-06512: à ligne 1",
     "Error" => "ORA-28144: Echec de l\\'exécution du gestionnaire d\\'audit détaillé",

[markov (galay)]

@Badger

%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} [(?<classname>[^]]+)] (?<Error>[^ ]+)%{GREEDYDATA:msg}

and i get this !!!

[parse_exception] [patterns] Invalid regex pattern found in: [%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<classname>[^\]]+)\] (?<Error>[^]+)%{GREEDYDATA:msg}]. empty char-class, with { header={ processor_type="grok" & property_name="patterns" } }

[Badger]

@markov

empty char-class

That would suggest you are trying to use

 grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<classname>[^\]]+)\] (?<Error>[^]+)%{GREEDYDATA:msg}" } }

rather than

        grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:Loglevel} \[(?<classname>[^\]]+)\] (?<Error>[^
]+)%{GREEDYDATA:msg}" } }

No need to quote the newline, just put a literal newline in the char class.

[markov (galay)]

sorry i didn't get it unfortunately i have to post a picture again so i can understand you.
this is what i'm getting

[Badger]

I cannot speak to the grok debugger, only to the grok filter.