Is posible to have iptional field with regex as grok optionals fields?

[rodri.gz (RGZ-ELK)]

Hello!

i want to know if a could set a field as optional in regular expresions as i do with grok

for example:

(%{DATA:Message})?

Thank you!

[Badger]

Yes, you can use exactly that syntax to match zero or more %{DATA:Message} patterns. Note that DATA can match zero characters, so sometimes it will not consume anything.

[AquaX (Andreas Helmer)]

Absolutely!
This pattern:

%{WORD}? %{NUMBER} %{WORD}?

Will match:

ABC 123 XZY

{
  "WORD": [
    [
      "ABC",
      "XZY"
    ]
  ],
  "NUMBER": [
    [
      "123"
    ]
  ],
  "BASE10NUM": [
    [
      "123"
    ]
  ]
}

or it will match

ABC 123 

{
  "WORD": [
    [
      "ABC",
      null
    ]
  ],
  "NUMBER": [
    [
      "123"
    ]
  ],
  "BASE10NUM": [
    [
      "123"
    ]
  ]
}

[rodri.gz (RGZ-ELK)]

can i do it with texts ?

my log follows this format:

2021-12-07 09:59:33,940 INFO [xxxxxxxxxxxxxxxxxxxxxxxx][YYYYYYYYYYYYYYYYYY] Parámetros enviados a eeeee:
	action: efdwfwf_ewfwef_ewf
	action2 : wefwefwe:_FWwefwe
	type : feAASDF
	log : logfile

and this parameter with the format name : value could not appear.

so im trying something like:

(action: %{USERNAME:Action})?%{DATA}(action2 : %{USERNAME:Action2})?%{DATA}(type: %{USERNAME:Type})?

how can i do that?

[AquaX (Andreas Helmer)]

This looks like a multiline log format so be sure you configure your input to also handle multiline messages, however afterwards I got this pattern to work with the grokdebugger

%{SPACE}(action: %{USERNAME:Action})?%{SPACE}(action2 : %{NOTSPACE:Action2})?%{SPACE}(type : %{USERNAME:Type})?%{SPACE}(log : %{NOTSPACE:log})?

{
  "SPACE": [
    [
      "\t",
      "\n\t",
      "\n\t",
      "\n\t"
    ]
  ],
  "Action": [
    [
      "efdwfwf_ewfwef_ewf"
    ]
  ],
  "Action2": [
    [
      "wefwefwe:_FWwefwe"
    ]
  ],
  "Type": [
    [
      "feAASDF"
    ]
  ],
  "log": [
    [
      "logfile"
    ]
  ]
}

[rodri.gz (RGZ-ELK)]

that's what i assumed i would have to do but when i make it optional it disappears.

Real example:

2021-12-09 08:50:03,938 INFO [WebContainer : x] I.xxxxxDatosPersonales [AfphLog.java:209] [09/12/2021 08:50:03][1YYYYYYYYY.XXXXXXDatosPersonales] Parámetros enviados a ZZZZZ:
    accion : ZZZZ
    accionProsa : null
    tipoServicio : YYYYYY
    log : Importass.XXXXXXXX

this grok works

(?m)%{TIMESTAMP_ISO8601:Fecha} %{LOGLEVEL:LogLevel} \[WebContainer : %{NUMBER:WebContainer}] %{DATA:Entidad} \[AfphLog.java:%{NUMBER:AfphLogJava}]%{GREEDYDATA}accion : %{USERNAME:Accion}

result:

{
  "Fecha": "2021-12-09 08:50:03,938",
  "WebContainer": "x",
  "AfphLogJava": "209",
  "Accion": "ZZZZ",
  "LogLevel": "INFO",
  "Entidad": "1YYYYYYYYY.XXXXXXDatosPersonales"
}

but if i put the grok filter like this:

(?m)%{TIMESTAMP_ISO8601:Fecha} %{LOGLEVEL:LogLevel} \[WebContainer : %{NUMBER:WebContainer}] %{DATA:Entidad} \[AfphLog.java:%{NUMBER:AfphLogJava}]%{GREEDYDATA}(accion : %{USERNAME:Accion})?

the accion field dissapears:

{
  "Fecha": "2021-12-09 08:50:03,938",
  "WebContainer": "7",
  "AfphLogJava": "209",
  "LogLevel": "INFO",
  "Entidad": "I.AFPH03DatosPersonales"
}

why the optional field dissapears?