Logstash extract additional fields from message field of Json

[Derya_Sayar]

I have logs files with json format.Here you can find single line of log.
{"instant":{"epochSecond":1628692763,"nanoOfSecond":792000000},"thread":"AWT-EventQueue-0","level":"INFO","loggerName":"com.client.logon.form.Logon","message":"errortype: SECURITY logContent:User Log on user 1","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":23,"threadPriority":6,"@timestamp":"2021-08-11T17:39:01.025+0300"}

I need to show fields like thread, level, logger... in elastic server. It work with json filter.I also need to also extract fields "errorType" and "logContent" by manipulating message field ""errortype: SECURITY logContent:User Log on user 1"

Here I use following configuration but it does not work. Each log line does not need to have errorType field. It will be included if level is ERROR. For other levels, I can have logContent inside "message" field. Could you please help me ?

filter{

json{
    source => "message"
}
grok {
    match => {
        "message" => [
            "errortype:%{GREEDYDATA:errorType} logContent: %{GREEDYDATA:logContent}"
        ]
    }
}

elastic and logstash versions are 7.3.14

[Cad]

Hi,

The grok pattern don't respect your values and it don't respect your description :

@Derya_Sayar

Each log line does not need to have errorType field. It will be included if level is ERROR. For other levels, I can have logContent inside "message" field

From what i understand your logs contains logContent every time and if the log level is ERROR the message also contains errorType.

In your example, you show us one INFO loglevel json with one errorType, that don't respect your description.

Plus, in the grok pattern, you have a space between logContent: and %{GREEDYDATA that is not the case of your logs.

I think, this configuration should be better if the errorType is not present every time

grok {
    match => {
        "message" => [
            "errortype:%{GREEDYDATA:errorType} logContent:%{GREEDYDATA:logContent}",
            "logContent:%{GREEDYDATA:logContent}"
        ]
    }
}

Cad.

[Derya_Sayar]

Thanks for your reply. It seems it is working