Logstash - Creating new field by taking first word from an other field

Carlos_T · May 28, 2023, 8:32pm

Hi all.

It must be something plenty of people has answered but I can´t find it

I've got a pipeline reading a log with the following structure:

[12/May/2022:19:04:50 +0200] 192.168.0.2 server2 "DROP: Este es un mensaje de error"
[12/May/2022:01:04:50 +0200] 192.168.0.1 server3 "WARNING: Este es un mensaje de aviso"
[12/May/2022:01:04:50 +0200] 192.168.0.2 server4 "DROP: Este es un mensaje de error"
[12/May/2022:01:04:50 +0200] 192.168.1.2 server4 "CRITICAL: Este es un mensaje de error"

And I'm using the following pipeline to get some field of them:

# El propósito de esta pipiline es leer en tradas de log como estas:
#[27/May/2022:19:04:50 +0200] 192.168.0.1 server1 "WARNING: Este es un mensaje de aviso"
#[27/May/2022:19:04:50 +0200] 192.168.0.2 server2 "CRITICAL: Este es un mensaje de error"
#[28/May/2022:01:04:50 +0200] 192.168.0.1 server3 "WARNING: Este es un mensaje de aviso"
#[28/May/2022:01:04:50 +0200] 192.168.0.2 server4 "CRITICAL: Este es un mensaje de error"


# Una vez leidas deberar extraer de MENSAJE la palabra WARNING o CRITICAL y meterla en un campo nuevo llamado SEVERIDAD.

input {
  file {
    path => "/var/log/genericlogs/genericlog02.log"
    start_position => "beginning"
  }

}

filter {
  if "WARNING" in [message] or "CRITICAL" in [message] {
    grok {
      match => {
        "message" => "\[%{HTTPDATE:FECHA}\] %{IP:DIR_IP} %{WORD:SERVIDOR} \"%{GREEDYDATA:MENSAJE}\""
      }
    }
    mutate {
      add_field => { "severity" => "%{MENSAJE}" }
    }
  } else {
    drop {}
  }
}


output {
  elasticsearch {
    hosts => ["https://192.168.0.111:9200","https://192.168.0.112:9200"]
    ssl_certificate_verification => false
    user => "elastic"
    password => "XXXXXXXXXXX"
    index => "generic_log02"
  }
}

So far everything is working good but in addition to the created fields. This is an example of the data I can see in Kibana:

{
  "@timestamp": [
    "2023-05-28T20:29:02.124Z"
  ],
  "@version": [
    "1"
  ],
  "@version.keyword": [
    "1"
  ],
  "DIR_IP": [
    "192.168.1.2"
  ],
  "DIR_IP.keyword": [
    "192.168.1.2"
  ],
  "event.original": [
    "[11/May/2022:01:04:50 +0200] 192.168.1.2 server4 \"CRITICAL: Este es un mensaje de error\""
  ],
  "event.original.keyword": [
    "[11/May/2022:01:04:50 +0200] 192.168.1.2 server4 \"CRITICAL: Este es un mensaje de error\""
  ],
  "FECHA": [
    "11/May/2022:01:04:50 +0200"
  ],
  "FECHA.keyword": [
    "11/May/2022:01:04:50 +0200"
  ],
  "host.name": [
    "ubuntuelk02"
  ],
  "host.name.keyword": [
    "ubuntuelk02"
  ],
  "log.file.path": [
    "/var/log/genericlogs/genericlog02.log"
  ],
  "log.file.path.keyword": [
    "/var/log/genericlogs/genericlog02.log"
  ],
  "MENSAJE": [
    "CRITICAL: Este es un mensaje de error"
  ],
  "MENSAJE.keyword": [
    "CRITICAL: Este es un mensaje de error"
  ],
  "message": [
    "[11/May/2022:01:04:50 +0200] 192.168.1.2 server4 \"CRITICAL: Este es un mensaje de error\""
  ],
  "message.keyword": [
    "[11/May/2022:01:04:50 +0200] 192.168.1.2 server4 \"CRITICAL: Este es un mensaje de error\""
  ],
  "SERVIDOR": [
    "server4"
  ],
  "SERVIDOR.keyword": [
    "server4"
  ],
  "severity": [
    "CRITICAL: Este es un mensaje de error"
  ],
  "severity.keyword": [
    "CRITICAL: Este es un mensaje de error"
  ],
  "_id": "4BIMZIgBKDCEOXT41Dhg",
  "_index": "generic_log02",
  "_score": null
}

What I intend to do is to create a field called 'severity' that will contain only the first word of the 'MENSAJE' field. For example 'CRITICAL' or 'WARNING' as the rest of the log's lines are dropped.

How can I do this?

Thank you very much in advance.

Carlos T

Rios · May 29, 2023, 6:05am

I think you need this:
\[%{HTTPDATE:FECHA}\] %{IP:DIR_IP} %{WORD:SERVIDOR} \"%{DATA:SEVERITY}:\s*%{GREEDYDATA}\"

or SEVERITY+MENSAJE
\[%{HTTPDATE:FECHA}\] %{IP:DIR_IP} %{WORD:SERVIDOR} \"%{DATA:SEVERITY}:\s*%{GREEDYDATA:MENSAJE}\"

Carlos_T · May 29, 2023, 8:53am

Hi Rios.

Thank you very much for your fast answer.

The second pattern you provided me is the one that works.

Anyway, this is not exactly what I was looking for, because now the string that goes to the SEVERITY FIELD doesn't appair in the MENSAJE field anymore.

With your solution this is what I got.

{
  "@timestamp": [
    "2023-05-29T07:38:53.033Z"
  ],
  "@version": [
    "1"
  ],
  "@version.keyword": [
    "1"
  ],
  "DIR_IP": [
    "192.168.0.1"
  ],
  "DIR_IP.keyword": [
    "192.168.0.1"
  ],
  "event.original": [
    "[11/May/2022:02:04:51 +0200] 192.168.0.1 server3 \"WARNING: Este es un mensaje de aviso\""
  ],
  "event.original.keyword": [
    "[11/May/2022:02:04:51 +0200] 192.168.0.1 server3 \"WARNING: Este es un mensaje de aviso\""
  ],
  "FECHA": [
    "11/May/2022:02:04:51 +0200"
  ],
  "FECHA.keyword": [
    "11/May/2022:02:04:51 +0200"
  ],
  "host.name": [
    "ubuntuelk02"
  ],
  "host.name.keyword": [
    "ubuntuelk02"
  ],
  "log.file.path": [
    "/var/log/genericlogs/genericlog02.log"
  ],
  "log.file.path.keyword": [
    "/var/log/genericlogs/genericlog02.log"
  ],
  "MENSAJE": [
    "Este es un mensaje de aviso"
  ],
  "MENSAJE.keyword": [
    "Este es un mensaje de aviso"
  ],
  "message": [
    "[11/May/2022:02:04:51 +0200] 192.168.0.1 server3 \"WARNING: Este es un mensaje de aviso\""
  ],
  "message.keyword": [
    "[11/May/2022:02:04:51 +0200] 192.168.0.1 server3 \"WARNING: Este es un mensaje de aviso\""
  ],
  "SERVIDOR": [
    "server3"
  ],
  "SERVIDOR.keyword": [
    "server3"
  ],
  "severity": [
    "Este es un mensaje de aviso"
  ],
  "SEVERITY": [
    "WARNING"
  ],
  "severity.keyword": [
    "Este es un mensaje de aviso"
  ],
  "SEVERITY.keyword": [
    "WARNING"
  ],
  "_id": "88pyZogBSFTYYEl2GFhG",
  "_index": "generic_log02",
  "_score": null
}

And this what I would like to get:

{
  "@timestamp": [
    "2023-05-29T07:38:53.033Z"
  ],
  "@version": [
    "1"
  ],
  "@version.keyword": [
    "1"
  ],
  "DIR_IP": [
    "192.168.0.1"
  ],
  "DIR_IP.keyword": [
    "192.168.0.1"
  ],
  "event.original": [
    "[11/May/2022:02:04:51 +0200] 192.168.0.1 server3 \"WARNING: Este es un mensaje de aviso\""
  ],
  "event.original.keyword": [
    "[11/May/2022:02:04:51 +0200] 192.168.0.1 server3 \"WARNING: Este es un mensaje de aviso\""
  ],
  "FECHA": [
    "11/May/2022:02:04:51 +0200"
  ],
  "FECHA.keyword": [
    "11/May/2022:02:04:51 +0200"
  ],
  "host.name": [
    "ubuntuelk02"
  ],
  "host.name.keyword": [
    "ubuntuelk02"
  ],
  "log.file.path": [
    "/var/log/genericlogs/genericlog02.log"
  ],
  "log.file.path.keyword": [
    "/var/log/genericlogs/genericlog02.log"
  ],
  "MENSAJE": [
    "WARNING: Este es un mensaje de aviso"
  ],
  "MENSAJE.keyword": [
    "WARNING: Este es un mensaje de aviso"
  ],
  "message": [
    "[11/May/2022:02:04:51 +0200] 192.168.0.1 server3 \"WARNING: Este es un mensaje de aviso\""
  ],
  "message.keyword": [
    "[11/May/2022:02:04:51 +0200] 192.168.0.1 server3 \"WARNING: Este es un mensaje de aviso\""
  ],
  "SERVIDOR": [
    "server3"
  ],
  "SERVIDOR.keyword": [
    "server3"
  ],
  "severity": [
    "Este es un mensaje de aviso"
  ],
  "SEVERITY": [
    "WARNING"
  ],
  "severity.keyword": [
    "Este es un mensaje de aviso"
  ],
  "SEVERITY.keyword": [
    "WARNING"
  ],
  "_id": "88pyZogBSFTYYEl2GFhG",
  "_index": "generic_log02",
  "_score": null
}

That's why I'm asking how to get the first word of the MENSAJE field without getting rid of it.

Thanks a lot

Rios · May 29, 2023, 9:25am

Add one more grok.

filter {
  if "WARNING" in [message] or "CRITICAL" in [message] {
    grok {
      match => {
        "message" => "\[%{HTTPDATE:FECHA}\] %{IP:DIR_IP} %{WORD:SERVIDOR} \"%{GREEDYDATA:MENSAJE}\""
      }
    }
   grok {
        match => {
            "MENSAJE" => [
                "^%{LOGLEVEL:SEVERITY}\:\s*"
            ]
        }
    }

  } else {
    drop {}
  }
}

Result:

{
      "message" => "[27/May/2022:19:04:50 +0200] 192.168.0.1 server1 \"WARNING: Este es un mensaje de aviso\"",
      "SERVIDOR" => "server1",
       "MENSAJE" => "WARNING: Este es un mensaje de aviso",
      "SEVERITY" => "WARNING",
         "FECHA" => "27/May/2022:19:04:50 +0200",
        "DIR_IP" => "192.168.0.1"
}

Carlos_T · May 29, 2023, 4:38pm

It works Sir.

Thank you very much

system · June 26, 2023, 4:38pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Add new field Logstash	1	1045	July 12, 2017
Logstash extract first 3 chars of a field into another field Logstash	3	665	July 6, 2017
Create a field with information from another field Logstash	2	258	May 26, 2020
Logstash filtering Logstash	1	156	August 29, 2023
On ingest I need to create a new field based on the value of a field in the document Logstash	5	428	July 6, 2017

Logstash - Creating new field by taking first word from an other field

Related topics