Process logs of different formats to JSON

Elastic Stack Logstash
1

I'm pretty new to ELK and I'm trying to push few of our service's logs to ES.

Log funneling flow is --> `

Fluentd --> Logstash --> ES --> Kibana.

`

A thing to note is that, each service has its own log format. Attached four different service's logs for reference.

How can I handle all four format in a generic way ? I'm looking to structure it or convert all the logs into a JSON format while being displayed on Kibana.

Current configuration.

input {
  http {
    port => 9100
  }
}

filter {
    json {
       source => "message"
       remove_field => ["message"]
    }
    if [kubernetes][labels][id] in ["service1", "service2", "service3", "service4"] {
      json {
        source => "log"
        remove_field => ["log"]
      }
    }
    mutate {
      remove_field => [ "_id", "headers", "_type", "character", "@version", "_index", "_score" ]
    }
}
output {
     elasticsearch {
       id => "ieo-elk-drp-log-aggregation"
       hosts => ["<%= @ipaddress%>:19200"]
       index => "drp-%{[kubernetes][labels][app]}-%{+YYYY.MM.dd}"
    }
}

On the logstash logs, continuously see the below errors and it indicates the logs are of not of proper JSON. Any help here is really appreciated, thank you.

[2023-07-20T00:06:41,900][WARN ][logstash.filters.json    ] Error parsing json {:source=>"log", :raw=>"INFO |2023-07-20 00:05:09 |1 | manageQueue.py 324 | There is nothing to process on the queue as it is empty. Returning...", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'INFO': was expecting 'null', 'true', 'false' or NaN

[2023-07-20T00:09:30,425][WARN ][logstash.filters.json    ] Error parsing json {:source=>"log", :raw=>"Thu Jul 20 00:09:07 2023 sent heartbeat", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Thu': was expecting ('true', 'false' or 'null')

[2023-07-20T00:11:55,679][WARN ][logstash.filters.json    ] Error parsing json {:source=>"log", :raw=>"No Jira servers defined at /usr/src/app/value.pm line 70.", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'No': was expecting 'null', 'true', 'false' or NaN
 at [Source: (byte[])"No Jira servers defined at /usr/src/app/value.pm line 70."; line: 1, column: 4]>}
 
 [2023-07-20T02:59:48,393][WARN ][logstash.filters.json    ] Error parsing json {:source=>"log", :raw=>"Thu Jul 20 02:59:23 2023 sent heartbeat", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Thu': was expecting ('true', 'false' or 'null')
 at [Source: (byte[])"Thu Jul 20 02:59:23 2023 sent heartbeat"; line: 1, column: 5]>}
[2023-07-20T02:59:56,626][WARN ][logstash.filters.json    ] Error parsing json {:source=>"log", :raw=>"malformed JSON string, neither array, object, number, string or atom, at character offset 11 (before \"<html>\\n\\n<head>\\n  ...\") at (eval 250) line 1.", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'malformed': was expecting ('true', 'false' or 'null')

[2023-07-20T06:53:05,389][WARN ][logstash.filters.json    ] Error parsing json {:source=>"log", :raw=>"2023/07/20 06:50:48| NETDB state saved; 1 entries, 0 msec", :exception=>#<LogStash::Json::ParserError: Unexpected character ('/' (code 47)): Expected space separating root-level values

Also, how can I attach text files to the question ?
Time being, I'm sharing the text location :slight_smile:

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.