Having trouble parsing my JSON apache log using Logstash

Elastic Stack Logstash
1

Hello. I have an existing Elastic stack that is pulling in app and web server logs. For the web site, I have 2 software stacks - 1 is using an older apache format (comma separated values) and 1 is using a newer JSON format. With the JSON format, I'm trying to leverage my existing YAML filter and modifying it to try and pick up the JSON format. So far, nothing I've tried is working. I can tell the log is getting picked up by FileBeat and it is making it to the Logstash server.

Here is an example of an Apache log entry:

{"type":"ihs-access","datetime":"2024-02-15T23:59:59.999 EST","@version":"1","protocol":"HTTP/1.1","remoteHost":"10.10.10.10","port":443,"method":"GET","url":"/this/is/a/path/product/details/abcd","query":"","status":200,"bytes":1826,"elapsedTime":3344,"referer":"-","libertyServer":"serverhost:5443","tealeafId":"-","sessionId":"-","requestId":"-","jsessionId":"-","userAgent":"Apache-HttpClient/4.5.2 (Java/1.7.0)"}

Here is a snippet of my logstash pipeline:

input {
    beats {
        port => "5044"
    }
}
filter {
    if [fields][log_type] == "weblogs" {
       mutate { replace => { type => "apache_access" } }
    }
    else if [fields][log_type] == "otherweblogs" {
    // working logs
    }
output {
    if [type] == "apache_access" {
        elasticsearch {
        // set all values
       }

For the above, I'm passing "[fields][log_type]" from FileBeat and this pipeline is properly detecting it and then setting the "mutate --> replace" properly because the output is section is picking it up and generating an index.

However, the entire Apache log is landing in the "message" field and nothing is getting parsed out. So, I tried this for the filter:

filter {
    if [fields][log_type] == "weblogs" {
       mutate { replace => { type => "apache_access" } }
       json {
          source => "message"
          tag_on_failure => [ "_jsonparsefailure" ]
       }
    else if [fields][log_type] == "otherweblogs" {
    // working logs
    }
}

However, as soon as I do this, no index gets created any longer. Could it be that the "type" field is getting lost once the JSON is parsed? Seems that the "output" section is no longer happening. Or maybe the JSON part is wrong because there is no "message" field on the inbound?

If anyone has any thoughts on how to properly parse my input and set the "type" value properly so my "output" conditional statement will pick up on it, I would appreciate it. Thanks!

2

The json filter will use that to set the [type] field, so your conditional in the output section will evaluate false.

3

Thanks. But, if the json filter is altering that field, how do I fix it? I suspected that the conditional was getting skipped since I'm not getting an index created. I'm just not clear how to both parse the JSON to get the fields I want in the index, but also have the "type" set to "apache_access" so my condition executes.

4

Move the mutate+replace to be after the json filter?

5

That did the trick. Thanks!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.