How to parse logstash's data to a json array?

Elastic Stack Logstash
1

How to parse logstash's data to a json array?

Using filebeat sending a file to logstash.
The file includes 2 line of records:

{"request_id": "m2ee22d045f6c5ce07fe43dbdaea1de0","method": "GET","status": "304","forwarded_for": "","host": "localhost","url": "/","referer": "","remote_ip": "10.0.2.2","server_ip": "10.0.2.15","user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36","latency": "0s","occurred_at": "2020-12-10T03:39:29+00:00"}
{"request_id": "m2ee22d045f6c5ce07fe43dbdaea1de0","method": "GET","status": "304","forwarded_for": "","host": "localhost","url": "/","referer": "","remote_ip": "10.0.2.2","server_ip": "10.0.2.15","user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36","latency": "0s","occurred_at": "2020-12-10T03:39:29+00:00"}

filebeat.yml

    filebeat.inputs:
    - type: log
      paths:
        - /var/log/nginx.log

    processors:
      - decode_json_fields:
          fields: ["message"]
          process_array: true
          max_depth: 1
          target: ""
          overwrite_keys: true
          add_error_key: false

    output.logstash:
      hosts: ["logstash:5044"]

logstash.conf

    input {
      beats {
        port => 5044
      }
    }

    filter {
      mutate {
        remove_field => [
          "agent",
          "event",
          "message"
          ...
        ]
      }
    }

    output {
      stdout { codec => rubydebug }
    }

Logstash got the data as

{
          "latency" => "0s",
              "url" => "/",
      "occurred_at" => "2020-12-10T08:08:10+00:00",
        "server_ip" => "10.0.2.15",
             "host" => "localhost",
       "request_id" => "ede6f0f02934cd18bcab1f9f531586e2",
        "remote_ip" => "10.0.2.2",
    "forwarded_for" => "",
       "user_agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
           "status" => "304",
          "referer" => "",
           "method" => "GET"
}
{
          "latency" => "0s",
              "url" => "/",
      "occurred_at" => "2020-12-10T08:08:10+00:00",
        "server_ip" => "10.0.2.15",
             "host" => "localhost",
       "request_id" => "ede6f0f02934cd18bcab1f9f531586e2",
        "remote_ip" => "10.0.2.2",
    "forwarded_for" => "",
       "user_agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
           "status" => "304",
          "referer" => "",
           "method" => "GET"
}

Want to get data as

    [
        {
              "latency" => "0s",
                  "url" => "/",
          "occurred_at" => "2020-12-10T08:08:10+00:00",
            "server_ip" => "10.0.2.15",
                 "host" => "localhost",
           "request_id" => "ede6f0f02934cd18bcab1f9f531586e2",
            "remote_ip" => "10.0.2.2",
        "forwarded_for" => "",
           "user_agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
               "status" => "304",
              "referer" => "",
               "method" => "GET"
        },
        {
              "latency" => "0s",
                  "url" => "/",
          "occurred_at" => "2020-12-10T08:08:10+00:00",
            "server_ip" => "10.0.2.15",
                 "host" => "localhost",
           "request_id" => "ede6f0f02934cd18bcab1f9f531586e2",
            "remote_ip" => "10.0.2.2",
        "forwarded_for" => "",
           "user_agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
               "status" => "304",
              "referer" => "",
               "method" => "GET"
        }
    ]

I researched both filebeat and logstash's documents. But don't know if make it in filebeat or logstash's filter.

All the necessary data are in message only. Just want to rewrite this context.

2

If you want to combine lines from the same file you could do that using a multiline configuration in filebeat. Alternatively, use an aggregate filter in logstash with the file path as the task id. Something similar to example 3 in the documentation.

3

Thank you.

I tried add this to filebeat.yml

filebeat.inputs:
      - input_type: log
        paths:
          - /var/log/nginx.log

        multiline.pattern: '^\{'
        multiline.negate: true
        multiline.match: after

It doesn't work.

In logstash.conf, I added this to filter

      aggregate {
        task_id => "/var/log/nginx.log"
      }

Can't run logstash successfully.

Do you mean this example? Example #3 : no end event

Could you give me an example for it?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.