Group apache org.apache.http request into a single object

Elastic Stack Logstash
1

hello all :slight_smile:
i have a log file the look like this

[DEBUG][DefaultHttpClient][2018-02-20 18:29:50,348]  Attempt 1 to execute request
[DEBUG][DefaultClientConnection][2018-02-20 18:29:50,348]  Sending request: POST /somepath HTTP/1.1
[DEBUG][wire][2018-02-20 18:29:50,349]   >> "POST /somepath HTTP/1.1[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350]   >> "Accept: */*[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350]   >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350]   >> "Content-Length: 100[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350]   >> "Host: some domain.com[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350]   >> "Connection: Keep-Alive[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350]   >> "Accept-Encoding: gzip,deflate[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350]   >> "[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,351]   >> "{"some_key":"some_data_value"}"
[DEBUG][wire][2018-02-20 18:29:51,709]   << "HTTP/1.1 200 OK[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,713]   << "Cache-Control: private, must-revalidate[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,713]   << "Cache-control: no-cache="set-cookie"[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,713]   << "Content-Type: application/json[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,713]   << "Date: Tue, 20 Feb 2018 16:29:51 GMT[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,713]   << "expires: 0[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,714]   << "pragma: no-cache[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,714]   << "Server: someserver/1.13.8[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,714]   << "Set-Cookie: thecookie=123adsdf.......[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,714]   << "Connection: keep-alive[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,714]   << "[\r][\n]"
[DEBUG][DefaultClientConnection][2018-02-20 18:29:51,715]  Receiving response: HTTP/1.1 200 OK
[DEBUG][DefaultHttpClient][2018-02-20 18:29:51,721]  Connection can be kept alive indefinitely
[DEBUG][RequestSpecificationImpl$RestAssuredHttpBuilder][2018-02-20 18:29:51,772]  Parsing response

is it possible to log a single object for each request.
i was thinking something like this (or something similar just to separate the request and response headers) :

{
"message" :"...the original message",
"request_headers" : [{"key":"Accept", "value":"/somepath HTTP/1.1"}, {"key":"Content-Type:", "value":"application/json; charset=UTF-8"}, ..... and so on],
"response_headers": [{"key":"HTTP/1.1", "value":"200"}, {"key":"Cache-Control:", "value":"private, must-revalidate"}, ..... and so on],
"response_body": "i am the response body"
}

thank you

2

You'll have to use a multiline codec (or the equivalent on the Filebeat side if you're using Filebeat to ship the file to Logstash). The logic would be: If the message part of the line is "<<" or ">>", join with the previous line.

3

FILTER :

filter {
  grok {
    match => {
      "message" => "\[%{LOGLEVEL:severity}.*\]\[wire\]\[%{TIMESTAMP_ISO8601:timestamp}\]   ((\>\> \"%{DATA:request_key}(\s|:\s)%{GREEDYDATA:request_value}\")|(\<\< \"%{DATA:response_key}(\s|:\s)%{GREEDYDATA:response_value}\"))"
    }
    overwrite => [ "key", "value" ]
  }

  multiline {
    pattern => "(\>\>|\<\<)"
    what => "previous"
    negate => false
  }
}

RESULT :

{
  "tags": [
    "multiline",
    "_grokparsefailure"
  ],
  "@version": "1",
  "response_key": [
    "HTTP/1.1",
    "Cache-Control",
    "Cache-control",
    "Content-Type",
    "Date",
    "expires",
    "pragma",
    "Server",
    "Set-Cookie",
    "Strict-Transport-Security",
    "X-Frame-Options",
    "X-XSS-Protection",
    "Content-Length",
    "Connection"
  ],
  "timestamp": [
    "2018-02-21 10:44:54,754",
    "2018-02-21 10:44:54,755",
    "2018-02-21 10:44:55,077",
    "2018-02-21 10:44:55,079"
  ],
  "host": "somehost.com",
  "severity": "DEBUG",
  "request_key": [
    "POST",
    "Accept",
    "Content-Type",
    "Content-Length",
    "Host",
    "Connection",
    "User-Agent",
    "Accept-Encoding",
    "{\"somedata_key\":\"some_data_value\"}"
  ],
  "response_value": [
    "200 OK[\\r][\\n]",
    "private, must-revalidate[\\r][\\n]",
    "no-cache=\"set-cookie\"[\\r][\\n]",
    "application/json[\\r][\\n]",
    "Wed, 21 Feb 2018 08:44:55 GMT[\\r][\\n]",
    "0[\\r][\\n]",
    "no-cache[\\r][\\n]",
    "someserver/1.13.8[\\r][\\n]",
    "somekey=somevalues[\\r][\\n]",
    "SAMEORIGIN[\\r][\\n]",
    "1; mode=block[\\r][\\n]",
    "1439[\\r][\\n]",
    "keep-alive[\\r][\\n]"
  ],
  "@timestamp": "2018-02-21T08:46:13.003Z",
  "message": "THIS IS A VERY LONG ORIGINAL LOG MESSAGE",
  "path": "the path to the log.log",
  "request_value": [
    "/access/login HTTP/1.1[\\r][\\n]",
    "*/*[\\r][\\n]",
    "application/json; charset=UTF-8[\\r][\\n]",
    "253[\\r][\\n]",
    "some.domain.com[\\r][\\n]",
    "Keep-Alive[\\r][\\n]",
    "Apache-HttpClient/4.5.3 (Java/1.8.0_45)[\\r][\\n]",
    "gzip,deflate[\\r][\\n]",
    "(Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.86 Safari/537.36\"}"
  ]
}

WANTED:
i wish to have

{
	.....
	"request": [ {"key": "as dad", "value" : "asdasd"}, {"key": "as 123", "value" : "123"}],
	"response": [ {"key": "as dad", "value" : "asdasd"}, {"key": "as 123", "value" : "123"}],
	.....
}
4

Use the multiline codec, not the filter.

You can use a ruby filter to join an array of keys with an array of values and produce a hash.

5

thank you.
where can u find an example to something similar

6

I think it's been discussed here before. As for how to join (zip) arrays in Ruby see e.g. https://stackoverflow.com/questions/23113759/how-to-zip-two-arrays-into-hash.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.