# Group apache org.apache.http request into a single object

**URL:** <https://discuss.elastic.co/t/group-apache-org-apache-http-request-into-a-single-object/120684>\
**Category:** Logstash\
**Created:** [February 20, 2018, 4:37pm UTC](https://discuss.elastic.co/t/group-apache-org-apache-http-request-into-a-single-object/120684 "2018-02-20T16:37:57Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![shacoshe](https://avatars.discourse-cdn.com/v4/letter/s/e19b73/32.png) [@shacoshe](https://discuss.elastic.co/u/shacoshe)\
**Post date:** [February 20, 2018, 4:37pm UTC](https://discuss.elastic.co/t/group-apache-org-apache-http-request-into-a-single-object/120684/1 "2018-02-20T16:37:57Z")

</div>

hello all 🙂  
i have a log file the look like this

```
[DEBUG][DefaultHttpClient][2018-02-20 18:29:50,348] Attempt 1 to execute request
[DEBUG][DefaultClientConnection][2018-02-20 18:29:50,348] Sending request: POST /somepath HTTP/1.1
[DEBUG][wire][2018-02-20 18:29:50,349] >> "POST /somepath HTTP/1.1[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350] >> "Accept: */*[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350] >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350] >> "Content-Length: 100[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350] >> "Host: some domain.com[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350] >> "Connection: Keep-Alive[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350] >> "Accept-Encoding: gzip,deflate[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,350] >> "[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:50,351] >> "{"some_key":"some_data_value"}"
[DEBUG][wire][2018-02-20 18:29:51,709] << "HTTP/1.1 200 OK[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,713] << "Cache-Control: private, must-revalidate[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,713] << "Cache-control: no-cache="set-cookie"[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,713] << "Content-Type: application/json[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,713] << "Date: Tue, 20 Feb 2018 16:29:51 GMT[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,713] << "expires: 0[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,714] << "pragma: no-cache[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,714] << "Server: someserver/1.13.8[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,714] << "Set-Cookie: thecookie=123adsdf.......[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,714] << "Connection: keep-alive[\r][\n]"
[DEBUG][wire][2018-02-20 18:29:51,714] << "[\r][\n]"
[DEBUG][DefaultClientConnection][2018-02-20 18:29:51,715] Receiving response: HTTP/1.1 200 OK
[DEBUG][DefaultHttpClient][2018-02-20 18:29:51,721] Connection can be kept alive indefinitely
[DEBUG][RequestSpecificationImpl$RestAssuredHttpBuilder][2018-02-20 18:29:51,772] Parsing response

```

is it possible to log a single object for each request.  
i was thinking something like this (or something similar just to separate the request and response headers) :

{  
"message" :"...the original message",  
"request\_headers" : [{"key":"Accept", "value":"/somepath HTTP/1.1"}, {"key":"Content-Type:", "value":"application/json; charset=UTF-8"}, ..... and so on],  
"response\_headers": [{"key":"HTTP/1.1", "value":"200"}, {"key":"Cache-Control:", "value":"private, must-revalidate"}, ..... and so on],  
"response\_body": "i am the response body"  
}

thank you

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 20, 2018, 8:18pm UTC](https://discuss.elastic.co/t/group-apache-org-apache-http-request-into-a-single-object/120684/2 "2018-02-20T20:18:03Z")

</div>

You'll have to use a multiline codec (or the equivalent on the Filebeat side if you're using Filebeat to ship the file to Logstash). The logic would be: If the message part of the line is "\<\<" or "\>\>", join with the previous line.

---

<div class="post-metadata">

**Author:** ![shacoshe](https://avatars.discourse-cdn.com/v4/letter/s/e19b73/32.png) [@shacoshe](https://discuss.elastic.co/u/shacoshe)\
**Post date:** [February 21, 2018, 11:27am UTC](https://discuss.elastic.co/t/group-apache-org-apache-http-request-into-a-single-object/120684/3 "2018-02-21T11:27:12Z")

</div>

> [@magnusbaeck](#):
>
> revious line.

* * *

## FILTER :

```
filter {
  grok {
    match => {
      "message" => "\[%{LOGLEVEL:severity}.*\]\[wire\]\[%{TIMESTAMP_ISO8601:timestamp}\] ((\>\> \"%{DATA:request_key}(\s|:\s)%{GREEDYDATA:request_value}\")|(\<\< \"%{DATA:response_key}(\s|:\s)%{GREEDYDATA:response_value}\"))"
    }
    overwrite => ["key", "value"]
  }

  multiline {
    pattern => "(\>\>|\<\<)"
    what => "previous"
    negate => false
  }
}

```

* * *

## RESULT :

```
{
  "tags": [
    "multiline",
    "_grokparsefailure"
  ],
  "@version": "1",
  "response_key": [
    "HTTP/1.1",
    "Cache-Control",
    "Cache-control",
    "Content-Type",
    "Date",
    "expires",
    "pragma",
    "Server",
    "Set-Cookie",
    "Strict-Transport-Security",
    "X-Frame-Options",
    "X-XSS-Protection",
    "Content-Length",
    "Connection"
  ],
  "timestamp": [
    "2018-02-21 10:44:54,754",
    "2018-02-21 10:44:54,755",
    "2018-02-21 10:44:55,077",
    "2018-02-21 10:44:55,079"
  ],
  "host": "somehost.com",
  "severity": "DEBUG",
  "request_key": [
    "POST",
    "Accept",
    "Content-Type",
    "Content-Length",
    "Host",
    "Connection",
    "User-Agent",
    "Accept-Encoding",
    "{\"somedata_key\":\"some_data_value\"}"
  ],
  "response_value": [
    "200 OK[\\r][\\n]",
    "private, must-revalidate[\\r][\\n]",
    "no-cache=\"set-cookie\"[\\r][\\n]",
    "application/json[\\r][\\n]",
    "Wed, 21 Feb 2018 08:44:55 GMT[\\r][\\n]",
    "0[\\r][\\n]",
    "no-cache[\\r][\\n]",
    "someserver/1.13.8[\\r][\\n]",
    "somekey=somevalues[\\r][\\n]",
    "SAMEORIGIN[\\r][\\n]",
    "1; mode=block[\\r][\\n]",
    "1439[\\r][\\n]",
    "keep-alive[\\r][\\n]"
  ],
  "@timestamp": "2018-02-21T08:46:13.003Z",
  "message": "THIS IS A VERY LONG ORIGINAL LOG MESSAGE",
  "path": "the path to the log.log",
  "request_value": [
    "/access/login HTTP/1.1[\\r][\\n]",
    "*/*[\\r][\\n]",
    "application/json; charset=UTF-8[\\r][\\n]",
    "253[\\r][\\n]",
    "some.domain.com[\\r][\\n]",
    "Keep-Alive[\\r][\\n]",
    "Apache-HttpClient/4.5.3 (Java/1.8.0_45)[\\r][\\n]",
    "gzip,deflate[\\r][\\n]",
    "(Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.86 Safari/537.36\"}"
  ]
}

```

* * *

## WANTED: i wish to have

```
{
	.....
	"request": [{"key": "as dad", "value" : "asdasd"}, {"key": "as 123", "value" : "123"}],
	"response": [{"key": "as dad", "value" : "asdasd"}, {"key": "as 123", "value" : "123"}],
	.....
}

```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 21, 2018, 11:33am UTC](https://discuss.elastic.co/t/group-apache-org-apache-http-request-into-a-single-object/120684/4 "2018-02-21T11:33:32Z")

</div>

Use the multiline _codec_, not the filter.

You can use a ruby filter to join an array of keys with an array of values and produce a hash.

---

<div class="post-metadata">

**Author:** ![shacoshe](https://avatars.discourse-cdn.com/v4/letter/s/e19b73/32.png) [@shacoshe](https://discuss.elastic.co/u/shacoshe)\
**Post date:** [February 21, 2018, 1:28pm UTC](https://discuss.elastic.co/t/group-apache-org-apache-http-request-into-a-single-object/120684/5 "2018-02-21T13:28:23Z")

</div>

> [@magnusbaeck](#):
>
> use a ruby filter to join an array of keys with an array of values and produce a has

thank you.  
where can u find an example to something similar

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 21, 2018, 1:34pm UTC](https://discuss.elastic.co/t/group-apache-org-apache-http-request-into-a-single-object/120684/6 "2018-02-21T13:34:40Z")

</div>

I think it's been discussed here before. As for how to join (zip) arrays in Ruby see e.g. [https://stackoverflow.com/questions/23113759/how-to-zip-two-arrays-into-hash](https://stackoverflow.com/questions/23113759/how-to-zip-two-arrays-into-hash).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 21, 2018, 1:35pm UTC](https://discuss.elastic.co/t/group-apache-org-apache-http-request-into-a-single-object/120684/7 "2018-03-21T13:35:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.