Decoding gzip response body from Packetbeat

[lusid (Marc Melvin)]

Hello everyone,

This has been driving me crazy lately, so any help that can be provided would be HUGELY appreciated.

I am using Packetbeat to monitor the requests/responses into/out of Elasticsearch client nodes using the http protocol watcher on port 9200. I am sending the output of Packetbeat through Logstash, and then from there out to a different instance of Elasticsearch. We have compression support enabled in the Elasticsearch that is being monitored, so I occasionally see requests with "Accept-Encoding: gzip, deflate" headers returning responses that are gzipped. Unfortunately, I have not been able to decode any of these gzip responses using any tools I have at my disposal (including the web-based converters, the gzip command line tool, and using Zlib::GzipReader in a Logstash ruby filter script). They all report that it is not a gzip format.

Does anyone know why I can't seem to decode the gzip content?

I have provided a sample of the filter I'm using in Logstash to try to do this on the fly as the event passes through Logstash (and it always reports that http.response.body is not in gzip format).

filter {
  if [type] == "http" {
    if [http][response][headers][content-encoding] == "gzip" {
      ruby {
        init => "
          require 'zlib'
          require 'stringio'
        "
        code => "
          body = event.get('[http][response][body]').to_s
          sio = StringIO.new(body)
          gz = Zlib::GzipReader.new(sio)
          result = gz.read.to_s
          event.set('[http][response][body]', result)
        "
      }
    }
  }
}

I'm also providing a sample of the logged event here which includes the gzip content in case you would like to try to decompress it yourself:

{
  "_index": "packetbeat-6.2.3-2018.05.19",
  "_type": "doc",
  "_id": "oH0bemMB2mAXfg5euIiP",
  "_score": 1,
  "_source": {
    "server": "",
    "client_server": "",
    "bytes_in": 160,
    "bytes_out": 361,
    "@timestamp": "2018-05-19T20:33:46.470Z",
    "client_port": 55863,
    "path": "/",
    "type": "http",
    "client_proc": "",
    "query": "GET /",
    "port": 9200,
    "host": "gke-main-production-elastic-clients-5728bab3-t1z8",
    "@version": "1",
    "responsetime": 0,
    "fields": {
      "nodePool": "production-elastic-clients"
    },
    "response": "HTTP/1.1 200 OK\r\ncontent-type: application/json; charset=UTF-8\r\ncontent-encoding: gzip\r\ncontent-length: 250\r\n\r\n\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000T��n�0\u0014Fw���\u001c\u0010\u0018�����&��vH\u0016d�K������\u0010��\u000b�C\u0018����{��\u0010š]\u0001�\u001aap1W\u0012�\u0018\u0017�,y)���oC�\n��A��\u001b�6/��\u001a�\u000e��\"l+�����\u001d\u000f\u0005y/���k�?�\u0005�\u0005���3���Y�_[���Mh�\u0007nzo�T����C�1�\u0011�]����\u0007H�\u0015q��)�&i��u^%iF�k�i6�ތs�c���)�9hh^�0�T2<�<���.J����x���}�:c�\u0011��=���\u001f\u0000\u0000\u0000��\u0003\u0000��.�S\u0001\u0000\u0000",
    "proc": "",
    "request": "GET / HTTP/1.1\r\nUser-Agent: vscode-restclient\r\nhost: es-http-dev.elastic-prod.svc.cluster.local:9200\r\naccept-encoding: gzip, deflate\r\nConnection: keep-alive\r\n\r\n",
    "beat": {
      "name": "gke-main-production-elastic-clients-5728bab3-t1z8",
      "version": "6.2.3",
      "hostname": "gke-main-production-elastic-clients-5728bab3-t1z8"
    },
    "status": "OK",
    "method": "GET",
    "client_ip": "10.24.20.6",
    "http": {
      "response": {
        "phrase": "OK",
        "headers": {
          "content-encoding": "gzip",
          "content-length": 250,
          "content-type": "application/json; charset=UTF-8"
        },
        "body": "\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000T��n�0\u0014Fw���\u001c\u0010\u0018�����&��vH\u0016d�K������\u0010��\u000b�C\u0018����{��\u0010š]\u0001�\u001aap1W\u0012�\u0018\u0017�,y)���oC�\n��A��\u001b�6/��\u001a�\u000e��\"l+�����\u001d\u000f\u0005y/���k�?�\u0005�\u0005���3���Y�_[���Mh�\u0007nzo�T����C�1�\u0011�]����\u0007H�\u0015q��)�&i��u^%iF�k�i6�ތs�c���)�9hh^�0�T2<�<���.J����x���}�:c�\u0011��=���\u001f\u0000\u0000\u0000��\u0003\u0000��.�S\u0001\u0000\u0000",
        "code": 200
      },
      "request": {
        "params": "",
        "headers": {
          "connection": "keep-alive",
          "user-agent": "vscode-restclient",
          "content-length": 0,
          "host": "es-http-dev.elastic-prod.svc.cluster.local:9200",
          "accept-encoding": "gzip, deflate"
        }
      }
    },
    "tags": [
      "beats",
      "beats_input_raw_event"
    ],
    "ip": "10.24.41.5"
  },
  "fields": {
    "@timestamp": [
      "2018-05-19T20:33:46.470Z"
    ]
  }
}

And this is the response for that message that I receive at the client after it has been decompressed successfully by the client:

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 250

{
  "name": "es-client-7688c8d9b9-qp9l7",
  "cluster_name": "esprod",
  "cluster_uuid": "8iRwLMMSR72F76ZEONYcUg",
  "version": {
    "number": "5.6.3",
    "build_hash": "1a2f265",
    "build_date": "2017-10-06T20:33:39.012Z",
    "build_snapshot": false,
    "lucene_version": "6.6.1"
  },
  "tagline": "You Know, for Search"
}