How would I mutate a header hash object that has headers that change from request to request?

Elastic Stack Logstash
1

I am parsing my filebeat logs,

The original string looks like this:

[2018-04-04 13:18:37 -0700] INFO -- 200 -- db=0.0 total=51.66 view=51.66 -- GET /trades headers={"Version"=>"HTTP/1.1", "Host"=>"localhost:3000", "Connection"=>"keep-alive", "Accept"=>"application/json, text/plain, */*", "Origin"=>"http://localhost:8080", "User-Agent"=>"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36", "Token"=>"YXE-RGNw12qogkJbHNDKSV_A_2i68wOgxrjFAxYRSG_kdnhCgjLIYSIvrRHYQfHXxc2loAu5U-4CbiX0qGyqZ1PJzgx3PULKm8ZAZsxUKNGcJj8BzXw", "Referer"=>"http://localhost:8080/employee-lists/control-air-union", "Accept-Encoding"=>"gzip, deflate, br", "Accept-Language"=>"en-US,en;q=0.9"} host=localhost ip=::1 params={} ua=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36

I am extracting out all the fields, but some of these fields are hash objects.... how can I parse these hash objects into json fields?

an example of one of the hash object fields, is my headers

{"Version"=>"HTTP/1.1", "Host"=>"localhost:3000", "Connection"=>"keep-alive", "Accept"=>"application/json, text/plain, */*", "Origin"=>"http://localhost:8080", "User-Agent"=>"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36", "Token"=>"YXE-RGNw12qogkJbHNDKSV_A_2i68wOgxrjFAxYRSG_kdnhCgjLIYSIvrRHYQfHXxc2loAu5U-4CbiX0qGyqZ1PJzgx3PULKm8ZAZsxUKNGcJj8BzXw", "Referer"=>"http://localhost:8080/employee-lists/control-air-union", "Accept-Encoding"=>"gzip, deflate, br", "Accept-Language"=>"en-US,en;q=0.9"}

so far my pipeline config file looks like the following:

input {
  beats {
    port => "5044"
  }
}
filter {
  dissect {
    mapping => { "message" => "[%{ts}] %{log_level} -- %{http_status} -- db=%{ms_till_db} total=%{ms_till_total} view=%{ms_till_view} -- %{http_verb} %{request_url} headers=%{headers} host=%{host_address} ip=%{ip} params=%{params} ua=%{ua}" }
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
  stdout {
    codec => rubydebug
  }
}

my end result is resulting in the following

{
  "_index": "logstash-2018.04.07",
  "_type": "doc",
  "_id": "qeflnmIBTBSHYDw-A-AS",
  "_version": 1,
  "_score": null,
  "_source": {
    "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36",
    "http_verb": "GET",
    "offset": 523592,
    "source": "/Users/aronlilland/Documents/dev_projects/ruby/backbone/arsenal_api/log/logfile.log",
    "ip": "::1",
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "prospector": {
      "type": "log"
    },
    "beat": {
      "hostname": "Arons-MBP.attlocal.net",
      "name": "Arons-MBP.attlocal.net",
      "version": "6.2.3"
    },
    "message": "[2018-04-04 13:18:37 -0700] INFO -- 200 -- db=0.0 total=51.66 view=51.66 -- GET /trades headers={\"Version\"=>\"HTTP/1.1\", \"Host\"=>\"localhost:3000\", \"Connection\"=>\"keep-alive\", \"Accept\"=>\"application/json, text/plain, */*\", \"Origin\"=>\"http://localhost:8080\", \"User-Agent\"=>\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36\", \"Token\"=>\"YXE-RGNw12qogkJbHNDKSV_A_2i68wOgxrjFAxYRSG_kdnhCgjLIYSIvrRHYQfHXxc2loAu5U-4CbiX0qGyqZ1PJzgx3PULKm8ZAZsxUKNGcJj8BzXw\", \"Referer\"=>\"http://localhost:8080/employee-lists/control-air-union\", \"Accept-Encoding\"=>\"gzip, deflate, br\", \"Accept-Language\"=>\"en-US,en;q=0.9\"} host=localhost ip=::1 params={} ua=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36",
    "ts": "2018-04-04 13:18:37 -0700",
    "params": "{}",
    "@timestamp": "2018-04-07T06:57:09.314Z",
    "headers": "{\"Version\"=>\"HTTP/1.1\", \"Host\"=>\"localhost:3000\", \"Connection\"=>\"keep-alive\", \"Accept\"=>\"application/json, text/plain, */*\", \"Origin\"=>\"http://localhost:8080\", \"User-Agent\"=>\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36\", \"Token\"=>\"YXE-RGNw12qogkJbHNDKSV_A_2i68wOgxrjFAxYRSG_kdnhCgjLIYSIvrRHYQfHXxc2loAu5U-4CbiX0qGyqZ1PJzgx3PULKm8ZAZsxUKNGcJj8BzXw\", \"Referer\"=>\"http://localhost:8080/employee-lists/control-air-union\", \"Accept-Encoding\"=>\"gzip, deflate, br\", \"Accept-Language\"=>\"en-US,en;q=0.9\"}",
    "ms_till_db": "0.0",
    "ms_till_total": "51.66",
    "request_url": "/trades",
    "log_level": "INFO",
    "ms_till_view": "51.66",
    "host": "Arons-MBP.attlocal.net",
    "host_address": "localhost",
    "http_status": "200",
    "@version": "1"
  },
  "fields": {
    "@timestamp": [
      "2018-04-07T06:57:09.314Z"
    ]
  },
  "sort": [
    1523084229314
  ]
}
2

You can use a json filter to parse the headers and it will populate them under whatever field you ask it to (or the root if you do not give it a target).

json { source => "headers" target => "somefield" }
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.