Why won't Logstash rename a parsed JSON field?

Elastic Stack Logstash
1

So I've got some Zeek logs. In particular I've got http.log which Filebeat is adding the tag "Zeek-HTTP" to. That's all working fine. I want to use the JSON plugin to parse the "message" field in Logstash to a new target and then rename one of the target field names. I figured this would be straightforward but it's not working. This is the relevant section of my Logstash .conf file:

filter {
    if "Zeek-HTTP" in [tags] {

        json {
            source => "message"
            target => "http"
        }

        mutate {
            rename => { "http.id.orig_h" => "src_ip" }
            rename => { "http.id.orig_p" => "src_port" }
            rename => { "http.id.resp_h" => "dst_ip" }
            rename => { "http.id.resp_p" => "dst_port" }
            rename => { "http.host" => "hostname" }
        }
    }
}

http.id.orig_h, http.id.orig_p, http.id.resp_h, http.id.resp_p and http.host are all fields that exist. Here's a snip:

Capture
Capture413×542 25.7 KB

I tried renaming the key before parsing "message" according to this forum post but I couldn't get that to work either. So why won't the mutate function just rename the field names?

BTW I'm doing this because, for whatever reason, the JSON plugin won't parse the http.log "message" and I can't figure out why. If I add a target to the JSON plugin then message is actually parsed. Of course the output has the http prefix which I'm ok with, but it would be nice to have src_ip and dst_ip and such instead of the other names.

2

That would work if your field name contained periods. If you have an orig_h field inside an id field inside an http field then use

rename => { "[http][id][orig_h]" => "src_ip" }
3

Unfortunately that didn't rename the field. Here's my edit in the .conf

filter {
    if "Zeek-HTTP" in [tags] {

        json {
            source => "message"
            target => "http"
        }

        mutate {
            rename => { "[http][id][orig_h]" => "source_ip" }
            rename => { "http.id.orig_p" => "src_port" }
            rename => { "http.id.resp_h" => "dst_ip" }
            rename => { "http.id.resp_p" => "dst_port" }
            rename => { "http.host" => "hostname" }
        }
    }
}

I would expect http.id.orig_h to rename to source_ip. Here's a failed rename event:

@timestamp	Feb 8, 2020 @ 11:22:45.733
	@version	1
	_id	N-5BJnABlIMgnCGwTWLQ
	_index	zeek-2020.02.08
	_score	 - 
	_type	_doc
	agent.ephemeral_id	cd308b0d-c942-440d-ba85-9fae0f75919b
	agent.hostname	qelk
	agent.id	dd92671c-36b0-4589-9080-c4ea6375aa8d
	agent.type	filebeat
	agent.version	7.5.2
	ecs.version	1.1.0
	host.name	qelk
	http.host	httpbin.org
	http.id.orig_h	10.254.10.62
	http.id.orig_p	63,475
	http.id.resp_h	100.25.11.135
	http.id.resp_p	80
	http.method	GET
	http.request_body_len	0
	http.resp_fuids	FAovP71vvDUJ4WNIIh
	http.resp_mime_types	text/json
	http.response_body_len	429
	http.status_code	200
	http.status_msg	OK
	http.tags	
	http.trans_depth	2
	http.ts	1,581,189,761.324
	http.uid	CtzUiT30pI6uQn6sTe
	http.uri	/json
	http.user_agent	Mozilla/5.0 (Windows NT; Windows NT 10.0; en-CA) WindowsPowerShell/5.1.18362.145
	http.version	1.1
	input.type	log
	log.file.path	/opt/zeek/logs/current/http.log
	log.offset	21,187
	message	{"ts":1581189761.323757,"uid":"CtzUiT30pI6uQn6sTe","id.orig_h":"10.254.10.62","id.orig_p":63475,"id.resp_h":"100.25.11.135","id.resp_p":80,"trans_depth":2,"method":"GET","host":"httpbin.org","uri":"/json","version":"1.1","user_agent":"Mozilla/5.0 (Windows NT; Windows NT 10.0; en-CA) WindowsPowerShell/5.1.18362.145","request_body_len":0,"response_body_len":429,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["FAovP71vvDUJ4WNIIh"],"resp_mime_types":["text/json"]}
	tags	zeek, Zeek-HTTP, beats_input_codec_plain_applied

And the JSON for the event.

{
  "_index": "zeek-2020.02.08",
  "_type": "_doc",
  "_id": "N-5BJnABlIMgnCGwTWLQ",
  "_version": 1,
  "_score": null,
  "_source": {
    "message": "{\"ts\":1581189761.323757,\"uid\":\"CtzUiT30pI6uQn6sTe\",\"id.orig_h\":\"10.254.10.62\",\"id.orig_p\":63475,\"id.resp_h\":\"100.25.11.135\",\"id.resp_p\":80,\"trans_depth\":2,\"method\":\"GET\",\"host\":\"httpbin.org\",\"uri\":\"/json\",\"version\":\"1.1\",\"user_agent\":\"Mozilla/5.0 (Windows NT; Windows NT 10.0; en-CA) WindowsPowerShell/5.1.18362.145\",\"request_body_len\":0,\"response_body_len\":429,\"status_code\":200,\"status_msg\":\"OK\",\"tags\":[],\"resp_fuids\":[\"FAovP71vvDUJ4WNIIh\"],\"resp_mime_types\":[\"text/json\"]}",
    "input": {
      "type": "log"
    },
    "http": {
      "id.orig_p": 63475,
      "uri": "/json",
      "id.orig_h": "10.254.10.62",
      "request_body_len": 0,
      "resp_fuids": [
        "FAovP71vvDUJ4WNIIh"
      ],
      "resp_mime_types": [
        "text/json"
      ],
      "id.resp_h": "100.25.11.135",
      "tags": [],
      "id.resp_p": 80,
      "response_body_len": 429,
      "status_code": 200,
      "status_msg": "OK",
      "trans_depth": 2,
      "method": "GET",
      "uid": "CtzUiT30pI6uQn6sTe",
      "user_agent": "Mozilla/5.0 (Windows NT; Windows NT 10.0; en-CA) WindowsPowerShell/5.1.18362.145",
      "ts": 1581189761.323757,
      "host": "httpbin.org",
      "version": "1.1"
    },
    "ecs": {
      "version": "1.1.0"
    },
    "@timestamp": "2020-02-08T19:22:45.733Z",
    "agent": {
      "version": "7.5.2",
      "ephemeral_id": "cd308b0d-c942-440d-ba85-9fae0f75919b",
      "id": "dd92671c-36b0-4589-9080-c4ea6375aa8d",
      "type": "filebeat",
      "hostname": "qelk"
    },
    "tags": [
      "zeek",
      "Zeek-HTTP",
      "beats_input_codec_plain_applied"
    ],
    "host": {
      "name": "qelk"
    },
    "log": {
      "offset": 21187,
      "file": {
        "path": "/opt/zeek/logs/current/http.log"
      }
    },
    "@version": "1"
  },
  "fields": {
    "@timestamp": [
      "2020-02-08T19:22:45.733Z"
    ]
  },
  "highlight": {
    "log.file.path": [
      "/@kibana-highlighted-field@opt@/kibana-highlighted-field@/@kibana-highlighted-field@zeek@/kibana-highlighted-field@/@kibana-highlighted-field@logs@/kibana-highlighted-field@/@kibana-highlighted-field@current@/kibana-highlighted-field@/@kibana-highlighted-field@http.log@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1581189765733
  ]
}
4

OK, you have fields inside [http] whose names contain periods. So you need

rename => { "[http][id.orig_h]" => "source_ip" }

Similarly for the other 3 host/port fields.

5

@Badger wow, thank you so much for your help. That worked. I had been trying to get these fields to rename for a long time and was failing. It sounds like I have a fundamental misunderstanding of JSON?

I was thinking the field was "message" but your reply says "http" is the field. So is it correct to say, for example, "http" is the field, "uri" is the key and "/json" is the value? What would "message" be called in this case?

Again, thanks a ton for helping a n00b understand.

6

I would call [http][uri] a field too (or nested field). "/json" is the value.

7

Thanks for clarifying. And thanks for helping everyone on the board here. I see your nick everywhere. Cheers!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.