Sticking same top-level fields

Elastic Stack Logstash
1

Hello. I've next json generated by tshark and exported with flag -T ek:
{
"timestamp": "1507110658491",
"layers": {
"frame": {
"frame_frame_interface_id": "0", "frame_frame_encap_type": "1", "frame_frame_time": "Oct 4, 2017 09:50:58.491647000 UTC", "frame_frame_offset_shift": "0.000000000", "frame_frame_time_epoch": "1507110658.491647000", "frame_frame_time_delta": "0.000034000", "frame_frame_time_delta_displayed": "0.000034000", "frame_frame_time_relative": "16.587719000", "frame_frame_number": "994", "frame_frame_len": "1414", "frame_frame_cap_len": "1414", "frame_frame_marked": "0", "frame_frame_ignored": "0", "frame_frame_protocols": "eth:ethertype:ip:tcp:http:ssl:http:ssl"
}
,
"eth": {
"eth_eth_dst": "90:b1:1c:a1:7f:18", "eth_dst_eth_dst_resolved": "Dell_a1:7f:18", "eth_dst_eth_addr": "90:b1:1c:a1:7f:18", "eth_dst_eth_addr_resolved": "Dell_a1:7f:18", "eth_dst_eth_lg": "0", "eth_dst_eth_ig": "0", "eth_eth_src": "30:f7:0d:b8:6c:7f", "eth_src_eth_src_resolved": "Cisco_b8:6c:7f", "eth_src_eth_addr": "30:f7:0d:b8:6c:7f", "eth_src_eth_addr_resolved": "Cisco_b8:6c:7f", "eth_src_eth_lg": "0", "eth_src_eth_ig": "0", "eth_eth_type": "0x00000800"
}
,
"ip": {
"ip_ip_version": "4", "ip_ip_hdr_len": "20", "ip_ip_dsfield": "0x00000000", "ip_dsfield_ip_dsfield_dscp": "0", "ip_dsfield_ip_dsfield_ecn": "0", "ip_ip_len": "1400", "ip_ip_id": "0x00006b18", "ip_ip_flags": "0x00000000", "ip_flags_ip_flags_rb": "0", "ip_flags_ip_flags_df": "0", "ip_flags_ip_flags_mf": "0", "ip_ip_frag_offset": "0", "ip_ip_ttl": "62", "ip_ip_proto": "6", "ip_ip_checksum": "0x000065ed", "ip_ip_checksum_status": "2", "ip_ip_src": "10.64.48.8", "ip_ip_addr": "10.64.48.8", "ip_ip_src_host": "10.64.48.8", "ip_ip_host": "10.64.48.8", "ip_ip_dst": "10.16.98.35", "ip_ip_addr": "10.16.98.35", "ip_ip_dst_host": "10.16.98.35", "ip_ip_host": "10.16.98.35", "ip_text": "Source GeoIP: Unknown", "ip_text": "Destination GeoIP: Unknown"
}
,
"tcp": {
"tcp_tcp_srcport": "8080", "tcp_tcp_dstport": "49658", "tcp_tcp_port": "8080", "tcp_tcp_port": "49658", "tcp_tcp_stream": "42", "tcp_tcp_len": "1360", "tcp_tcp_seq": "30867", "tcp_tcp_nxtseq": "32227", "tcp_tcp_ack": "2603", "tcp_tcp_hdr_len": "20", "tcp_tcp_flags": "0x00000010", "tcp_flags_tcp_flags_res": "0", "tcp_flags_tcp_flags_ns": "0", "tcp_flags_tcp_flags_cwr": "0", "tcp_flags_tcp_flags_ecn": "0", "tcp_flags_tcp_flags_urg": "0", "tcp_flags_tcp_flags_ack": "1", "tcp_flags_tcp_flags_push": "0", "tcp_flags_tcp_flags_reset": "0", "tcp_flags_tcp_flags_syn": "0", "tcp_flags_tcp_flags_fin": "0", "tcp_flags_tcp_flags_str": "\u00c2\u00b7\u00c2\u00b7\u00c2\u00b7\u00c2\u00b7\u00c2\u00b7\u00c2\u00b7\u00c2\u00b7A\u00c2\u00b7\u00c2\u00b7\u00c2\u00b7\u00c2\u00b7", "tcp_tcp_window_size_value": "65535", "tcp_tcp_window_size": "65535", "tcp_tcp_window_size_scalefactor": "1", "tcp_tcp_checksum": "0x00000a44", "tcp_tcp_checksum_status": "2", "tcp_tcp_urgent_pointer": "0", "tcp_tcp_analysis": "", "tcp_analysis_tcp_analysis_initial_rtt": "0.000300000", "tcp_analysis_tcp_analysis_bytes_in_flight": "1360", "tcp_analysis_tcp_analysis_push_bytes_sent": "5440"
}
,
"tcp_segments": "",
"tcp_segments_tcp_segment": "989",
"tcp_segments_tcp_segment": "991",
"tcp_segments_tcp_segment": "992",
"tcp_segments_tcp_segment": "994",
"tcp_segments_tcp_segment_count": "4",
"tcp_segments_tcp_reassembled_length": "4077",
"http": {
"http_http_proxy_connect_host": "62.102.148.67", "http_http_proxy_connect_port": "443"
}
,
"ssl": {
"ssl_ssl_record": "", "ssl_record_ssl_record_content_type": "23", "ssl_record_ssl_record_version": "0x00000303", "ssl_record_ssl_record_length": "4072"
}
,
"http": {
"http_http_proxy_connect_host": "62.102.148.67", "http_http_proxy_connect_port": "443"
}
,
"ssl": "Secure Sockets Layer"
}
}

I'm making subfields as topfields with the mutate plugin, for example:
mutate {
rename => {
"[layers][ssl][ssl_record_ssl_record_content_type]" => "ssl_record_ssl_record_content_type"
}
}

but the problem is that i have 2 top lvl fields named 'ssl' (and that's not the only field like this). So when i'm sending this data to elastic it writes only last field - "ssl": "Secure Sockets Layer" and other data is lost.

How can i stick this top level fields (ssl for example)? I know that i can use ruby filter for that, but can't find any examples of sticking same named topfields.

Thank you.

2

I think this maybe what you are looking for ?

  grok {
    match => {
    "[layers][ssl][ssl_record_ssl_record_content_type]" => "%{NUMBER:choose_your_name}$"
    }
  }
3

I think I was too quick to answer... is the real problem here that the duplicated JSON keys not showing up in the event at all ?

4

It seems that duplicate key names in JSON is arguably within the specification, but that most libraries (the one Logstash uses included) doesn't support duplicated keys.

You may need a way to prevent the duplicate keys from the source or do some String based manipulation of the JSON (yuk) prior to treating as JSON.

I should also mention that https://www.elastic.co/products/beats/packetbeat -> Logstash (beats input) may be a better solution here.

5

Thank you for reply.

I use filebeat, cause writing logs to .json with tshark.

Yes the problem was with duplicate keys and i found a solution. The reason of duplicated keys was a bug in wireshark log jsoned by tshark -T ek.

Also packetbeat doesn't support all the protocols i need.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.