In v.7.0.1 Filebeat and Elasticsearch with v.3.0.2 Wireshark I am having the problem that my fields are not mapping and I get json decode errors.
Tshark run command:
.\tshark -i Ethernet0 -T ek > c:\packetLogs\packetLog.json
A sample of resulting data, it contains one tshark event, however to Elastic this is actually two events:
{"index":{"_index":"packets-2019-06-14","_type":"pcap_file"}}
{"timestamp":"1560533483560","layers":{"frame":{"frame_frame_interface_id":"0","frame_interface_id_frame_interface_name":"\\Device\\NPF_{0E28C29D-0EA5-4B2E-9456-23AFBF2EC7EC}","frame_interface_id_frame_interface_description":"Ethernet0","frame_frame_encap_type":"1","frame_frame_time":"Jun 14, 2019 10:31:23.560650000 US Mountain Standard Time","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1560533483.560650000","frame_frame_time_delta":"0.000000000","frame_frame_time_delta_displayed":"0.000000000","frame_frame_time_relative":"0.000000000","frame_frame_number":"1","frame_frame_len":"60","frame_frame_cap_len":"60","frame_frame_marked":"0","frame_frame_ignored":"0","frame_frame_protocols":"eth:ethertype:ip:tcp:data"},"eth":{"eth_eth_dst":"00:0c:29:a1:d7:ce","eth_dst_eth_dst_resolved":"Vmware_a1:d7:ce","eth_dst_eth_addr":"00:0c:29:a1:d7:ce","eth_dst_eth_addr_resolved":"Vmware_a1:d7:ce","eth_dst_eth_lg":"0","eth_dst_eth_ig":"0","eth_eth_src":"00:0c:29:05:fa:2d","eth_src_eth_src_resolved":"Vmware_05:fa:2d","eth_src_eth_addr":"00:0c:29:05:fa:2d","eth_src_eth_addr_resolved":"Vmware_05:fa:2d","eth_src_eth_lg":"0","eth_src_eth_ig":"0","eth_eth_type":"0x00000800","eth_eth_padding":"00:00:00:00:00"},"ip":{"ip_ip_version":"4","ip_ip_hdr_len":"20","ip_ip_dsfield":"0x00000000","ip_dsfield_ip_dsfield_dscp":"0","ip_dsfield_ip_dsfield_ecn":"0","ip_ip_len":"41","ip_ip_id":"0x0000745b","ip_ip_flags":"0x00004000","ip_flags_ip_flags_rb":"0","ip_flags_ip_flags_df":"1","ip_flags_ip_flags_mf":"0","ip_flags_ip_frag_offset":"0","ip_ip_ttl":"128","ip_ip_proto":"6","ip_ip_checksum":"0x0000fa1f","ip_ip_checksum_status":"2","ip_ip_src":"192.168.5.129","ip_ip_addr":["192.168.5.129","192.168.5.130"],"ip_ip_src_host":"192.168.5.129","ip_ip_host":["192.168.5.129","192.168.5.130"],"ip_ip_dst":"192.168.5.130","ip_ip_dst_host":"192.168.5.130"},"tcp":{"tcp_tcp_srcport":"59960","tcp_tcp_dstport":"9200","tcp_tcp_port":["59960","9200"],"tcp_tcp_stream":"0","tcp_tcp_len":"1","tcp_tcp_seq":"1","tcp_tcp_nxtseq":"2","tcp_tcp_ack":"1","tcp_tcp_hdr_len":"20","tcp_tcp_flags":"0x00000010","tcp_flags_tcp_flags_res":"0","tcp_flags_tcp_flags_ns":"0","tcp_flags_tcp_flags_cwr":"0","tcp_flags_tcp_flags_ecn":"0","tcp_flags_tcp_flags_urg":"0","tcp_flags_tcp_flags_ack":"1","tcp_flags_tcp_flags_push":"0","tcp_flags_tcp_flags_reset":"0","tcp_flags_tcp_flags_syn":"0","tcp_flags_tcp_flags_fin":"0","tcp_flags_tcp_flags_str":"┬╖┬╖┬╖┬╖┬╖┬╖┬╖A┬╖┬╖┬╖┬╖","tcp_tcp_window_size_value":"2049","tcp_tcp_window_size":"2049","tcp_tcp_window_size_scalefactor":"-1","tcp_tcp_checksum":"0x00004755","tcp_tcp_checksum_status":"2","tcp_tcp_urgent_pointer":"0","tcp_tcp_analysis":null,"tcp_analysis_tcp_analysis_bytes_in_flight":"1","tcp_analysis_tcp_analysis_push_bytes_sent":"1","tcp_text":"Timestamps","text_tcp_time_relative":"0.000000000","text_tcp_time_delta":"0.000000000","tcp_tcp_payload":"00"},"data":{"data_data_data":"00","data_data_len":"1"}}}
I use the following Filebeat command to generate the log and run Filebeat:
.\filebeat -e -d "*"
Here is my filebeat.yml and my template.json.
I then get the runtime report.
I don't think my filebeat.yml is wrong. At the end I'm not sure if this is a template issue with Elasticsearch causing fields not to map, or if Filebeat is not parsing the json file properly, or if Tshark is perhaps not generating the json file correctly.
I will say each independent json object passes validation and also that Elastic will map this to the message field.