Dear community,
I have some .pcap files that I want to index using elastic search.
After reading the following article, I wanted to follow the same strategy, namely convert the .pcap to .json using tshark -T ek test.pcap > test.json :
When I then tried to load the json into elasticsearch (6.2) I got an error that the json contained duplicate fields. After some research i realized that the import fails, because there are duplicate names of the fields. I think, since version 6.0 elasticsearch is more strictly when it comes to checking for duplicates.
So I updated to Wirteshark developement release 2.5 where they fixed this issue.
The problem I am now having is that when I try to import the .json data I get the following response for some of the packets: (curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/_bulk?pretty' --data-binary @test.json)
{
"index" : {
"_index" : "packets-2018-04-13",
"_type" : "pcap_file",
"_id" : "sS1_v2IB91sQiYI82aPO",
"status" : 400,
"error" : {
"type" : "illegal_argument_exception",
"reason" : "Limit of total fields [1000] in index [packets-2018-04-13] has been exceeded"
}
}
}
The article above comments this issue and suggets to introduce a mapping:
"Consequently, creating an index on all these fields is usually not the right thing to do. In fact, since a large number of fields can slow down both indexing and query speed Elasticsearch 5.5 limits the number of fields in an index to 1000 by default."
"To index numbers as numbers, timestamps as timestamps, etc. and to prevent an explosion of indexed fields, you should explicitly specify an Elasticsearch mapping. Here is an example:"
So I loaded the suggested mapping into elastic search, but I still get the same error.
What might be the problem here?
Here are the curl commands I used to load the mapping, and a curl command to upload a document for a packet where the error arrises, so that you can reproduce the problem:
curl -XPUT "http://localhost:9200/_template/packets" -H 'Content-Type: application/json' -d'
{
"template": "packets-*",
"mappings": {
"pcap_file": {
"dynamic": "false",
"properties": {
"timestamp": {
"type": "date"
},
"layers": {
"properties": {
"frame": {
"properties": {
"frame_frame_len": {
"type": "long"
},
"frame_frame_protocols": {
"type": "keyword"
}
}
},
"ip": {
"properties": {
"ip_ip_src": {
"type": "ip"
},
"ip_ip_dst": {
"type": "ip"
}
}
},
"udp": {
"properties": {
"udp_udp_srcport": {
"type": "integer"
},
"udp_udp_dstport": {
"type": "integer"
}
}
}
}
}
}
}
}
}'
curl -XPOST "http://localhost:9200/packets/_doc/_bulk?pretty" -H 'Content-Type: application/json' -d'
{"index" : {"_index": "packets-2018-04-13", "_type": "pcap_file"}}
{"timestamp" : "1493208068301", "layers" : {"frame": {"frame_frame_encap_type": "1","frame_frame_time": "Apr 26, 2017 14:01:08.301052000 CEST","frame_frame_offset_shift": "0.000000000","frame_frame_time_epoch": "1493208068.301052000","frame_frame_time_delta": "0.000007000","frame_frame_time_delta_displayed": "0.000007000","frame_frame_time_relative": "47.264793000","frame_frame_number": "5000","frame_frame_len": "165","frame_frame_cap_len": "165","frame_frame_marked": "0","frame_frame_ignored": "0","frame_frame_protocols": "eth:ethertype:ip:udp:mndp"},"eth": {"eth_eth_dst": "ff:ff:ff:ff:ff:ff","eth_dst_eth_dst_resolved": "Broadcast","eth_dst_eth_addr": "ff:ff:ff:ff:ff:ff","eth_dst_eth_addr_resolved": "Broadcast","eth_dst_eth_lg": "1","eth_dst_eth_ig": "1","eth_eth_src": "00:50:56:9e:05:1b","eth_src_eth_src_resolved": "Vmware_9e:05:1b","eth_src_eth_addr": "00:50:56:9e:05:1b","eth_src_eth_addr_resolved": "Vmware_9e:05:1b","eth_src_eth_lg": "0","eth_src_eth_ig": "0","eth_eth_type": "0x00000800"},"ip": {"ip_ip_version": "4","ip_ip_hdr_len": "20","ip_ip_dsfield": "0x00000000","ip_dsfield_ip_dsfield_dscp": "0","ip_dsfield_ip_dsfield_ecn": "0","ip_ip_len": "151","ip_ip_id": "0x00000000","ip_ip_flags": "0x00004000","ip_flags_ip_flags_rb": "0","ip_flags_ip_flags_df": "1","ip_flags_ip_flags_mf": "0","ip_flags_ip_frag_offset": "0","ip_ip_ttl": "64","ip_ip_proto": "17","ip_ip_checksum": "0x00009b7d","ip_ip_checksum_status": "2","ip_ip_src": "151.216.7.1","ip_ip_addr": ["151.216.7.1","255.255.255.255"],"ip_ip_src_host": "151.216.7.1","ip_ip_host": ["151.216.7.1","255.255.255.255"],"ip_ip_dst": "255.255.255.255","ip_ip_dst_host": "255.255.255.255"},"udp": {"udp_udp_srcport": "60474","udp_udp_dstport": "5678","udp_udp_port": ["60474","5678"],"udp_udp_length": "131","udp_udp_checksum": "0x00005f36","udp_checksum_udp_checksum_calculated": "0x00005f36","udp_udp_checksum_status": "1","udp_udp_stream": "260"},"mndp": {"mndp_mndp_header_unknown": "aa:09","mndp_mndp_header_seqno": "0","mndp_text": ["T 1, L 6: MAC-Address","T 5, L 13: Identity","T 7, L 18: Version","T 8, L 8: Platform","T 10, L 4: Uptime","T 11, L 0: Software-ID","T 12, L 3: Board","T 14, L 1: Unpack","T 15, L 16: IPv6-Address","T 16, L 10: Interface name"],"text_mndp_tlv_type": ["1","5","7","8","10","11","12","14","15","16"],"text_mndp_tlv_length": ["6","13","18","8","4","0","3","1","16","10"],"text_mndp_mac": "00:50:56:9e:05:1b","text_mndp_identity": "fw1.blue07.ex","text_mndp_version": "6.39rc38 (testing)","text_mndp_platform": "MikroTik","text_mndp_uptime": "74231.000000000","text_mndp_board": "CHR","text_mndp_unpack": "1","text_mndp_ipv6address": "2a07:1182:7:1::1","text_mndp_interfacename": "ether5-ops"}}}
'