Error when definining a mapping for pcap index

[l4m4l]

Hi everyone,

I am currently attempting to use tshark 3.2.0 and elasticsearch 7.4.1 to visualise network traffic from a pcap file. The steps I am following are:

1. Generate test.json in elasticsearch bulk API format:
   tshark -r test.pcap -T ek > test.json

2. Generate elasticsearch mapping file (and remove duplicate keys using 'jq'):
   tshark -G elastic-mapping | jq '.' > pcap_mapping.json

3. Create a new index on elasticsearch:
   curl -X PUT localhost:9200/packets-2004-05-13

4. Assign this index my mapping file from step 2:
   curl "localhost:9200/packets-2004-05-13/_mapping?pretty" -H 'Content-Type: application/json' --data-binary "@/data/pcap/pcap_mapping.json"

5. Upload the test.json file using elasticsearch bulk API method:
   curl -H "Content-Type: application/x-ndjson" -XPOST "localhost:9200/packets-2004-05-13/_bulk" --data-binary "@/data/pcap/test_trace.pcap.json"

Unfortunately, this process fails at steps 4 and 5. The mapping file is quite large so I'm hesitant to post all of it but here is the start:

{
  "template": "packets-*",
  "settings": {
"index.mapping.total_fields.limit": 1000000
  },
  "mappings": {
"pcap_file": {
  "dynamic": false,
  "properties": {
    "timestamp": {
      "type": "date"
    },
    "layers": {
      "properties": {
        "dns": {
          "properties": {
            "dns_length": {
              "type": "integer" ... 

And the error produced by step 4:

"error" : {
"root_cause" : [
  {
    "type" : "mapper_parsing_exception",
    "reason" : "Root mapping definition has unsupported parameters:  [template : packets-*] [settings : {index.mapping.total_fields.limit=1000000}] [mappings : {pcap_file={dynamic=false,...

I've tried uploading the test.json file (step 5) without defining the mapping, but elasticsearch assigns type: keyword to all values in my index which renders me unable to manipulate the data correctly.

I have also tried uploading the mapping but removing the start up to the second "properties".

PUT /packets-2004-05-13/_mapping {
      "properties": {
        "dns": {
          "properties": {
            "dns_length": {
              "type": "integer"
            },
            "dns_flags": {
              "type": "integer"
            },
            "dns_flags_response": {
              "type": "boolean"
            },...

This successfully completes, but then when I attempt to load my json bulk API data in (step 5), it fails with:
"error":{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [packets-2004-05-13] as the final mapping would have more than 1 type: [_doc, pcap_file]"}}}

I'm pretty lost as to what to try next, any help would be greatly appreciated! Thanks

[l4m4l]

For anyone that is experiencing similar problems, I think I have found a workaround. The main issue with what I am trying to do appears to be stemming from the incompatibility of the tshark output, and what elasticsearch is expecting.

The workaround steps are as follows:

1. Generate test.json in ES bulk API format and remove all references to "_type":"pcap_file" (I just used find+replace in gedit to replace with nothing):
   tshark -r test.pcap -T ek > test.json

2. Generate elasticsearch mapping file (and remove duplicate keys using 'jq'). Use the --elastic-mapping-filter option to reduce the size of this file (source):
   tshark -G elastic-mapping --elastic-mapping-filter tcp,ip,udp,dns | jq '.' > pcap_mapping.json

3. Open the mapping file and remove all fields before "dynamic": false, ....

4. Still in the mapping file, find and replace all instances of "string" with "text"

Now you should be able to follow from steps 3-5 as above. Hope this helps anyone! Feel free to message if you are having issues.

It would still be great if anyone can offer more insight into what I could do instead of this tedious workaround. I'm not certain what the latest version of ES is that is compatible with tshark output, but that would be very useful to know as I might downgrade until tshark is updated.