Unable to add my JSON file to elasticsearch get error "Can't merge a non object mapping"

[zebulyon]

Hello,
I'm currently running the ELK-stack through the official docker containers version 7.6.0.

I have some json files generated by using tshark to convert pcap files to JSON.
As per this guide:

Analyzing network packets with Wireshark, Elasticsearch, and Kibana

Now I'm trying to add these JSON files to Elasticsearch. And as far as I can find the best way to this is by using cUrl

I'm working on debian 10 stretch with curl version:

curl 7.64.0 (x86_64-pc-linux-gnu) libcurl/7.64.0 OpenSSL/1.1.1d zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.36.0 librtmp/2.3
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL 

To add my file to Elasticsearch I use this command:

curl -XPOST 'http://localhost:9200/test/_doc/1' -H'Content-Type: application/json' -d @packets.json

But I get this error in return:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't merge a non object mapping [_items._source.layers.ip.ip.checksum] with an object mapping [_items._source.layers.ip.ip.checksum]"}],"type":"illegal_argument_exception","reason":"Can't merge a non object mapping [_items._source.layers.ip.ip.checksum] with an object mapping [_items._source.layers.ip.ip.checksum]"},"status":400}

This is how my Json file look like:

{"_items":
  [
    {
      "_index": "packets-2020-02-06",
      "_type": "pcap_file",
      "_score": null,
      "_source": {
        "layers": {
          "frame": {
            "frame.encap_type": "25",
            "frame.time": "Dec  2, 2019 07:37:53.050476000 CET",
            "frame.offset_shift": "0.000000000",
            "frame.time_epoch": "1575268673.050476000",
            "frame.time_delta": "0.000000000",
            "frame.time_delta_displayed": "0.000000000",
            "frame.time_relative": "0.000000000",
            "frame.number": "1",
            "frame.len": "360",
            "frame.cap_len": "360",
            "frame.marked": "0",
            "frame.ignored": "0",
            "frame.protocols": "sll:ethertype:ip:sctp:data"
          },
          "sll": {
            "sll.pkttype": "0",
            "sll.hatype": "1",
            "sll.halen": "6",
            "sll.src.eth": "fa:16:3e:e6:39:ec",
            "sll.unused": "00:00",
            "sll.etype": "0x00000800"
          },
          "ip": {
            "ip.version": "4",
            "ip.hdr_len": "20",
            "ip.dsfield": "0x00000000",
            "ip.dsfield_tree": {
              "ip.dsfield.dscp": "0",
              "ip.dsfield.ecn": "0"
            },
            "ip.len": "344",
            "ip.id": "0x00000000",
            "ip.flags": "0x00004000",
            "ip.flags_tree": {
              "ip.flags.rb": "0",
              "ip.flags.df": "1",
              "ip.flags.mf": "0",
              "ip.frag_offset": "0"
            },
            "ip.ttl": "64",
            "ip.proto": "132",
            "ip.checksum": "0x0000b5f8",
            "ip.checksum.status": "2",
            "ip.src": "172.2.21.149",
            "ip.addr.sender": "172.2.21.149",
            "ip.src_host": "172.2.21.149",
            "ip.host.sender": "172.2.21.149",
            "ip.dst": "172.2.21.144",
            "ip.addr.receiver": "172.2.21.144",
            "ip.dst_host": "172.2.21.144",
            "ip.host.receiver": "172.2.21.144"
          },
          "sctp": {
            "sctp.srcport": "3906",
            "sctp.dstport": "3906",
            "sctp.verification_tag": "0x04a0a9d5",
            "sctp.assoc_index": "0",
            "sctp.port.sender": "3906",
            "sctp.port.receiver": "3906",
            "sctp.checksum": "0xfffeb617",
            "sctp.checksum.status": "2",
            "DATA chunk(ordered, complete segment, TSN: 45809598, SID: 13, SSN: 33049, PPID: 0, payload length: 296 bytes)": {
              "sctp.chunk_type": "0",
              "sctp.chunk_type_tree": {
                "sctp.chunk_bit_1": "0",
                "sctp.chunk_bit_2": "0"
              },
              "sctp.chunk_flags": "0x00000003",
              "sctp.chunk_flags_tree": {
                "sctp.data_e_bit": "1",
                "sctp.data_b_bit": "1",
                "sctp.data_u_bit": "0",
                "sctp.data_i_bit": "0"
              },
              "sctp.chunk_length": "312",
              "sctp.data_tsn": "45809598",
              "sctp.data_sid": "0x0000000d",
              "sctp.data_ssn": "33049",
              "sctp.data_payload_proto_id": "0"
            }
          },
          "data": {
            "data.data": "01:00",
            "data.len": "2"
          }
        }
      }
    }
  ]
}

I have tried to look for solution and found this post:

Data Import Fail: Can't merge a non object mapping

First, apologies if this is a simple question, kinda new to this still. Help is greatly appreciated. I am attempting to import zeek/bro logs via the data visualizer and for the most part, most things seem to be going okay. However, I just ran into a problem trying to import conn.log throwing the error below. I thought this type of input was okay to send, am I wrong in that? What can I do to get the data to import? SAMPLE DATA {"_path":"conn","_system_name":"sensor","_write_ts":"2019-07-02T1…

I however am unable to understand their solution on my situation.

I have also found other problems with the automated conversion from pcap to JSOn that one gets from tshark:

1. The file is originally formatted [{obj}] instead of {"_items":[{obj}]}
2. There were fields with identical names which caused a Duplicate field error

[Christian_Dahlqvist]

@zebulyon

"ip.checksum": "0x0000b5f8",
"ip.checksum.status": "2",

When you have dots in field names these gets expanded, which means that ip.checksum need to hold an object containing the status field as well as a string. As every field must have a distinct mapping, this is not allowed and causes an error. You therefore need to change the structure or rename your keys to get away from the dot notation.

See this thread for an example.

[zebulyon]

Ok, thank you very much for the assistance that seems to have solved the problem.

A quick question, what method do you recommend for easily fixing all instances of these errors in the files. I ask as the actual JSON files are quite enormous and it would be very tedious to go and changing them by hand.

Would bash or python be the preferred method, or maybe even creating my own template for tshark to use when converting the files as it might be necessary to convert a lot of files at a somewhat rapid rate.

Once again thank you for the help

[Christian_Dahlqvist]

I think Logstash has a de-dot filter but have not used it.