General Question : Best method for handling JSON files

Elastic Stack Beats
1

Hello,
So I have a collection of PCAP files and I want to upload their content to Kibana.

Now I am fairly certain that I will need to use tshark to convert the files to JSON, which I have done. Though I have a question regarding the conversion process, when converting the pcap files it becomes a multiline format this is the command I use:
tshark -r FILEPATH_TO_PCACP -Tjson > output.json
This is a problem as it seem that Filebeat is only able to handle single line JSON.

But now the big question, how should I go about sending this information to Kibana.
Should I go
Filebeat -> Elasticsearch -> Kibana
or
Filebeat -> Logstash -> Elasticsearch -> Kibana

I have been testing both methods and I do not get it work properly.

The problem seem to be when the information is sent to Elasticsearch. Because I can see the JSON file being sent properly to Logstash from Filebeat. But when it arraives at Kibana I only get a single Entry saying that

Error decoding JSON: unexpected EOF

Even though as said I can see that Logstash received multiple JSON events.

And when I go Filebeat -> Elasticsearch -> Kibana I get another weird error among them being that the index is becomes ilm-history-1-000001 Even when I set it manually in filebeat.
And secondly the bigger problem is that none of the JSON events arrives at Kibana instead I just get these weird event that that are almost empty.

2

I'm not familar with tshark so you might have to help me out with that portion. When you pipe it's output to output.json, does output.json contain pretty-printed JSON? Perhaps posting a sample of that file here might help (or if you want to post it to pastebin.com or gist.github.com first and post a link here, that's fine too).

Thanks,

Shaunak

3

Here is one json entry all of the other ones follow a very similar formatting.

[
  {
    "_index": "packets-2020-02-06",
    "_type": "pcap_file",
    "_score": null,
    "_source": {
      "layers": {
        "frame": {
          "frame.encap_type": "25",
          "frame.time": "Dec  2, 2019 07:37:53.050476000 CET",
          "frame.offset_shift": "0.000000000",
          "frame.time_epoch": "1575268673.050476000",
          "frame.time_delta": "0.000000000",
          "frame.time_delta_displayed": "0.000000000",
          "frame.time_relative": "0.000000000",
          "frame.number": "1",
          "frame.len": "360",
          "frame.cap_len": "360",
          "frame.marked": "0",
          "frame.ignored": "0",
          "frame.protocols": "sll:ethertype:ip:sctp:data"
        },
        "sll": {
          "sll.pkttype": "0",
          "sll.hatype": "1",
          "sll.halen": "6",
          "sll.src.eth": "fa:16:3e:e6:39:ec",
          "sll.unused": "00:00",
          "sll.etype": "0x00000800"
        },
        "ip": {
          "ip.version": "4",
          "ip.hdr_len": "20",
          "ip.dsfield": "0x00000000",
          "ip.dsfield_tree": {
            "ip.dsfield.dscp": "0",
            "ip.dsfield.ecn": "0"
          },
          "ip.len": "344",
          "ip.id": "0x00000000",
          "ip.flags": "0x00004000",
          "ip.flags_tree": {
            "ip.flags.rb": "0",
            "ip.flags.df": "1",
            "ip.flags.mf": "0",
            "ip.frag_offset": "0"
          },
          "ip.ttl": "64",
          "ip.proto": "132",
          "ip.checksum": "0x0000b5f8",
          "ip.checksum.status": "2",
          "ip.src": "172.2.21.149",
          "ip.addr": "172.2.21.149",
          "ip.src_host": "172.2.21.149",
          "ip.host": "172.2.21.149",
          "ip.dst": "172.2.21.144",
          "ip.addr": "172.2.21.144",
          "ip.dst_host": "172.2.21.144",
          "ip.host": "172.2.21.144"
        },
        "sctp": {
          "sctp.srcport": "3906",
          "sctp.dstport": "3906",
          "sctp.verification_tag": "0x04a0a9d5",
          "sctp.assoc_index": "0",
          "sctp.port": "3906",
          "sctp.port": "3906",
          "sctp.checksum": "0xfffeb617",
          "sctp.checksum.status": "2",
          "DATA chunk(ordered, complete segment, TSN: 45809598, SID: 13, SSN: 33049, PPID: 0, payload length: 296 bytes)": {
            "sctp.chunk_type": "0",
            "sctp.chunk_type_tree": {
              "sctp.chunk_bit_1": "0",
              "sctp.chunk_bit_2": "0"
            },
            "sctp.chunk_flags": "0x00000003",
            "sctp.chunk_flags_tree": {
              "sctp.data_e_bit": "1",
              "sctp.data_b_bit": "1",
              "sctp.data_u_bit": "0",
              "sctp.data_i_bit": "0"
            },
            "sctp.chunk_length": "312",
            "sctp.data_tsn": "45809598",
            "sctp.data_sid": "0x0000000d",
            "sctp.data_ssn": "33049",
            "sctp.data_payload_proto_id": "0"
          }
        },
        "data": {
          "data.data": "01:00:01:28:80:00:01:10:01:00:00:33:01:1b:05:fa:00:06:70:ed:00:00:01:07:40:00:00:2c:4f:67:50:72:48:65:49:4d:43:4c:34:43:69:6a:75:3b:33:36:36:37:37:30:3b:33:35:31:37:39:30:32:36:36:30:3b:34:33:00:00:01:02:40:00:00:0c:01:00:00:33:00:00:01:a0:40:00:00:0c:00:00:00:01:00:00:01:08:40:00:00:13:72:6e:39:2e:6d:6d:65:2e:63:6f:6d:00:00:00:01:28:40:00:00:0f:6d:6d:65:2e:63:6f:6d:00:00:00:01:1b:40:00:00:0f:68:73:73:2e:63:6f:6d:00:00:00:01:9f:40:00:00:0c:00:00:00:01:00:00:01:bb:40:00:00:2c:00:00:01:c2:40:00:00:0c:00:00:00:00:00:00:01:bc:40:00:00:17:34:37:39:35:33:36:32:37:33:31:36:31:32:38:39:00:00:00:00:08:40:00:00:17:6b:73:2e:72:64:2e:74:69:65:74:6f:2e:63:6f:6d:00:00:00:01:cd:40:00:00:17:65:75:2e:76:69:73:69:74:69:6e:67:2e:63:6f:6d:00:00:00:01:c8:40:00:00:34:00:00:01:b0:40:00:00:0c:00:00:00:01:00:00:01:b7:40:00:00:0c:00:00:00:01:00:00:01:b5:40:00:00:14:00:00:01:a4:40:00:00:0c:00:00:00:01",
          "data.len": "296"
        }
      }
    }
  }
 ]

So this is the output from tshark

4

You will have to do something about that :slight_smile:

First, it looks like the output in an array (based on the [], start and end)

You should be able to pass everything through jq

Something like this should do the trick

cat tshark_out.json | jq -c .[] > compact_json_per_line.json

That should give you something you can feed into Filebeat

2 Likes
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.