General Question : Best method for handling JSON files

So I have a collection of PCAP files and I want to upload their content to Kibana.

Now I am fairly certain that I will need to use tshark to convert the files to JSON, which I have done. Though I have a question regarding the conversion process, when converting the pcap files it becomes a multiline format this is the command I use:
tshark -r FILEPATH_TO_PCACP -Tjson > output.json
This is a problem as it seem that Filebeat is only able to handle single line JSON.

But now the big question, how should I go about sending this information to Kibana.
Should I go
Filebeat -> Elasticsearch -> Kibana
Filebeat -> Logstash -> Elasticsearch -> Kibana

I have been testing both methods and I do not get it work properly.

The problem seem to be when the information is sent to Elasticsearch. Because I can see the JSON file being sent properly to Logstash from Filebeat. But when it arraives at Kibana I only get a single Entry saying that

Error decoding JSON: unexpected EOF

Even though as said I can see that Logstash received multiple JSON events.

And when I go Filebeat -> Elasticsearch -> Kibana I get another weird error among them being that the index is becomes ilm-history-1-000001 Even when I set it manually in filebeat.
And secondly the bigger problem is that none of the JSON events arrives at Kibana instead I just get these weird event that that are almost empty.

I'm not familar with tshark so you might have to help me out with that portion. When you pipe it's output to output.json, does output.json contain pretty-printed JSON? Perhaps posting a sample of that file here might help (or if you want to post it to or first and post a link here, that's fine too).



Here is one json entry all of the other ones follow a very similar formatting.

    "_index": "packets-2020-02-06",
    "_type": "pcap_file",
    "_score": null,
    "_source": {
      "layers": {
        "frame": {
          "frame.encap_type": "25",
          "frame.time": "Dec  2, 2019 07:37:53.050476000 CET",
          "frame.offset_shift": "0.000000000",
          "frame.time_epoch": "1575268673.050476000",
          "frame.time_delta": "0.000000000",
          "frame.time_delta_displayed": "0.000000000",
          "frame.time_relative": "0.000000000",
          "frame.number": "1",
          "frame.len": "360",
          "frame.cap_len": "360",
          "frame.marked": "0",
          "frame.ignored": "0",
          "frame.protocols": "sll:ethertype:ip:sctp:data"
        "sll": {
          "sll.pkttype": "0",
          "sll.hatype": "1",
          "sll.halen": "6",
          "sll.src.eth": "fa:16:3e:e6:39:ec",
          "sll.unused": "00:00",
          "sll.etype": "0x00000800"
        "ip": {
          "ip.version": "4",
          "ip.hdr_len": "20",
          "ip.dsfield": "0x00000000",
          "ip.dsfield_tree": {
            "ip.dsfield.dscp": "0",
            "ip.dsfield.ecn": "0"
          "ip.len": "344",
          "": "0x00000000",
          "ip.flags": "0x00004000",
          "ip.flags_tree": {
            "ip.flags.rb": "0",
            "ip.flags.df": "1",
            "": "0",
            "ip.frag_offset": "0"
          "ip.ttl": "64",
          "ip.proto": "132",
          "ip.checksum": "0x0000b5f8",
          "ip.checksum.status": "2",
          "ip.src": "",
          "ip.addr": "",
          "ip.src_host": "",
          "": "",
          "ip.dst": "",
          "ip.addr": "",
          "ip.dst_host": "",
          "": ""
        "sctp": {
          "sctp.srcport": "3906",
          "sctp.dstport": "3906",
          "sctp.verification_tag": "0x04a0a9d5",
          "sctp.assoc_index": "0",
          "sctp.port": "3906",
          "sctp.port": "3906",
          "sctp.checksum": "0xfffeb617",
          "sctp.checksum.status": "2",
          "DATA chunk(ordered, complete segment, TSN: 45809598, SID: 13, SSN: 33049, PPID: 0, payload length: 296 bytes)": {
            "sctp.chunk_type": "0",
            "sctp.chunk_type_tree": {
              "sctp.chunk_bit_1": "0",
              "sctp.chunk_bit_2": "0"
            "sctp.chunk_flags": "0x00000003",
            "sctp.chunk_flags_tree": {
              "sctp.data_e_bit": "1",
              "sctp.data_b_bit": "1",
              "sctp.data_u_bit": "0",
              "sctp.data_i_bit": "0"
            "sctp.chunk_length": "312",
            "sctp.data_tsn": "45809598",
            "sctp.data_sid": "0x0000000d",
            "sctp.data_ssn": "33049",
            "sctp.data_payload_proto_id": "0"
        "data": {
          "": "01:00:01:28:80:00:01:10:01:00:00:33:01:1b:05:fa:00:06:70:ed:00:00:01:07:40:00:00:2c:4f:67:50:72:48:65:49:4d:43:4c:34:43:69:6a:75:3b:33:36:36:37:37:30:3b:33:35:31:37:39:30:32:36:36:30:3b:34:33:00:00:01:02:40:00:00:0c:01:00:00:33:00:00:01:a0:40:00:00:0c:00:00:00:01:00:00:01:08:40:00:00:13:72:6e:39:2e:6d:6d:65:2e:63:6f:6d:00:00:00:01:28:40:00:00:0f:6d:6d:65:2e:63:6f:6d:00:00:00:01:1b:40:00:00:0f:68:73:73:2e:63:6f:6d:00:00:00:01:9f:40:00:00:0c:00:00:00:01:00:00:01:bb:40:00:00:2c:00:00:01:c2:40:00:00:0c:00:00:00:00:00:00:01:bc:40:00:00:17:34:37:39:35:33:36:32:37:33:31:36:31:32:38:39:00:00:00:00:08:40:00:00:17:6b:73:2e:72:64:2e:74:69:65:74:6f:2e:63:6f:6d:00:00:00:01:cd:40:00:00:17:65:75:2e:76:69:73:69:74:69:6e:67:2e:63:6f:6d:00:00:00:01:c8:40:00:00:34:00:00:01:b0:40:00:00:0c:00:00:00:01:00:00:01:b7:40:00:00:0c:00:00:00:01:00:00:01:b5:40:00:00:14:00:00:01:a4:40:00:00:0c:00:00:00:01",
          "data.len": "296"

So this is the output from tshark

You will have to do something about that :slight_smile:

First, it looks like the output in an array (based on the [], start and end)

You should be able to pass everything through jq

Something like this should do the trick

cat tshark_out.json | jq -c .[] > compact_json_per_line.json

That should give you something you can feed into Filebeat


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.