Wireshark json file cannot display in separate fields in Elasticsearch

I have investigated for a month and still cannot find a solution to structure the json data captured by Wireshark. I am using FileBeat in Elastic to import json data. However, all the json data are packed to the "message" fields.

What I need to do is that I have to search the fields in that json (e.g. ip_ip_dst_host, tcp_tcp_dstport, timestamp, tcp_tcp_payload)

The below is the result showing in Elastic:

I know FileBeat has some default fields, but for me, they are useless. I only take care of the fields in "message"

And below is the FileBeat configuraton:

filebeat.inputs:

- type: log

  enabled: true

  paths:
    - C:\wireshark\*.json
  
  json.keys_under_root: true
  json.add_error_key: true


filebeat.config.modules:

  path: ${path.config}/modules.d/*.yml

  reload.enabled: false


setup.template.settings:
  index.number_of_shards: 1

setup.kibana:

  host: "http://localhost:5601"


output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["http://localhost:9200"]
  username: "xxx"
  password: "xxx" 
  pipeline: message_process

  #index: "testserver01-%{+yyyy-MM-dd}"
  
  indices:
  - index: "packet-%{[agent.version]}-%{+yyyy.MM.dd}"
  
  template.enabled: true

setup.template.name: "packet"
setup.template.pattern: packet-*"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  #- drop_event:
   #  when:
   #    equals:
    #     index._type: "pcap_file"
  - decode_json_fields:
      fields: ["message"]
      process_array: true
      max_depth: 6
      target: ""
      overwrite_keys: true
      add_error_key: true
      when:
        regexp:
          message: '^\{'

My json file is like that:

{"timestamp":"1639101979659","layers":{"frame":{"frame_frame_interface_id":"0","frame_frame_encap_type":"1","frame_frame_time":"2021-12-10T02:06:19.659398000Z","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1639101979.659398000","frame_frame_time_delta":"0.000000000","frame_frame_time_delta_displayed":"0.000000000","frame_frame_time_relative":"0.000000000","frame_frame_number":"1","frame_frame_len":"112","frame_frame_cap_len":"112","frame_frame_marked":false,"frame_frame_ignored":false,"frame_frame_protocols":"eth:ethertype:ip:tcp:tls"},"eth":{"eth_eth_dst":"ee:ee:ee:ee:ee:ee","eth_eth_dst_resolved":"ee:ee:ee:ee:ee:ee","eth_eth_dst_oui":"15658734","eth_eth_addr":"ee:ee:ee:ee:ee:ee","eth_eth_addr_resolved":"ee:ee:ee:ee:ee:ee","eth_eth_addr_oui":"15658734","eth_eth_dst_lg":true,"eth_eth_lg":true,"eth_eth_dst_ig":false,"eth_eth_ig":false,"eth_eth_src_oui":"12503387","eth_eth_addr_oui":"12503387","eth_eth_src_lg":true,"eth_eth_lg":true,"eth_eth_src_ig":false,"eth_eth_ig":false,"eth_eth_type":"0x0800"},"ip":{"ip_ip_version":"4","ip_ip_hdr_len":"20","ip_ip_dsfield":"0x00","ip_ip_dsfield_dscp":"0","ip_ip_dsfield_ecn":"0","ip_ip_len":"98","ip_ip_id":"0xa688","ip_ip_flags":"0x40","ip_ip_flags_rb":false,"ip_ip_flags_df":true,"ip_ip_flags_mf":false,"ip_ip_frag_offset":"0","ip_ip_ttl":"64","ip_ip_proto":"6","ip_ip_checksum":"0x65e1","ip_ip_checksum_status":"2","ip_ip_src":"10.200.25.98","ip_ip_addr":["10.200.25.98","10.2.0.1"],"ip_ip_src_host":"10.200.25.98","ip_ip_host":["10.200.25.98","10.2.0.1"],"ip_ip_dst":"10.2.0.1","ip_ip_dst_host":"10.2.0.1"},"tcp":{"tcp_tcp_srcport":"51344","tcp_tcp_dstport":"443","tcp_tcp_port":["51344","443"],"tcp_tcp_stream":"0","tcp_tcp_completeness":"0","tcp_tcp_len":"46","tcp_tcp_seq":"1","tcp_tcp_seq_raw":"278114923","tcp_tcp_nxtseq":"47","tcp_tcp_ack":"1","tcp_tcp_ack_raw":"212180543","tcp_tcp_hdr_len":"32","tcp_tcp_flags":"0x0018","tcp_tcp_flags_res":false,"tcp_tcp_flags_ns":false,"tcp_tcp_flags_cwr":false,"tcp_tcp_flags_ecn":false,"tcp_tcp_flags_urg":false,"tcp_tcp_flags_ack":true,"tcp_tcp_flags_push":true,"tcp_tcp_flags_reset":false,"tcp_tcp_flags_syn":false,"tcp_tcp_flags_fin":false,"tcp_tcp_flags_str":"┬╖┬╖┬╖┬╖┬╖┬╖┬╖AP┬╖┬╖┬╖","tcp_tcp_window_size_value":"1424","tcp_tcp_window_size":"1424","tcp_tcp_window_size_scalefactor":"-1","tcp_tcp_checksum":"0x2e81","tcp_tcp_checksum_status":"2","tcp_tcp_urgent_pointer":"0","tcp_options_nop":["01","01"],"tcp_tcp_option_kind":["1","1"],"tcp_tcp_option_kind":"8","tcp_tcp_option_len":"10","tcp_tcp_options_timestamp_tsval":"3018575007","tcp_tcp_options_timestamp_tsecr":"3018573006","text":"Timestamps","tcp_tcp_time_relative":"0.000000000","tcp_tcp_time_delta":"0.000000000","tcp_tcp_analysis":null,"tcp_tcp_analysis_bytes_in_flight":"46","tcp_tcp_analysis_push_bytes_sent":"46","tcp_tcp_payload":"17:03:03:00:29:00:00:00:00:00:5b:ed:22:56:ab:29:f8:ad:4d:f2:5a:65:f0:17:15:46:3c:ae:3e:30:8e:ff:16:46:46:42:9a:56:e5:6b:85:53:46:6a:f9:ee"},"tls":{"tls_tls_record":null,"tls_tls_record_content_type":"23","tls_tls_record_version":"0x0303","tls_tls_record_length":"41","tls_tls_app_data":"00:00:00:00:00:5b:ed:22:56:ab:29:f8:ad:4d:f2:5a:65:f0:17:15:46:3c:ae:3e:30:8e:ff:16:46:46:42:9a:56:e5:6b:85:53:46:6a:f9:ee","tls_tls_app_data_proto":"http-over-tls"}}}
{"timestamp":"1639101979662","layers":{"frame":{"frame_frame_interface_id":"0","frame_frame_encap_type":"1","frame_frame_time":"2021-12-10T02:06:19.662332000Z","frame_frame_offset_shift":"0.000000000","frame_frame_time_epoch":"1639101979.662332000","frame_frame_time_delta":"0.002934000","frame_frame_time_delta_displayed":"0.002934000","frame_frame_time_relative":"0.002934000","frame_frame_number":"2","frame_frame_len":"132","frame_frame_cap_len":"132","frame_frame_marked":false,"frame_frame_ignored":false,"frame_frame_protocols":"eth:ethertype:ip:tcp:tls"},"eth":{"eth_eth_dst_oui":"12503387","eth_eth_addr_oui":"12503387","eth_eth_dst_lg":true,"eth_eth_lg":true,"eth_eth_dst_ig":false,"eth_eth_ig":false,"eth_eth_src":"ee:ee:ee:ee:ee:ee","eth_eth_src_resolved":"ee:ee:ee:ee:ee:ee","eth_eth_src_oui":"15658734","eth_eth_addr":"ee:ee:ee:ee:ee:ee","eth_eth_addr_resolved":"ee:ee:ee:ee:ee:ee","eth_eth_addr_oui":"15658734","eth_eth_src_lg":true,"eth_eth_lg":true,"eth_eth_src_ig":false,"eth_eth_ig":false,"eth_eth_type":"0x0800"},"ip":{"ip_ip_version":"4","ip_ip_hdr_len":"20","ip_ip_dsfield":"0x00","ip_ip_dsfield_dscp":"0","ip_ip_dsfield_ecn":"0","ip_ip_len":"118","ip_ip_id":"0xa750","ip_ip_flags":"0x40","ip_ip_flags_rb":false,"ip_ip_flags_df":true,"ip_ip_flags_mf":false,"ip_ip_frag_offset":"0","ip_ip_ttl":"64","ip_ip_proto":"6","ip_ip_checksum":"0x6505","ip_ip_checksum_status":"2","ip_ip_src":"10.2.0.1","ip_ip_addr":["10.2.0.1","10.200.25.98"],"ip_ip_src_host":"10.2.0.1","ip_ip_host":["10.2.0.1","10.200.25.98"],"ip_ip_dst":"10.200.25.98","ip_ip_dst_host":"10.200.25.98"},"tcp":{"tcp_tcp_srcport":"443","tcp_tcp_dstport":"51344","tcp_tcp_port":["443","51344"],"tcp_tcp_stream":"0","tcp_tcp_completeness":"0","tcp_tcp_len":"66","tcp_tcp_seq":"1","tcp_tcp_seq_raw":"212180543","tcp_tcp_nxtseq":"67","tcp_tcp_ack":"47","tcp_tcp_ack_raw":"278114969","tcp_tcp_hdr_len":"32","tcp_tcp_flags":"0x0018","tcp_tcp_flags_res":false,"tcp_tcp_flags_ns":false,"tcp_tcp_flags_cwr":false,"tcp_tcp_flags_ecn":false,"tcp_tcp_flags_urg":false,"tcp_tcp_flags_ack":true,"tcp_tcp_flags_push":true,"tcp_tcp_flags_reset":false,"tcp_tcp_flags_syn":false,"tcp_tcp_flags_fin":false,"tcp_tcp_flags_str":"┬╖┬╖┬╖┬╖┬╖┬╖┬╖AP┬╖┬╖┬╖","tcp_tcp_window_size_value":"1432","tcp_tcp_window_size":"1432","tcp_tcp_window_size_scalefactor":"-1","tcp_tcp_checksum":"0x2e95","tcp_tcp_checksum_status":"2","tcp_tcp_urgent_pointer":"0","tcp_options_nop":["01","01"],"tcp_tcp_option_kind":["1","1"],"tcp_tcp_option_kind":"8","tcp_tcp_option_len":"10","tcp_tcp_options_timestamp_tsval":"3018575010","tcp_tcp_options_timestamp_tsecr":"3018575007","text":"Timestamps","tcp_tcp_time_relative":"0.002934000","tcp_tcp_time_delta":"0.002934000","tcp_tcp_analysis":null,"tcp_tcp_analysis_acks_frame":"1","tcp_tcp_analysis_ack_rtt":"0.002934000","tcp_tcp_analysis_bytes_in_flight":"66","tcp_tcp_analysis_push_bytes_sent":"66","tcp_tcp_payload":"17:03:03:00:3d:00:00:00:00:00:8a:0b:87:f3:5d:2a:90:58:80:23:e8:75:b4:44:be:41:70:62:66:ae:d7:95:d2:de:3b:b2:5e:35:63:7e:b6:27:a9:37:45:bc:4a:b7:a1:4c:47:fa:37:58:14:83:e8:9b:ce:b7:5f:2a:8a:ce:6d:41"},"tls":{"tls_tls_record":null,"tls_tls_record_content_type":"23","tls_tls_record_version":"0x0303","tls_tls_record_length":"61","tls_tls_app_data":"00:00:00:00:00:8a:0b:87:f3:5d:2a:90:58:80:23:e8:75:b4:44:be:41:70:62:66:ae:d7:95:d2:de:3b:b2:5e:35:63:7e:b6:27:a9:37:45:bc:4a:b7:a1:4c:47:fa:37:58:14:83:e8:9b:ce:b7:5f:2a:8a:ce:6d:41","tls_tls_app_data_proto":"http-over-tls"}}}

What does your message_process pipeline look like.

Please post it.

[
  {
    "json": {
      "field": "message",
      "tag": "1",
      "ignore_failure": true
    }
  }
]

The above codes can skip the following line between each .json:

{"index":{"_index":"packets-2021-12-10","_type":"doc"}}

Hi @lauyatkin Lets back up a bit and take some steps.

How did this data get created it looks like it is already in a _bulk loading format see the bottom.

But here is a start I did with filebeat

I took your 2 lines above and put them in a file wireshark.json

Here is my complete filebeat.yml
Since the file is ndjson that is all you need
I know there is other parsing you probably want to do but you should get this to work first.
Of course you will probably want to create the correct mappings and probably set the time stamp but why don't you try to get this working first,

[/quote]

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/wireshark/wireshark.json
  json.keys_under_root: true
  json.add_error_key: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
  hosts: ["localhost:9200"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Here is a sample document in Elasticsearch completely parsed.

GET filebeat-7.15.2-2021.12.10-000001/_search

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 8,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-7.15.2-2021.12.10-000001",
        "_type" : "_doc",
        "_id" : "GRlzon0BkMxR-1fZyUvP",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-12-10T03:47:13.032Z",
          "timestamp" : "1639101979659",
          "log" : {
            "offset" : 0,
            "file" : {
              "path" : "/Users/sbrown/workspace/sample-data/discuss/wireshark/wireshark.json"
            }
          },
          "input" : {
            "type" : "log"
          },
          "ecs" : {
            "version" : "1.11.0"
          },
          "host" : {
            "id" : "9E46F076-B7F1-53AA-921B-C2F983746B79",
            "ip" : [
              "fe80::aede:48ff:fe00:1122",
              "fe80::18d9:1a40:b8b2:bea0",
              "192.168.2.107",
              "fe80::d56c:87be:9b2:383d",
              "fe80::39ce:303b:f110:c7ac"
            ],
            "mac" : [
              "ac:de:48:00:11:22",
              "7e:52:30:9c:ef:e0",
              "5c:52:30:9c:ef:e0",
              "8e:50:33:07:fa:e5",
              "82:b2:58:49:30:05",
              "82:b2:58:49:30:04",
              "82:b2:58:49:30:01",
              "82:b2:58:49:30:00",
              "a0:ce:c8:51:95:38",
              "82:b2:58:49:30:01"
            ],
            "hostname" : "hyperion",
            "architecture" : "x86_64",
            "os" : {
              "version" : "10.16",
              "family" : "darwin",
              "name" : "Mac OS X",
              "kernel" : "20.6.0",
              "build" : "20G224",
              "type" : "macos",
              "platform" : "darwin"
            },
            "name" : "hyperion"
          },
          "agent" : {
            "hostname" : "hyperion",
            "ephemeral_id" : "04bb1eb3-05b1-4435-95a7-db76224b7b26",
            "id" : "9d17417c-1db0-49b3-99b1-beea4649a61d",
            "name" : "hyperion",
            "type" : "filebeat",
            "version" : "7.15.2"
          },
          "container" : {
            "id" : "discuss"
          },
          "layers" : {
            "eth" : {
              "eth_eth_dst_oui" : "15658734",
              "eth_eth_dst_resolved" : "ee:ee:ee:ee:ee:ee",
              "eth_eth_dst_ig" : false,
              "eth_eth_addr_oui" : "12503387",
              "eth_eth_addr" : "ee:ee:ee:ee:ee:ee",
              "eth_eth_addr_resolved" : "ee:ee:ee:ee:ee:ee",
              "eth_eth_dst" : "ee:ee:ee:ee:ee:ee",
              "eth_eth_lg" : true,
              "eth_eth_src_oui" : "12503387",
              "eth_eth_dst_lg" : true,
              "eth_eth_src_lg" : true,
              "eth_eth_src_ig" : false,
              "eth_eth_ig" : false,
              "eth_eth_type" : "0x0800"
            },
            "ip" : {
              "ip_ip_dsfield" : "0x00",
              "ip_ip_flags_df" : true,
              "ip_ip_hdr_len" : "20",
              "ip_ip_ttl" : "64",
              "ip_ip_flags_rb" : false,
              "ip_ip_dst_host" : "10.2.0.1",
              "ip_ip_len" : "98",
              "ip_ip_src" : "10.200.25.98",
              "ip_ip_flags_mf" : false,
              "ip_ip_addr" : [
                "10.200.25.98",
                "10.2.0.1"
              ],
              "ip_ip_src_host" : "10.200.25.98",
              "ip_ip_dst" : "10.2.0.1",
              "ip_ip_id" : "0xa688",
              "ip_ip_checksum_status" : "2",
              "ip_ip_flags" : "0x40",
              "ip_ip_proto" : "6",
              "ip_ip_version" : "4",
              "ip_ip_dsfield_dscp" : "0",
              "ip_ip_dsfield_ecn" : "0",
              "ip_ip_frag_offset" : "0",
              "ip_ip_checksum" : "0x65e1",
              "ip_ip_host" : [
                "10.200.25.98",
                "10.2.0.1"
              ]
            },
            "tcp" : {
              "tcp_tcp_flags_reset" : false,
              "tcp_tcp_checksum" : "0x2e81",
              "tcp_tcp_ack_raw" : "212180543",
              "tcp_tcp_flags_fin" : false,
              "tcp_tcp_seq_raw" : "278114923",
              "tcp_tcp_flags" : "0x0018",
              "tcp_tcp_payload" : "17:03:03:00:29:00:00:00:00:00:5b:ed:22:56:ab:29:f8:ad:4d:f2:5a:65:f0:17:15:46:3c:ae:3e:30:8e:ff:16:46:46:42:9a:56:e5:6b:85:53:46:6a:f9:ee",
              "tcp_tcp_completeness" : "0",
              "tcp_tcp_flags_cwr" : false,
              "tcp_tcp_flags_urg" : false,
              "tcp_tcp_option_kind" : "8",
              "tcp_tcp_checksum_status" : "2",
              "tcp_tcp_urgent_pointer" : "0",
              "tcp_tcp_hdr_len" : "32",
              "tcp_tcp_len" : "46",
              "tcp_tcp_nxtseq" : "47",
              "tcp_tcp_flags_ack" : true,
              "tcp_tcp_options_timestamp_tsval" : "3018575007",
              "tcp_tcp_srcport" : "51344",
              "text" : "Timestamps",
              "tcp_tcp_option_len" : "10",
              "tcp_tcp_options_timestamp_tsecr" : "3018573006",
              "tcp_options_nop" : [
                "01",
                "01"
              ],
              "tcp_tcp_analysis_bytes_in_flight" : "46",
              "tcp_tcp_flags_res" : false,
              "tcp_tcp_seq" : "1",
              "tcp_tcp_window_size_value" : "1424",
              "tcp_tcp_analysis_push_bytes_sent" : "46",
              "tcp_tcp_dstport" : "443",
              "tcp_tcp_stream" : "0",
              "tcp_tcp_window_size" : "1424",
              "tcp_tcp_port" : [
                "51344",
                "443"
              ],
              "tcp_tcp_flags_str" : "┬╖┬╖┬╖┬╖┬╖┬╖┬╖AP┬╖┬╖┬╖",
              "tcp_tcp_flags_syn" : false,
              "tcp_tcp_time_delta" : "0.000000000",
              "tcp_tcp_time_relative" : "0.000000000",
              "tcp_tcp_flags_ns" : false,
              "tcp_tcp_flags_push" : true,
              "tcp_tcp_window_size_scalefactor" : "-1",
              "tcp_tcp_ack" : "1",
              "tcp_tcp_flags_ecn" : false
            },
            "tls" : {
              "tls_tls_app_data" : "00:00:00:00:00:5b:ed:22:56:ab:29:f8:ad:4d:f2:5a:65:f0:17:15:46:3c:ae:3e:30:8e:ff:16:46:46:42:9a:56:e5:6b:85:53:46:6a:f9:ee",
              "tls_tls_app_data_proto" : "http-over-tls",
              "tls_tls_record_content_type" : "23",
              "tls_tls_record_version" : "0x0303",
              "tls_tls_record_length" : "41"
            },
            "frame" : {
              "frame_frame_time_relative" : "0.000000000",
              "frame_frame_marked" : false,
              "frame_frame_protocols" : "eth:ethertype:ip:tcp:tls",
              "frame_frame_number" : "1",
              "frame_frame_interface_id" : "0",
              "frame_frame_ignored" : false,
              "frame_frame_encap_type" : "1",
              "frame_frame_time_delta_displayed" : "0.000000000",
              "frame_frame_time" : "2021-12-10T02:06:19.659398000Z",
              "frame_frame_time_epoch" : "1639101979.659398000",
              "frame_frame_len" : "112",
              "frame_frame_cap_len" : "112",
              "frame_frame_offset_shift" : "0.000000000",
              "frame_frame_time_delta" : "0.000000000"
            }
          }
        }
      }

I am not not sure why you are doing some of the other things.
Like this...

How was this data created ... it looks like it was created to use the _bulk Insert API.

Those lines / format are used by _bulk API to bulk insert the data directly into Elasticsearch ... See Here
That would simply insert the document in the correct Index etc...
If it is _bulk data format you can use the __bulk loading API.

Let me explain in a little bit details.

First, I don't know any _bulk Insert API that you are talking about.

The .json file is created with tshark at the first begining by typing the following command:

.\tshark.exe -i rpcap://x.x.x.x:6666/cali18dd3a4d0a8 -T ek -a duration:10 > C:\wireshark\20211210_packets_97.json

Then the output will be like that: (there will be a useless line in between for each packets)

{"index":{"_index":"packets-2021-12-10","_type":"doc"}}
{"timestamp"......}  //<--that is the only line displaying in Elastic after adding the pipeline
{"index":{"_index":"packets-2021-12-10","_type":"doc"}}
{"timestamp"......}  //<--that is the only line displaying in Elastic after adding the pipeline
{"index":{"_index":"packets-2021-12-10","_type":"doc"}}

The above is what you are asking. The second point I would like to ask is that I cannot parse the document in Elasticsearch like the format / pattern you are showing. May I know if I have anything missing or mis-configurate

I am not overly familiar with Wireshark.

Perhaps you should read this .. it is a little old but it may help you there are been a few changes.

It refers to the _bulk API which I gave you the link to above... note the article says there are several ways to load the data.

That said lets focus on filebeat for now.

OK So with this with this pattern

{"index":{"_index":"packets-2021-12-10","_type":"doc"}}
{"timestamp":"1639101979659","layers":{"frame":{"frame_frame_interface_id":"0"...
{"index":{"_index":"packets-2021-12-10","_type":"doc"}}
{"timestamp":"1639101979662","layers":{"frame":{"frame_frame_interface_id":"0",....

You should create a template like it says... that may take a little work

There is a change with respect to the document _type is always doc now call removal of mapping types.

So Here I updated and created a valid template for you which sets the mappings. I am not sure if it is perfect or not, it is based on the article above.

# NOTE FIXED!!!

DELETE _template/packets

PUT _template/packets
{
  "index_patterns": "packets-*",
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date"
      },
      "layers": {
        "properties": {
          "frame": {
            "properties": {
              "frame_frame_len": {
                "type": "long"
              },
              "frame_frame_protocols": {
                "type": "keyword"
              }
            }
          },
          "ip": {
            "properties": {
              "ip_ip_src": {
                "type": "ip"
              },
              "ip_ip_dst": {
                "type": "ip"
              }
            }
          },
          "udp": {
            "properties": {
              "udp_udp_srcport": {
                "type": "integer"
              },
              "udp_udp_dstport": {
                "type": "integer"
              }
            }
          }
        }
      }
    }
  }
}

Here is my entire filebeat.yml that works... it only ingests the 2nd, 4th lines etc.,,,
There are a couple tricky settings with the setup stuff but this works some point you might want to adjust.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/wireshark/wireshark.json
  json.keys_under_root: true
  json.add_error_key: true

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.ilm.enabled: false
setup.template.enabled: false
setup.template.settings.index.number_of_shards: 1

setup.kibana:

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "packets-%{[agent.version]}-%{+yyyy.MM.dd}"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - drop_event:
      when:
        equals:
          index._type: "doc"

Then run filebeat. If you ran it already clean out the database to reload your file.

from the filebeat directory

filebeat-7.15.2-darwin-x86_64$ rm -fr data
filebeat-7.15.2-darwin-x86_64$ ./filebeat -e

And the results

Take a look

GET packets-7.15.2-2021.12.10/_search

May I know where to input the RESTful code you mentioned above??

DELETE _template/packets

PUT _template/packets

In the Kibana - Dev Tools

I have type this to Dev Tools and the result is success:

#! Legacy index templates are deprecated in favor of composable templates.
#! Deprecated field [template] used, replaced by [index_patterns]
{
  "acknowledged" : true
}

Then I go to Kibana > Index Patterns to add a Index patterns named "packets-*)
However, I re-captured the packets, nothing was shown in the discover page:

Here is my setting of Filebeats:

# ============================== Filebeat inputs ===============================

filebeat.inputs:



- type: log
  enabled: true
  paths:
    - C:\wireshark\*.json  
  json.keys_under_root: true
  json.add_error_key: true

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

# ======================= Elasticsearch template setting =======================

setup.ilm.enabled: false
setup.template.enabled: false
setup.template.settings.index.number_of_shards: 1

# =================================== Kibana ===================================
setup.kibana:
  host: "http://localhost:5601"

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:

  hosts: ["http://localhost:9200"]
  username: "admin"
  password: "P@ssw0rd" 
  #pipeline: message_process
 
  index: "packets-%{[agent.version]}-%{+yyyy.MM.dd}"

setup.template.name: "packets"
setup.template.pattern: packets-*"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - drop_event:
     when:
       equals:
         index._type: "doc"
  #- decode_json_fields:
  #    fields: ["message"]
  #    process_array: true
  #    max_depth: 6
  #    target: ""
  #   overwrite_keys: true
  #    add_error_key: true
  #    when:
  #      regexp:
  #        message: '^\{'

I guess I am still missing something.

Oh, I find some message here:

That above is incorrect I would try to delete the template and PUT it back.

DELETE _template/packets

Make sure it is gone... Then PUT is back

I'm not sure where that came from but that is not the same as what I posted above.

You can delete that from the Kibana UI are if you wanted.

Also in discover instead of last 15 minutes select a larger time frame like last 30 days just to make sure you're not missing it.

I have Delete all the template using DELETE _template/packet* and put it back.

After adding this, there will be a template added in stack Management > Index Management > Index Templates > Legacy index templates

When I Edit the template by pressing "Next" several times, there will be an error at the middle step and at the end:

The mappings for this template uses multiple types, which are not supported.

The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true.

Just like the screenshot I showing in previous reply

Hmmm let me look perhaps I had a typo... why are you trying to edit the template?

@lauyatkin Apologies Darn I had a typo!!! so sorry for the wasted time!

Try this... I fixed it above as well ... I actually had 2 issues... argg!

DELETE _template/packets

PUT _template/packets
{
  "index_patterns": "packets-*",
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date"
      },
      "layers": {
        "properties": {
          "frame": {
            "properties": {
              "frame_frame_len": {
                "type": "long"
              },
              "frame_frame_protocols": {
                "type": "keyword"
              }
            }
          },
          "ip": {
            "properties": {
              "ip_ip_src": {
                "type": "ip"
              },
              "ip_ip_dst": {
                "type": "ip"
              }
            }
          },
          "udp": {
            "properties": {
              "udp_udp_srcport": {
                "type": "integer"
              },
              "udp_udp_dstport": {
                "type": "integer"
              }
            }
          }
        }
      }
    }
  }
}

This works in the editor as well...

I ran filebeat it loaded the data...

Also I see this

- type: log
  enabled: true
  paths:
    - C:\wireshark\*.json  

See Here...

Wrap paths in single quotation marks

Windows paths in particular sometimes contain spaces or characters, such as drive letters or triple dots, that may be misinterpreted by the YAML parser.

To avoid this problem, it’s a good idea to wrap paths in single quotation marks.

Should be

- type: log
  enabled: true
  paths:
    - 'C:\wireshark\*.json' 

No problem. At least you are helping me.

However, don't know I am unlucky or not. I followed all of your steps, still nothing showing in "Discover".........

I have checked and confirmed the index is successfully added. But it is nothing in "Discover". I discover that the newly added "packets" has something different from the default filebeat index. I am not sure it is related or not.

The data should be here

You have data! Your almost there...

Did you create an index pattern?

Kibana - Stack Management - Index Pattern

Discover needs an index pattern.

How old is the data that you ingested ? Did you looks at say last 30 days?

Also go to Dev Tools and look at your data.

Go to Dev Tools and run this show me the results

GET packets-*/_search

I have created an index pattern named "packets-*", but still nothing show even I change to Last 30 days.

Here is the result of the above codes:
{
  "took" : 14,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 280,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "packets-7.15.2-2021.12.16",
        "_type" : "_doc",
        "_id" : "-eprwn0B3kIeJbGgtZ19",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-12-16T08:46:17.303Z",
          "input" : {
            "type" : "log"
          },
          "ecs" : {
            "version" : "1.11.0"
          },
          "host" : {
            "mac" : [
              "38:f3:ab:59:37:d8",
              "0a:00:27:00:00:09",
              "c0:3c:59:2a:0c:11",
              "c2:3c:59:2a:0c:10",
              "c0:3c:59:2a:0c:10",
              "00:09:0f:fe:00:01",
              "00:50:56:c0:00:01",
              "00:50:56:c0:00:08",
              "c0:3c:59:2a:0c:14"
            ],
            "hostname" : "Jacky-PC",
            "architecture" : "x86_64",
            "os" : {
              "kernel" : "10.0.19041.1348 (WinBuild.160101.0800)",
              "build" : "19043.1348",
              "type" : "windows",
              "platform" : "windows",
              "version" : "10.0",
              "family" : "windows",
              "name" : "Windows 10 Home"
            },
            "id" : "c9d7195b-3b45-46c3-b16c-22d0ef14a76e",
            "name" : "Jacky-PC",
            "ip" : [
              "fe80::65bb:bf84:739b:d967",
              "169.254.217.103",
              "fe80::1dd5:7b01:f06c:e191",
              "192.168.56.1",
              "fe80::98ac:60d7:edfd:d821",
              "169.254.216.33",
              "fe80::25ec:6c7f:a77b:5b51",
              "169.254.91.81",
              "fe80::a95e:de7:b656:10eb",
              "192.168.20.109",
              "fe80::d84d:3e67:d544:71f5",
              "169.254.113.245",
              "fe80::c448:3670:9e0b:7307",
              "192.168.37.1",
              "fe80::448e:5dd1:27d5:159a",
              "192.168.124.1",
              "fe80::4802:eabc:3888:dcb9",
              "169.254.220.185"
            ]
          },
          "agent" : {
            "version" : "7.15.2",
            "hostname" : "Jacky-PC",
            "ephemeral_id" : "3aca6811-4c80-43d4-808e-8d54ddc239b6",
            "id" : "c4cd5ddc-9d41-4bf0-878c-d70cc23fe629",
            "name" : "Jacky-PC",
            "type" : "filebeat"
          },
          "log" : {
            "offset" : 221137,
            "file" : {
              "path" : """C:\wireshark\20211216_packets_129.json"""
            }
          },
          "error" : {
            "message" : """Error decoding JSON: invalid character '\x00' looking for beginning of value""",
            "type" : "json"
          },
          "message" : """

I don't know why the result cannot be shown correctly, let me screen capture for you:

This is the result look like

Ohh looks like you have Unicode ....That explains it!

You need to figure out which encoding it is or set your Wireshark to write normal ASCII text.

It's probably UTF-8 or something.

You will need to set

encoding : utf-8

Or the proper encoding.

Also these documents should show up in discover, you should see them with timestamp just the message is not decoded.

Again not a Wireshark expert ... But that is what it looks like to me.