Getting JSON data out of message field imported with filebeat

Elastic Stack Beats
,
1

Hi everybody,

I have a problem with indexing filebeat output generated from a JSON file. This JSON file has been created by converting a PCAP file and using jq to make it not pretty. After using filebeat this file has been imported to Elastic. Which created an output where all the imported data is visualized in the column message. As soon below.

{"_index":"packets-2015-07-25","_type":"doc","_score":null,"_source":{"layers":{"frame":{"frame.encap_type":"1","frame.time":"Jul 25, 2015 21:43:13.284118000 West-Europa (zomertijd)","frame.offset_shift":"0.000000000","frame.time_epoch":"1437853393.284118000","frame.time_delta":"0.001450000","frame.time_delta_displayed":"0.001450000","frame.time_relative":"13896.477788000","frame.number":"265183","frame.len":"1506","frame.cap_len":"1506","frame.md5_hash":"8a65a638823026fd96ab71401e31fb0b","frame.marked":"0","frame.ignored":"0","frame.file_off":"278075158","frame.protocols":"eth:ethertype:ip:tcp","frame.coloring_rule.name":"HTTP","frame.coloring_rule.string":"http || tcp.port == 80 || http2"},"eth":{"eth.dst":"78:e4:00:6c:39:cd","eth.dst_tree":{"eth.dst_resolved":"HonHaiPr_6c:39:cd","eth.dst.oui":"7922688","eth.dst.oui_resolved":"Hon Hai Precision Ind. Co.,Ltd.","eth.addr":"78:e4:00:6c:39:cd","eth.addr_resolved":"HonHaiPr_6c:39:cd","eth.addr.oui":"7922688","eth.addr.oui_resolved":"Hon Hai Precision Ind. Co.,Ltd.","eth.dst.lg":"0","eth.lg":"0","eth.dst.ig":"0","eth.ig":"0"},"eth.src":"38:72:c0:5e:6b:22","eth.src_tree":{"eth.src_resolved":"Comtrend_5e:6b:22","eth.src.oui":"3699392","eth.src.oui_resolved":"Comtrend Corporation","eth.addr":"38:72:c0:5e:6b:22","eth.addr_resolved":"Comtrend_5e:6b:22","eth.addr.oui":"3699392","eth.addr.oui_resolved":"Comtrend Corporation","eth.src.lg":"0","eth.lg":"0","eth.src.ig":"0","eth.ig":"0"},"eth.type":"0x0800"},"ip":{"ip.version":"4","ip.hdr_len":"20","ip.dsfield":"0x00","ip.dsfield_tree":{"ip.dsfield.dscp":"0","ip.dsfield.ecn":"0"},"ip.len":"1492","ip.id":"0x9b2c","ip.flags":"0x02","ip.flags_tree":{"ip.flags.rb":"0","ip.flags.df":"1","ip.flags.mf":"0"},"ip.frag_offset":"0","ip.ttl":"52","ip.proto":"6","ip.checksum":"0x86fd","ip.checksum.status":"1","ip.checksum_calculated":"0x86fd","ip.src":"147.251.128.210","ip.addr":"10.0.0.45","ip.src_host":"147.251.128.210","ip.host":"10.0.0.45","ip.dst":"10.0.0.45","ip.dst_host":"10.0.0.45"},"tcp":{"tcp.srcport":"80","tcp.dstport":"50578","tcp.port":"50578","tcp.stream":"705","tcp.completeness":"31","tcp.len":"1452","tcp.seq":"116121","tcp.seq_raw":"500544689","tcp.nxtseq":"117573","tcp.ack":"420","tcp.ack_raw":"1086510192","tcp.hdr_len":"20","tcp.flags":"0x0010","tcp.flags_tree":{"tcp.flags.res":"0","tcp.flags.ae":"0","tcp.flags.cwr":"0","tcp.flags.ece":"0","tcp.flags.urg":"0","tcp.flags.ack":"1","tcp.flags.push":"0","tcp.flags.reset":"0","tcp.flags.syn":"0","tcp.flags.fin":"0","tcp.flags.str":",%V%,%V%,%V%,%V%,%V%,%V%,%V%A,%V%,%V%,%V%,%V%"},"tcp.window_size_value":"245","tcp.window_size":"15680","tcp.window_size_scalefactor":"64","tcp.checksum":"0x9179","tcp.checksum.status":"1","tcp.checksum_calculated":"0x9179","tcp.urgent_pointer":"0","Timestamps":{"tcp.time_relative":"0.862885000","tcp.time_delta":"0.001450000"},"tcp.analysis":{"tcp.analysis.initial_rtt":"0.048000000","tcp.analysis.bytes_in_flight":"4356","tcp.analysis.push_bytes_sent":"4356"},"tcp.payload":"51:80:60:e4:a0:ce:32:4a:1f:95:be:b4:a6:21:2b:3b:b1:65:00:80:57:82:0e:71:81:9f:a7:f2:a7:ee:39:da:c0:6d:04:e0:74:07:d2:91:d5:64:c1:19:e0:7c:ea:14:13:db:1c:83:9a:4d:da:c4:34:af:e6:03:86:f9:b3:cf:2a:a0:64:67:a1:e6:92:1e:32:a4:1c:05:eb:9f:d6:94:6f:8f:2a:a4:f9:67:24:30:3d:01:1c:e3:3e:f4:f0:56:3c:1c:6e:dc:40:c6:7b:77:c1:e9:f8:53:6d:59:a3:44:af:ef:45:91:c8:cc:54:3a:60:f2:39:07:93:eb:42:63:25:53:2b:18:23:6f:98:72:7d:f3:4a:cc:52:60:37:05:52:38:61:c7:b8:18:fa:8e:be:f4:f6:25:90:16:74:2c:46:42:28:c6:3d:33:50:bb:a2:6d:29:3b:f4:1a:d2:2a:86:eb:bb:a6:32:4f:38:e3:03:eb:4c:2"}}}}

I tried following the staps given in this article Analyzing network packets with Wireshark, Elasticsearch, and Kibana | Elastic Blog to now avail.

Which actions do I need to take to reindex my output properly and get my imported data out of the field message?

2

Hi everybody!

I had an idea on how to fix my problem, but I wanted to ask what your opinion is.

Would it be possible to rename via a pipeline fieldnames in the message, to already known ones by Elastic? So my theory being that Elastic would recognize these fieldnames as fieldnames instead of just text-based data. I am looking forward to your opinions.

3

Can you share the various configs you have setup for this?

4

Hereby the configuration file for filebeat

filebeat.inputs: 

- input_type: log 

  paths: 

    - "path file"
  json.keys_under_root: true
  json.add_error_key: true

# =============================== Elastic Cloud ================================


# These settings simplify using Packetbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
cloud.id: "cloudid"

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
cloud.auth: "user:ww"


output.elasticsearch:
pipeline: .rename ip.src as source.ip + geoip
output.elasticsearch.index: "test-%{[agent.version]}"
setup.template.name: "test"
setup.template.pattern: "test-%{[agent.version]}"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - decode_json_fields:
      fields: ["message"]
      process_array: false
      max_depth: 1
      target: ""
      overwrite_keys: false
      add_error_key: true

This is the configuration of the used pipeline

[
  {
    "dot_expander": {
      "field": "ip.src"
    }
  },
  {
    "rename": {
      "field": "ip.src",
      "target_field": "source.ip"
    }
  },
  {
    "geoip": {
      "field": "source.ip",
      "target_field": "source.nat.ip"
    }
  }
]

And hereby a snapshot of the imported data as seen in Kibana discover in JSON format

"\u0000{\u0000\"\u0000_\u0000i\u0000n\u0000d\u0000e\u0000x\u0000\"\u0000:\u0000\"\u0000p\u0000a\u0000c\u0000k\u0000e\u0000t\u0000s\u0000-\u00002\u00000\u00001\u00005\u0000-\u00000\u00007\u0000-\u00002\u00005\u0000\"\u0000,\u0000\"\u0000_\u0000t\u0000y\u0000p\u0000e\u0000\"\u0000:\u0000\"\u0000d\u0000o\u0000c\u0000\"\u0000,\u0000\"\u0000_\u0000s\u0000c\u0000o\u0000r\u0000e\u0000\"\u0000:\u0000n\u0000u\u0000l\u0000l\u0000,
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.