# Getting JSON data out of message field imported with filebeat
**URL:** https://discuss.elastic.co/t/getting-json-data-out-of-message-field-imported-with-filebeat/335872
**Category:** Beats
**Tags:** filebeat, reindex
**Created:** [June 13, 2023, 12:59pm UTC](https://discuss.elastic.co/t/getting-json-data-out-of-message-field-imported-with-filebeat/335872 "2023-06-13T12:59:02Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Lou003](https://discuss.elastic.co/u/Lou003)
#### Post date: [June 13, 2023, 12:59pm UTC](https://discuss.elastic.co/t/getting-json-data-out-of-message-field-imported-with-filebeat/335872/1 "2023-06-13T12:59:02Z")
Hi everybody,
I have a problem with indexing filebeat output generated from a JSON file. This JSON file has been created by converting a PCAP file and using jq to make it not pretty. After using filebeat this file has been imported to Elastic. Which created an output where all the imported data is visualized in the column message. As soon below.
```auto
{"_index":"packets-2015-07-25","_type":"doc","_score":null,"_source":{"layers":{"frame":{"frame.encap_type":"1","frame.time":"Jul 25, 2015 21:43:13.284118000 West-Europa (zomertijd)","frame.offset_shift":"0.000000000","frame.time_epoch":"1437853393.284118000","frame.time_delta":"0.001450000","frame.time_delta_displayed":"0.001450000","frame.time_relative":"13896.477788000","frame.number":"265183","frame.len":"1506","frame.cap_len":"1506","frame.md5_hash":"8a65a638823026fd96ab71401e31fb0b","frame.marked":"0","frame.ignored":"0","frame.file_off":"278075158","frame.protocols":"eth:ethertype:ip:tcp","frame.coloring_rule.name":"HTTP","frame.coloring_rule.string":"http || tcp.port == 80 || http2"},"eth":{"eth.dst":"78:e4:00:6c:39:cd","eth.dst_tree":{"eth.dst_resolved":"HonHaiPr_6c:39:cd","eth.dst.oui":"7922688","eth.dst.oui_resolved":"Hon Hai Precision Ind. Co.,Ltd.","eth.addr":"78:e4:00:6c:39:cd","eth.addr_resolved":"HonHaiPr_6c:39:cd","eth.addr.oui":"7922688","eth.addr.oui_resolved":"Hon Hai Precision Ind. Co.,Ltd.","eth.dst.lg":"0","eth.lg":"0","eth.dst.ig":"0","eth.ig":"0"},"eth.src":"38:72:c0:5e:6b:22","eth.src_tree":{"eth.src_resolved":"Comtrend_5e:6b:22","eth.src.oui":"3699392","eth.src.oui_resolved":"Comtrend Corporation","eth.addr":"38:72:c0:5e:6b:22","eth.addr_resolved":"Comtrend_5e:6b:22","eth.addr.oui":"3699392","eth.addr.oui_resolved":"Comtrend Corporation","eth.src.lg":"0","eth.lg":"0","eth.src.ig":"0","eth.ig":"0"},"eth.type":"0x0800"},"ip":{"ip.version":"4","ip.hdr_len":"20","ip.dsfield":"0x00","ip.dsfield_tree":{"ip.dsfield.dscp":"0","ip.dsfield.ecn":"0"},"ip.len":"1492","ip.id":"0x9b2c","ip.flags":"0x02","ip.flags_tree":{"ip.flags.rb":"0","ip.flags.df":"1","ip.flags.mf":"0"},"ip.frag_offset":"0","ip.ttl":"52","ip.proto":"6","ip.checksum":"0x86fd","ip.checksum.status":"1","ip.checksum_calculated":"0x86fd","ip.src":"147.251.128.210","ip.addr":"10.0.0.45","ip.src_host":"147.251.128.210","ip.host":"10.0.0.45","ip.dst":"10.0.0.45","ip.dst_host":"10.0.0.45"},"tcp":{"tcp.srcport":"80","tcp.dstport":"50578","tcp.port":"50578","tcp.stream":"705","tcp.completeness":"31","tcp.len":"1452","tcp.seq":"116121","tcp.seq_raw":"500544689","tcp.nxtseq":"117573","tcp.ack":"420","tcp.ack_raw":"1086510192","tcp.hdr_len":"20","tcp.flags":"0x0010","tcp.flags_tree":{"tcp.flags.res":"0","tcp.flags.ae":"0","tcp.flags.cwr":"0","tcp.flags.ece":"0","tcp.flags.urg":"0","tcp.flags.ack":"1","tcp.flags.push":"0","tcp.flags.reset":"0","tcp.flags.syn":"0","tcp.flags.fin":"0","tcp.flags.str":",%V%,%V%,%V%,%V%,%V%,%V%,%V%A,%V%,%V%,%V%,%V%"},"tcp.window_size_value":"245","tcp.window_size":"15680","tcp.window_size_scalefactor":"64","tcp.checksum":"0x9179","tcp.checksum.status":"1","tcp.checksum_calculated":"0x9179","tcp.urgent_pointer":"0","Timestamps":{"tcp.time_relative":"0.862885000","tcp.time_delta":"0.001450000"},"tcp.analysis":{"tcp.analysis.initial_rtt":"0.048000000","tcp.analysis.bytes_in_flight":"4356","tcp.analysis.push_bytes_sent":"4356"},"tcp.payload":"51:80:60:e4:a0:ce:32:4a:1f:95:be:b4:a6:21:2b:3b:b1:65:00:80:57:82:0e:71:81:9f:a7:f2:a7:ee:39:da:c0:6d:04:e0:74:07:d2:91:d5:64:c1:19:e0:7c:ea:14:13:db:1c:83:9a:4d:da:c4:34:af:e6:03:86:f9:b3:cf:2a:a0:64:67:a1:e6:92:1e:32:a4:1c:05:eb:9f:d6:94:6f:8f:2a:a4:f9:67:24:30:3d:01:1c:e3:3e:f4:f0:56:3c:1c:6e:dc:40:c6:7b:77:c1:e9:f8:53:6d:59:a3:44:af:ef:45:91:c8:cc:54:3a:60:f2:39:07:93:eb:42:63:25:53:2b:18:23:6f:98:72:7d:f3:4a:cc:52:60:37:05:52:38:61:c7:b8:18:fa:8e:be:f4:f6:25:90:16:74:2c:46:42:28:c6:3d:33:50:bb:a2:6d:29:3b:f4:1a:d2:2a:86:eb:bb:a6:32:4f:38:e3:03:eb:4c:2"}}}}
```
I tried following the staps given in this article [Analyzing network packets with Wireshark, Elasticsearch, and Kibana | Elastic Blog](https://www.elastic.co/blog/analyzing-network-packets-with-wireshark-elasticsearch-and-kibana) to now avail.
Which actions do I need to take to reindex my output properly and get my imported data out of the field message?
---
### Author:  [@Lou003](https://discuss.elastic.co/u/Lou003)
#### Post date: [June 15, 2023, 2:26pm UTC](https://discuss.elastic.co/t/getting-json-data-out-of-message-field-imported-with-filebeat/335872/2 "2023-06-15T14:26:47Z")
Hi everybody!
I had an idea on how to fix my problem, but I wanted to ask what your opinion is.
Would it be possible to rename via a pipeline fieldnames in the message, to already known ones by Elastic? So my theory being that Elastic would recognize these fieldnames as fieldnames instead of just text-based data. I am looking forward to your opinions.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [June 20, 2023, 4:46am UTC](https://discuss.elastic.co/t/getting-json-data-out-of-message-field-imported-with-filebeat/335872/3 "2023-06-20T04:46:14Z")
Can you share the various configs you have setup for this?
---
### Author:  [@Lou003](https://discuss.elastic.co/u/Lou003)
#### Post date: [June 22, 2023, 8:49am UTC](https://discuss.elastic.co/t/getting-json-data-out-of-message-field-imported-with-filebeat/335872/4 "2023-06-22T08:49:29Z")
Hereby the configuration file for filebeat
```auto
filebeat.inputs:
- input_type: log
paths:
- "path file"
json.keys_under_root: true
json.add_error_key: true
# =============================== Elastic Cloud ================================
# These settings simplify using Packetbeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
cloud.id: "cloudid"
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `:`.
cloud.auth: "user:ww"
output.elasticsearch:
pipeline: .rename ip.src as source.ip + geoip
output.elasticsearch.index: "test-%{[agent.version]}"
setup.template.name: "test"
setup.template.pattern: "test-%{[agent.version]}"
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: 1
target: ""
overwrite_keys: false
add_error_key: true
```
This is the configuration of the used pipeline
```auto
[
{
"dot_expander": {
"field": "ip.src"
}
},
{
"rename": {
"field": "ip.src",
"target_field": "source.ip"
}
},
{
"geoip": {
"field": "source.ip",
"target_field": "source.nat.ip"
}
}
]
```
And hereby a snapshot of the imported data as seen in Kibana discover in JSON format
```auto
"\u0000{\u0000\"\u0000_\u0000i\u0000n\u0000d\u0000e\u0000x\u0000\"\u0000:\u0000\"\u0000p\u0000a\u0000c\u0000k\u0000e\u0000t\u0000s\u0000-\u00002\u00000\u00001\u00005\u0000-\u00000\u00007\u0000-\u00002\u00005\u0000\"\u0000,\u0000\"\u0000_\u0000t\u0000y\u0000p\u0000e\u0000\"\u0000:\u0000\"\u0000d\u0000o\u0000c\u0000\"\u0000,\u0000\"\u0000_\u0000s\u0000c\u0000o\u0000r\u0000e\u0000\"\u0000:\u0000n\u0000u\u0000l\u0000l\u0000,
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 20, 2023, 10:50am UTC](https://discuss.elastic.co/t/getting-json-data-out-of-message-field-imported-with-filebeat/335872/5 "2023-07-20T10:50:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.