# Ingest pipeline split MESSAGE field into multiple fields **URL:** \ **Category:** Kibana\ **Created:** [July 4, 2022, 8:23pm UTC](https://discuss.elastic.co/t/ingest-pipeline-split-message-field-into-multiple-fields/308834 "2022-07-04T20:23:06Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![moberreiter](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moberreiter](https://discuss.elastic.co/u/moberreiter)\ **Post date:** [July 4, 2022, 8:23pm UTC](https://discuss.elastic.co/t/ingest-pipeline-split-message-field-into-multiple-fields/308834/1 "2022-07-04T20:23:07Z")
Hello everybody, I am new to elasticsearch and kibana and wanted to ask how to split a message into multiple fields. I tried creating an ingest pipeline with MESSAGE field and "," as separator, but it doesn't work as I expected. We have syslog-ng receiving firewall logs and sending them into elasticsearch. All relevant data is in the MESSAGE field: {"zone\_src":"SRC","zone\_dst":"DST","reason":"rule","rule\_id":XXXXX,"rule\_description":"RULE-Decription","action":"ACCEPT","@timestamp":"2022-07-04T22:11:30.501591+0200","timestamp":1656965490,"timestamp\_usec":501591,"iface\_in":"ethXX.XXXX","iface\_out":"ethXX.XXX","ip\_src":"XXX.XXX.XXX.XXX","ip\_dst":"XXX.XXX.XXX.XXX","protocol":17,"port\_src":XXXXX,"port\_dst":XX,"mark":17825792,"tos":0,"host\_id":164119,"host\_name":"some\_hostname","logtype":"forward"} How can I create an ingest pipeline to split the message to separate fields with it's values, e.g.: zone\_src: SRC zone\_dst: DST reason: rule rule\_id: XXXXX ..... Thank you in advance, Best regards, M ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [July 4, 2022, 8:44pm UTC](https://discuss.elastic.co/t/ingest-pipeline-split-message-field-into-multiple-fields/308834/2 "2022-07-04T20:44:04Z")
Hi @moberreiter Welcome to the community The best way is to show us the whole raw document... And your ingest pipeline. Please format your code going forward with the `` button It is unclear if you're saying that is the `message` field or the whole document? If you're `message` field is actually a JSON document, then use the JSON processor > **[JSON processor | Elasticsearch Guide \[8.3\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/json-processor.html)** If it's key value pairs, then use the KV processor > **[KV processor | Elasticsearch Guide \[8.3\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/kv-processor.html)** Show us your whole raw document and your whole and ingest pipeline and we'll be able to help. ---
**Author:** ![moberreiter](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moberreiter](https://discuss.elastic.co/u/moberreiter)\ **Post date:** [July 4, 2022, 9:05pm UTC](https://discuss.elastic.co/t/ingest-pipeline-split-message-field-into-multiple-fields/308834/3 "2022-07-04T21:05:25Z")
Hello Stephen, Here is the complete document: ````auto { "_index": "xxxx-fw-000001", "_type": "_doc", "_id": "mVtmyoEBGm8vzSyzjInY", "_version": 1, "_score": 1, "_ignored": [ "MESSAGE.keyword" ], "_source": { "SOURCE": "s_network_fw", "PROGRAM": "xxxxx", "PRIORITY": "notice", "MESSAGE": "{\"zone_src\":\"SRC\",\"zone_dst\":\"DST\",\"reason\":\"rule\",\"rule_id\":533462,\"rule_description\":\"RULE_DSC\",\"action\":\"ACCEPT\",\"@timestamp\":\"2022-07-04T20:08:46.278529+0200\",\"timestamp\":1656958126,\"timestamp_usec\":278529,\"iface_in\":\"ethXX.XXXX\",\"iface_out\":\"ethXX.XXX\",\"ip_src\":\"XXX.XXX.XXX.XXX\",\"ip_dst\":\"XXX.XXX.XXX.XXX\",\"protocol\":17,\"port_src\":49375,\"port_dst\":53,\"mark\":17825792,\"tos\":0,\"host_id\":164119,\"host_name\":\"some_hostname\",\"logtype\":\"forward\"}", "ISODATE": "2022-07-04T20:08:47+02:00", "HOST_FROM": "XXX.XXX.XXX.XXX", "HOST": "XXX.XXX.XXX.XXX", "FACILITY": "user", "@timestamp": "2022-07-04T20:08:47+02:00" }, "fields": { "FACILITY": [ "user" ], "PRIORITY.keyword": [ "notice" ], "PROGRAM.keyword": [ "XX-Firewall" ], "HOST": [ "XXX.XXX.XXX.XXX" ], "PRIORITY": [ "notice" ], "HOST_FROM.keyword": [ "XXX.XXX.XXX.XXX" ], "ISODATE": [ "2022-07-04T18:08:47.000Z" ], "MESSAGE": [ "{\"zone_src\":\"SRC\",\"zone_dst\":\"DST\",\"reason\":\"rule\",\"rule_id\":533462,\"rule_description\":\"rule_description\",\"action\":\"ACCEPT\",\"@timestamp\":\"2022-07-04T20:08:46.278529+0200\",\"timestamp\":1656958126,\"timestamp_usec\":278529,\"iface_in\":\"ethXX.XXXX\",\"iface_out\":\"ethXX.XXX\",\"ip_src\":\"XXX.XXX.XXX.XXX\",\"ip_dst\":\"XXX.XXX.XXX.XXX\",\"protocol\":17,\"port_src\":49375,\"port_dst\":53,\"mark\":17825792,\"tos\":0,\"host_id\":164119,\"host_name\":\"some_hostname\",\"logtype\":\"forward\"}" ], "@timestamp": [ "2022-07-04T18:08:47.000Z" ], "HOST_FROM": [ "XXX.XXX.XXX.XXX" ], "SOURCE": [ "s_network_fw" ], "FACILITY.keyword": [ "user" ], "HOST.keyword": [ "XXX.XXX.XXX.XXX" ], "SOURCE.keyword": [ "s_network_fw" ], "PROGRAM": [ "XX-Firewall" ] }, "ignored_field_values": { "MESSAGE.keyword": [ "{\"zone_src\":\"SRC\",\"zone_dst\":\"DST\",\"reason\":\"rule\",\"rule_id\":533462,\"rule_description\":\"rule_description\",\"action\":\"ACCEPT\",\"@timestamp\":\"2022-07-04T20:08:46.278529+0200\",\"timestamp\":1656958126,\"timestamp_usec\":278529,\"iface_in\":\"ethXX.XXXX\",\"iface_out\":\"ethXX.XXX\",\"ip_src\":\"XXX.XXX.XXX.XXX\",\"ip_dst\":\"XXX.XXX.XXX.XXX\",\"protocol\":17,\"port_src\":49375,\"port_dst\":53,\"mark\":17825792,\"tos\":0,\"host_id\":164119,\"host_name\":\"some_hostname\",\"logtype\":\"forward\"}" ] } }``` ```` ---
**Author:** ![moberreiter](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moberreiter](https://discuss.elastic.co/u/moberreiter)\ **Post date:** [July 4, 2022, 9:06pm UTC](https://discuss.elastic.co/t/ingest-pipeline-split-message-field-into-multiple-fields/308834/4 "2022-07-04T21:06:22Z")
and thank you for your quick response!! Best regards, M ---
**Author:** ![moberreiter](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moberreiter](https://discuss.elastic.co/u/moberreiter)\ **Post date:** [July 4, 2022, 9:13pm UTC](https://discuss.elastic.co/t/ingest-pipeline-split-message-field-into-multiple-fields/308834/5 "2022-07-04T21:13:07Z")
Forgot the ingest pipeline: ````auto PUT _ingest/pipeline/xx-fw { "processors": [ { "split": { "field": "MESSAGE", "separator": "," } } ] }``` ```` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [July 5, 2022, 4:13am UTC](https://discuss.elastic.co/t/ingest-pipeline-split-message-field-into-multiple-fields/308834/6 "2022-07-05T04:13:13Z")
It also looks like you're using a default mapping, but you should create your own through an index template Not sure why you have \_ignored. message.keyword So here is a mapping (you should complete it with all the fields from the message. I used KV processor and cleaned up some stuff. There is an issues and that is your message has an `@timestamp` so you either put the message fields under a `target_field` or you exclude the timestamp `"exclude_keys" : ["@timestamp"]` .. your choice.... or you could add more logic to save to target fields replace the time stamp then copy everything to root... I like the `target_field` approach 🙂 ```auto # Mapping DELETE discuss-test-syslog PUT discuss-test-syslog/ { "mappings": { "properties": { "SOURCE": { "type": "keyword" }, "PROGRAM": { "type": "keyword" }, "PRIORITY": { "type": "keyword" }, "ISODATE": { "type": "date" }, "HOST_FROM": { "type": "ip" }, "HOST": { "type": "ip" }, "FACILITY": { "type": "keyword" }, "@timestamp": { "type": "date" }, "MESSAGE": { "type": "keyword" } } } } GET discuss-test-syslog/_search DELETE discuss-test-syslog PUT discuss-test-syslog/ { "mappings": { "properties": { "SOURCE": { "type": "keyword" }, "PROGRAM": { "type": "keyword" }, "PRIORITY": { "type": "keyword" }, "ISODATE": { "type": "date" }, "HOST_FROM": { "type": "ip" }, "HOST": { "type": "ip" }, "FACILITY": { "type": "keyword" }, "@timestamp": { "type": "date" }, "MESSAGE": { "type": "keyword" } } } } # Ingest Pipeline DELETE _ingest/pipeline/discuss-test-syslog PUT _ingest/pipeline/discuss-test-syslog { "processors": [ { "kv": { "field": "MESSAGE", "field_split": ",", "value_split": ":", "trim_key": "{\\\"", "trim_value": "\\\"}", "target_field": "details" } } ] } # Post with the pipeline POST discuss-test-syslog/_doc?pipeline=discuss-test-syslog { "SOURCE": "s_network_fw", "PROGRAM": "xxxxx", "PRIORITY": "notice", "MESSAGE": "{\"zone_src\":\"SRC\",\"zone_dst\":\"DST\",\"reason\":\"rule\",\"rule_id\":533462,\"rule_description\":\"RULE_DSC\",\"action\":\"ACCEPT\",\"@timestamp\":\"2022-07-04T20:08:46.278529+0200\",\"timestamp\":1656958126,\"timestamp_usec\":278529,\"iface_in\":\"ethXX.XXXX\",\"iface_out\":\"ethXX.XXX\",\"ip_src\":\"XXX.XXX.XXX.XXX\",\"ip_dst\":\"XXX.XXX.XXX.XXX\",\"protocol\":17,\"port_src\":49375,\"port_dst\":53,\"mark\":17825792,\"tos\":0,\"host_id\":164119,\"host_name\":\"some_hostname\",\"logtype\":\"forward\"}", "ISODATE": "2022-07-04T20:08:47+02:00", "HOST_FROM": "192.168.1.1", "HOST": "192.168.2.1", "FACILITY": "user", "@timestamp": "2022-07-04T20:08:47+02:00" } GET discuss-test-syslog/_search ``` Easy way to quickly iterate and test pipeline and simulate all in one. ```auto POST _ingest/pipeline/_simulate { "pipeline": { "processors": [ { "kv": { "field": "MESSAGE", "field_split": ",", "value_split": ":", "trim_key": "{\\\"", "trim_value": "\\\"}", "exclude_keys" : ["@timestamp"] } } ] }, "docs": [ { "_source": { "SOURCE": "s_network_fw", "PROGRAM": "xxxxx", "PRIORITY": "notice", "MESSAGE": "{\"zone_src\":\"SRC\",\"zone_dst\":\"DST\",\"reason\":\"rule\",\"rule_id\":533462,\"rule_description\":\"RULE_DSC\",\"action\":\"ACCEPT\",\"@timestamp\":\"2022-07-04T20:08:46.278529+0200\",\"timestamp\":1656958126,\"timestamp_usec\":278529,\"iface_in\":\"ethXX.XXXX\",\"iface_out\":\"ethXX.XXX\",\"ip_src\":\"XXX.XXX.XXX.XXX\",\"ip_dst\":\"XXX.XXX.XXX.XXX\",\"protocol\":17,\"port_src\":49375,\"port_dst\":53,\"mark\":17825792,\"tos\":0,\"host_id\":164119,\"host_name\":\"some_hostname\",\"logtype\":\"forward\"}", "ISODATE": "2022-07-04T20:08:47+02:00", "HOST_FROM": "192.168.1.1", "HOST": "192.168.2.1", "FACILITY": "user", "@timestamp": "2022-07-04T20:08:47+02:00" } } ] } ``` ---
**Author:** ![moberreiter](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moberreiter](https://discuss.elastic.co/u/moberreiter)\ **Post date:** [July 5, 2022, 6:20am UTC](https://discuss.elastic.co/t/ingest-pipeline-split-message-field-into-multiple-fields/308834/7 "2022-07-05T06:20:37Z")
Hello Stephen, works like a charm, you made my day! Thank you very much for your help!! BR, M ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 2, 2022, 6:21am UTC](https://discuss.elastic.co/t/ingest-pipeline-split-message-field-into-multiple-fields/308834/8 "2022-08-02T06:21:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.