# Logstash input json splitted by newline "\\n" **URL:** \ **Category:** Logstash\ **Created:** [November 4, 2022, 12:59am UTC](https://discuss.elastic.co/t/logstash-input-json-splitted-by-newline-n/318146 "2022-11-04T00:59:52Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![dmrlixos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmrlixos/32/99836_2.png) [@dmrlixos](https://discuss.elastic.co/u/dmrlixos)\ **Post date:** [November 4, 2022, 12:59am UTC](https://discuss.elastic.co/t/logstash-input-json-splitted-by-newline-n/318146/1 "2022-11-04T00:59:53Z")
Hi I have a logstash pipeline running on tcp port 8080, that receive a log from another application. These application send logs with header "application/json;charset=UTF-8" and maybe send 1 json line or many json spplited by "\n" Like this: - One Json > { > "id" 123, > "message": "hellow" > } - Many Json > { > "id" 123, > "message": "hellow" > } > { > "id" 123, > "message": "hellow" > } > { > "id" 123, > "message": "hellow" > } > { > "id" 123, > "message": "hellow" > } > { > "id" 123, > "message": "hellow" > } On my logstash i have these on input and filter: > input { > tcp { > port =\> 8080 > } > } > > filter { > json { > source =\> "message" > } > } Sometimes the logs are insertd on elastic, sometimes not. On the logs have a lot of difrent erros. ```auto Error parsing json {:source=>"message", :raw=>"content-length: 280946\r", :exception=>#}ntent-length: 280946 Error parsing json {:source=>"message", :raw=>"host: 35.9.13.77:5515\r", :exception=>#}st: 35.9.13.77:5515 Error parsing json {:source=>"message", :raw=>"accept: */*\r", :exception=>#}cept: */* Error parsing json {:source=>"message", :raw=>"user-agent: AHC/2.1\r", :exception=>#}er-agent: AHC/2.1 Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\r"} ``` Some help about parse json using logstash with multilines? Thks ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [November 4, 2022, 1:44pm UTC](https://discuss.elastic.co/t/logstash-input-json-splitted-by-newline-n/318146/2 "2022-11-04T13:44:10Z")
Have you tried like this? Are your the `"id" 123` field really without : colon? If it's, you have to handle. Correct JSON format is: `"id": 123` ```auto input { tcp { port => 8080 codec => multiline { pattern => "{" negate => true what => "previous" } } } filter { mutate { gsub => ["message", "[\r\n]", "" ] } json { source => "message" } } ``` ---
**Author:** ![dmrlixos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmrlixos/32/99836_2.png) [@dmrlixos](https://discuss.elastic.co/u/dmrlixos)\ **Post date:** [November 7, 2022, 12:24am UTC](https://discuss.elastic.co/t/logstash-input-json-splitted-by-newline-n/318146/3 "2022-11-07T00:24:18Z")
Hi Rios, Sorry, i wrote wrong, you are correct, the json is "id": 123. This is the original string ```auto { "time": "2022-11-06T23:21:54+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "http", "request_uri": "/heapdump", "request_length": "251", "bytes_sent": "208", "tcpinfo_rtt": "142131", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "80", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "42398", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "-", "ssl_cipher": "-", "ssl_session_reused": "-", "ssl_server_name": "-", "request_id": "615af3c3a0527c1582f0c09ddb4212e2", "requestPath": "/heapdump", "requestQuery": "", "configuration": "1633703625" } { "time": "2022-11-06T23:22:48+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "https", "request_uri": "/config/", "request_length": "263", "bytes_sent": "208", "tcpinfo_rtt": "143154", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "443", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "41412", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "TLSv1.3", "ssl_cipher": "TLS_AES_256_GCM_SHA384", "ssl_session_reused": ".", "ssl_server_name": "hom.app.com.br", "request_id": "1ab3ccb50e83312722354a51c703a19f", "requestPath": "/config/", "requestQuery": "", "configuration": "1633703625" } { "time": "2022-11-06T23:22:44+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "http", "request_uri": "/console.html", "request_length": "418", "bytes_sent": "208", "tcpinfo_rtt": "142426", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "80", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "46478", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "-", "ssl_cipher": "-", "ssl_session_reused": "-", "ssl_server_name": "-", "request_id": "ca1048d658008aa82f49085e3ebb47ac", "requestPath": "console.html", "requestQuery": "", "configuration": "1633703625" } { "time": "2022-11-06T23:22:46+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "http", "request_uri": "/redirect.php", "request_length": "377", "bytes_sent": "208", "tcpinfo_rtt": "143804", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (X11; OpenBSD i386) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "80", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "44980", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "-", "ssl_cipher": "-", "ssl_session_reused": "-", "ssl_server_name": "-", "request_id": "fe81bb65f4f3f320e00230ad8be0d74b", "requestPath": " redirect.php", "requestQuery": "/", "configuration": "1633703625" } { "time": "2022-11-06T23:22:45+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "http", "request_uri": "/download.php?file=invoice.pdf", "request_length": "276", "bytes_sent": "208", "tcpinfo_rtt": "142362", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (X11; OpenBSD i386) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "80", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "46568", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "-", "ssl_cipher": "-", "ssl_session_reused": "-", "ssl_server_name": "-", "request_id": "82579e372c4f027da194ef4a81b328e6", "requestPath": "/download.php", "requestQuery": "file=/etc/passwd", "configuration": "1633703625" } ``` Applying your suggested the error changes to bellow. ```auto Error parsing json {:source=>"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 17969\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 12225\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 18604\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 62574\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 7675\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=># **Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [November 7, 2022, 8:55am UTC](https://discuss.elastic.co/t/logstash-input-json-splitted-by-newline-n/318146/4 "2022-11-07T08:55:23Z") Your original string is multiline JSON, clean structure. Are you get these 5 lines in a single message? Then you should use json\_lines codec. Try this: ```auto input { tcp { port => 8080 codec => "json_lines" } } filter { } output { stdout { codec => rubydebug{ metadata => true} } } ``` _This codec will decode streamed JSON that is newline delimited. Encoding will emit a single JSON string ending in a `@delimiter` NOTE: Do not use this codec if your source input is line-oriented JSON, for example, redis or file inputs. Rather, use the json codec. More info: This codec is expecting to receive a stream (string) of newline terminated lines. The file input will produce a line string without a newline. Therefore this codec cannot work with line oriented inputs._ ---
**Author:** ![dmrlixos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmrlixos/32/99836_2.png) [@dmrlixos](https://discuss.elastic.co/u/dmrlixos)\ **Post date:** [November 7, 2022, 12:14pm UTC](https://discuss.elastic.co/t/logstash-input-json-splitted-by-newline-n/318146/5 "2022-11-07T12:14:26Z")
Hi Analysing all requests with tcpdump, i can see incoming with a single line and request with multiline split by \n Your suggestion was my first test using ```auto input { tcp { port => 8080 codec => "json_lines" } } filter { split {} json { source => "message" } } ``` I will try apply or new suggestion using json\_lines. thks ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [November 7, 2022, 2:06pm UTC](https://discuss.elastic.co/t/logstash-input-json-splitted-by-newline-n/318146/6 "2022-11-07T14:06:56Z")
Remove from the filter and show what is in the message field. ```auto split {} json { source => "message" } ``` ---
**Author:** ![dmrlixos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmrlixos/32/99836_2.png) [@dmrlixos](https://discuss.elastic.co/u/dmrlixos)\ **Post date:** [November 8, 2022, 1:56am UTC](https://discuss.elastic.co/t/logstash-input-json-splitted-by-newline-n/318146/7 "2022-11-08T01:56:35Z")
Hi This is output. ```auto { "upstream_bytes_sent" => "10", "requestPath" => "/", "upstream_addr" => "122.22.19.13:443", "waf_attack_action" => "-", "ssl_cipher" => "TLS_AES_256", "session_id" => "-", "scheme" => "https", "sent_http_content_type" => "image/png", "waf_total_blocked" => "0", "ssl_protocol" => "TLSv1.3", "proxy_status" => "-", "tags" => [ [0] "multiline" ], "remote_port" => "46126", "message" => "{ \"time\": \"2022-11-08T01:35:11+00:00\", \"client\": \"123123\", \"session_id\": \"-\", \"stream\": \"-\", \"host\": \"demo.com\", \"request_time\": \"0.003\", \"request_method\": \"GET\", \"status\": \"200\", \"proxy_status\": \"-\", \"scheme\": \"https\", \"request_uri\": \"/\", \"request_length\":\n \"202\", \"bytes_sent\": \"8713\", \"tcpinfo_rtt\": \"339618\", \"upstream_cache_status\": \"BYPASS\", \"upstream_status\": \"200\", \"upstream_bytes_received\": \"8756\", \"upstream_connect_time\": \"0.000\", \"upstream_header_time\": \"0.004\", \"upstream_response_time\": \"0.004\", \"upstream_addr\": \"122.22.19.13:443\", \"upstream_bytes_sent\": \"1006\", \"sent_http_content_type\": \"image/png\", \"http_user_agent\": \"Mozilla/5.0 (Linux; Android 10; moto g(8) power lite) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.115 Mobile Safari/537.36\", \"http_referer\": \"https://demo.com/\", \"sent_http_x_original_image_size\": \"-\", \"server_protocol\": \"HTTP/2.0\", \"server_port\": \"443\", \"server_addr\": \"121.11.18.125\", \"remote_addr\": \"222.18.136.25\", \"remote_port\": \"46126\", \"waf_attack_family\": \"-\", \"waf_attack_action\": \"-\", \"waf_learning\": \"-\", \"waf_block\": \"-\", \"waf_total_processed\": \"0\", \"waf_total_blocked\": \"0\", \"waf_score\": \"-\", \"waf_match\": \"-\", \"waf_headers\": \"-\", \"country\": \"Brazil\", \"state\": \"SaoPaulo\", \"asn\": \"AS8167 V tal\", \"ssl_protocol\": \"TLSv1.3\", \"ssl_cipher\": \"TLS_AES_256\", \"ssl_session_reused\": \".\", \"ssl_server_name\": \"demo.com\", \"request_id\": \"33ec66acf1f44f96df900b42dca9d4e4\", \"requestPath\": \"/\", \"requestQuery\": \"\", \"configuration\": \"1610714330\" }", "server_addr" => "121.11.18.125", "configuration" => "1610714330", "ssl_server_name" => "demo.com", "stream" => "-", "request_length" => "202", "remote_addr" => "222.18.136.25", "server_protocol" => "HTTP/2.0", "upstream_response_time" => "0.004", "http_referer" => "https://demo.com/", "waf_score" => "-", "http_user_agent" => "Mozilla/5.0 (Linux; Android 10; moto g(8) power lite) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.115 Mobile Safari/537.36", "request_time" => "0.003", "client" => "123123", "upstream_bytes_received" => "8756", "sent_http_x_original_image_size" => "-", "upstream_status" => "200", "bytes_sent" => "8713", "waf_block" => "-", "country" => "Brazil", "state" => "SaoPaulo", "requestQuery" => "", "ssl_session_reused" => ".", "request_uri" => "/", "@version" => "1", "@timestamp" => 2022-11-08T01:36:00.335045685Z, "asn" => "AS8167 V tal", "waf_match" => "-", "event" => { "original" => "{ \"time\": \"2022-11-08T01:35:11+00:00\", \"client\": \"123123\", \"session_id\": \"-\", \"stream\": \"-\", \"host\": \"demo.com\", \"request_time\": \"0.003\", \"request_method\": \"GET\", \"status\": \"200\", \"proxy_status\": \"-\", \"scheme\": \"https\", \"request_uri\": \"/\", \"request_length\":\n \"202\", \"bytes_sent\": \"8713\", \"tcpinfo_rtt\": \"339618\", \"upstream_cache_status\": \"BYPASS\", \"upstream_status\": \"200\", \"upstream_bytes_received\": \"8756\", \"upstream_connect_time\": \"0.000\", \"upstream_header_time\": \"0.004\", \"upstream_response_time\": \"0.004\", \"upstream_addr\": \"122.22.19.13:443\", \"upstream_bytes_sent\": \"1006\", \"sent_http_content_type\": \"image/png\", \"http_user_agent\": \"Mozilla/5.0 (Linux; Android 10; moto g(8) power lite) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.115 Mobile Safari/537.36\", \"http_referer\": \"https://demo.com/\", \"sent_http_x_original_image_size\": \"-\", \"server_protocol\": \"HTTP/2.0\", \"server_port\": \"443\", \"server_addr\": \"121.11.18.125\", \"remote_addr\": \"222.18.136.25\", \"remote_port\": \"46126\", \"waf_attack_family\": \"-\", \"waf_attack_action\": \"-\", \"waf_learning\": \"-\", \"waf_block\": \"-\", \"waf_total_processed\": \"0\", \"waf_total_blocked\": \"0\", \"waf_score\": \"-\", \"waf_match\": \"-\", \"waf_headers\": \"-\", \"country\": \"Brazil\", \"state\": \"SaoPaulo\", \"asn\": \"AS8167 V tal\", \"ssl_protocol\": \"TLSv1.3\", \"ssl_cipher\": \"TLS_AES_256\", \"ssl_session_reused\": \".\", \"ssl_server_name\": \"demo.com\", \"request_id\": \"33ec66acf1f44f96df900b42dca9d4e4\", \"requestPath\": \"/\", \"requestQuery\": \"\", \"configuration\": \"1610714330\" }" },...(more json) ``` ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [November 9, 2022, 4:49pm UTC](https://discuss.elastic.co/t/logstash-input-json-splitted-by-newline-n/318146/8 "2022-11-09T16:49:45Z")
And? You have your fields. Now you can only convert to numeric or date types. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 7, 2022, 4:50pm UTC](https://discuss.elastic.co/t/logstash-input-json-splitted-by-newline-n/318146/9 "2022-12-07T16:50:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.