Logstash input json splitted by newline "\n"

Hi

I have a logstash pipeline running on tcp port 8080, that receive a log from another application.
These application send logs with header "application/json;charset=UTF-8" and maybe send 1 json line or many json spplited by "\n"

Like this:

  • One Json

{
"id" 123,
"message": "hellow"
}

  • Many Json

{
"id" 123,
"message": "hellow"
}
{
"id" 123,
"message": "hellow"
}
{
"id" 123,
"message": "hellow"
}
{
"id" 123,
"message": "hellow"
}
{
"id" 123,
"message": "hellow"
}

On my logstash i have these on input and filter:

input {
tcp {
port => 8080
}
}

filter {
json {
source => "message"
}
}

Sometimes the logs are insertd on elastic, sometimes not.
On the logs have a lot of difrent erros.

Error parsing json {:source=>"message", :raw=>"content-length: 280946\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'content': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
"; line: 1, column: 9]>}ntent-length: 280946
Error parsing json {:source=>"message", :raw=>"host: 35.9.13.77:5515\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'host': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
"; line: 1, column: 6]>}st: 35.9.13.77:5515
Error parsing json {:source=>"message", :raw=>"accept: */*\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'accept': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
"; line: 1, column: 8]>}cept: */*
Error parsing json {:source=>"message", :raw=>"user-agent: AHC/2.1\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'user': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
"; line: 1, column: 6]>}er-agent: AHC/2.1
Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\r"}

Some help about parse json using logstash with multilines?

Thks

Have you tried like this? Are your the "id" 123 field really without : colon? If it's, you have to handle. Correct JSON format is: "id": 123

input {
 tcp {
  port => 8080
  codec => multiline { 
      pattern => "{"
      negate => true
      what => "previous"
      }
 }
}
      
filter {

 mutate { gsub => [ "message", "[\r\n]", "" ]  }
 
 json { source => "message" }
 
}

Hi Rios,
Sorry, i wrote wrong, you are correct, the json is "id": 123.

This is the original string

{ "time": "2022-11-06T23:21:54+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "http", "request_uri": "/heapdump", "request_length": "251", "bytes_sent": "208", "tcpinfo_rtt": "142131", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "80", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "42398", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "-", "ssl_cipher": "-", "ssl_session_reused": "-", "ssl_server_name": "-", "request_id": "615af3c3a0527c1582f0c09ddb4212e2", "requestPath": "/heapdump", "requestQuery": "", "configuration": "1633703625" }
{ "time": "2022-11-06T23:22:48+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "https", "request_uri": "/config/", "request_length": "263", "bytes_sent": "208", "tcpinfo_rtt": "143154", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "443", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "41412", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "TLSv1.3", "ssl_cipher": "TLS_AES_256_GCM_SHA384", "ssl_session_reused": ".", "ssl_server_name": "hom.app.com.br", "request_id": "1ab3ccb50e83312722354a51c703a19f", "requestPath": "/config/", "requestQuery": "", "configuration": "1633703625" }
{ "time": "2022-11-06T23:22:44+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "http", "request_uri": "/console.html", "request_length": "418", "bytes_sent": "208", "tcpinfo_rtt": "142426", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "80", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "46478", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "-", "ssl_cipher": "-", "ssl_session_reused": "-", "ssl_server_name": "-", "request_id": "ca1048d658008aa82f49085e3ebb47ac", "requestPath": "console.html", "requestQuery": "", "configuration": "1633703625" }
{ "time": "2022-11-06T23:22:46+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "http", "request_uri": "/redirect.php", "request_length": "377", "bytes_sent": "208", "tcpinfo_rtt": "143804", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (X11; OpenBSD i386) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "80", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "44980", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "-", "ssl_cipher": "-", "ssl_session_reused": "-", "ssl_server_name": "-", "request_id": "fe81bb65f4f3f320e00230ad8be0d74b", "requestPath": " redirect.php", "requestQuery": "/", "configuration": "1633703625" }
{ "time": "2022-11-06T23:22:45+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "http", "request_uri": "/download.php?file=invoice.pdf", "request_length": "276", "bytes_sent": "208", "tcpinfo_rtt": "142362", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (X11; OpenBSD i386) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "80", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "46568", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "-", "ssl_cipher": "-", "ssl_session_reused": "-", "ssl_server_name": "-", "request_id": "82579e372c4f027da194ef4a81b328e6", "requestPath": "/download.php", "requestQuery": "file=/etc/passwd", "configuration": "1633703625" }

Applying your suggested the error changes to bellow.

Error parsing json {:source=>"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 17969\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'POST': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
Error parsing json {:source=>"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 12225\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'POST': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
Error parsing json {:source=>"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 18604\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'POST': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
Error parsing json {:source=>"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 62574\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'POST': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
Error parsing json {:source=>"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 7675\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'POST': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')

Your original string is multiline JSON, clean structure.
Are you get these 5 lines in a single message? Then you should use json_lines codec. Try this:

input {
  tcp {
    port => 8080
    codec => "json_lines"
  }
}
filter {
}
output {
    stdout {
        codec => rubydebug{ metadata => true}
    }
}

This codec will decode streamed JSON that is newline delimited. Encoding will emit a single JSON string ending in a @delimiter NOTE: Do not use this codec if your source input is line-oriented JSON, for example, redis or file inputs. Rather, use the json codec. More info: This codec is expecting to receive a stream (string) of newline terminated lines. The file input will produce a line string without a newline. Therefore this codec cannot work with line oriented inputs.

Hi

Analysing all requests with tcpdump, i can see incoming with a single line and request with multiline split by \n

Your suggestion was my first test using

input {
  tcp {
    port => 8080
    codec => "json_lines"
  }
}
filter {
split {}
json { source => "message" }
}

I will try apply or new suggestion using json_lines.

thks

Remove from the filter and show what is in the message field.

split {}
json { source => "message" }

Hi

This is output.

{
  "upstream_bytes_sent" => "10",
          "requestPath" => "/",
        "upstream_addr" => "122.22.19.13:443",
    "waf_attack_action" => "-",
           "ssl_cipher" => "TLS_AES_256",
           "session_id" => "-",
               "scheme" => "https",
"sent_http_content_type" => "image/png",
    "waf_total_blocked" => "0",
         "ssl_protocol" => "TLSv1.3",
         "proxy_status" => "-",
                 "tags" => [
[0] "multiline"
],
          "remote_port" => "46126",
              "message" => "{ \"time\": \"2022-11-08T01:35:11+00:00\", \"client\": \"123123\", \"session_id\": \"-\", \"stream\": \"-\", \"host\": \"demo.com\", \"request_time\": \"0.003\", \"request_method\": \"GET\", \"status\": \"200\", \"proxy_status\": \"-\", \"scheme\": \"https\", \"request_uri\": \"/\", \"request_length\":\n \"202\", \"bytes_sent\": \"8713\", \"tcpinfo_rtt\": \"339618\", \"upstream_cache_status\": \"BYPASS\", \"upstream_status\": \"200\", \"upstream_bytes_received\": \"8756\", \"upstream_connect_time\": \"0.000\", \"upstream_header_time\": \"0.004\", \"upstream_response_time\": \"0.004\", \"upstream_addr\": \"122.22.19.13:443\", \"upstream_bytes_sent\": \"1006\", \"sent_http_content_type\": \"image/png\", \"http_user_agent\": \"Mozilla/5.0 (Linux; Android 10; moto g(8) power lite) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.115 Mobile Safari/537.36\", \"http_referer\": \"https://demo.com/\", \"sent_http_x_original_image_size\": \"-\", \"server_protocol\": \"HTTP/2.0\", \"server_port\": \"443\", \"server_addr\": \"121.11.18.125\", \"remote_addr\": \"222.18.136.25\", \"remote_port\": \"46126\", \"waf_attack_family\": \"-\", \"waf_attack_action\": \"-\", \"waf_learning\": \"-\", \"waf_block\": \"-\", \"waf_total_processed\": \"0\", \"waf_total_blocked\": \"0\", \"waf_score\": \"-\", \"waf_match\": \"-\", \"waf_headers\": \"-\", \"country\": \"Brazil\", \"state\": \"SaoPaulo\", \"asn\": \"AS8167 V tal\", \"ssl_protocol\": \"TLSv1.3\", \"ssl_cipher\": \"TLS_AES_256\", \"ssl_session_reused\": \".\", \"ssl_server_name\": \"demo.com\", \"request_id\": \"33ec66acf1f44f96df900b42dca9d4e4\", \"requestPath\": \"/\", \"requestQuery\": \"\", \"configuration\": \"1610714330\" }",
          "server_addr" => "121.11.18.125",
        "configuration" => "1610714330",
      "ssl_server_name" => "demo.com",
               "stream" => "-",
       "request_length" => "202",
          "remote_addr" => "222.18.136.25",
      "server_protocol" => "HTTP/2.0",
"upstream_response_time" => "0.004",
         "http_referer" => "https://demo.com/",
            "waf_score" => "-",
      "http_user_agent" => "Mozilla/5.0 (Linux; Android 10; moto g(8) power lite) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.115 Mobile Safari/537.36",
         "request_time" => "0.003",
               "client" => "123123",
"upstream_bytes_received" => "8756",
"sent_http_x_original_image_size" => "-",
      "upstream_status" => "200",
           "bytes_sent" => "8713",
            "waf_block" => "-",
              "country" => "Brazil",
                "state" => "SaoPaulo",
         "requestQuery" => "",
   "ssl_session_reused" => ".",
          "request_uri" => "/",
             "@version" => "1",
           "@timestamp" => 2022-11-08T01:36:00.335045685Z,
                  "asn" => "AS8167 V tal",
            "waf_match" => "-",
                "event" => {
"original" => "{ \"time\": \"2022-11-08T01:35:11+00:00\", \"client\": \"123123\", \"session_id\": \"-\", \"stream\": \"-\", \"host\": \"demo.com\", \"request_time\": \"0.003\", \"request_method\": \"GET\", \"status\": \"200\", \"proxy_status\": \"-\", \"scheme\": \"https\", \"request_uri\": \"/\", \"request_length\":\n \"202\", \"bytes_sent\": \"8713\", \"tcpinfo_rtt\": \"339618\", \"upstream_cache_status\": \"BYPASS\", \"upstream_status\": \"200\", \"upstream_bytes_received\": \"8756\", \"upstream_connect_time\": \"0.000\", \"upstream_header_time\": \"0.004\", \"upstream_response_time\": \"0.004\", \"upstream_addr\": \"122.22.19.13:443\", \"upstream_bytes_sent\": \"1006\", \"sent_http_content_type\": \"image/png\", \"http_user_agent\": \"Mozilla/5.0 (Linux; Android 10; moto g(8) power lite) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.115 Mobile Safari/537.36\", \"http_referer\": \"https://demo.com/\", \"sent_http_x_original_image_size\": \"-\", \"server_protocol\": \"HTTP/2.0\", \"server_port\": \"443\", \"server_addr\": \"121.11.18.125\", \"remote_addr\": \"222.18.136.25\", \"remote_port\": \"46126\", \"waf_attack_family\": \"-\", \"waf_attack_action\": \"-\", \"waf_learning\": \"-\", \"waf_block\": \"-\", \"waf_total_processed\": \"0\", \"waf_total_blocked\": \"0\", \"waf_score\": \"-\", \"waf_match\": \"-\", \"waf_headers\": \"-\", \"country\": \"Brazil\", \"state\": \"SaoPaulo\", \"asn\": \"AS8167 V tal\", \"ssl_protocol\": \"TLSv1.3\", \"ssl_cipher\": \"TLS_AES_256\", \"ssl_session_reused\": \".\", \"ssl_server_name\": \"demo.com\", \"request_id\": \"33ec66acf1f44f96df900b42dca9d4e4\", \"requestPath\": \"/\", \"requestQuery\": \"\", \"configuration\": \"1610714330\" }"
},...(more json)

And? You have your fields. Now you can only convert to numeric or date types.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.