Logstash input json splitted by newline "\n"

Elastic Stack Logstash
1

Hi

I have a logstash pipeline running on tcp port 8080, that receive a log from another application.
These application send logs with header "application/json;charset=UTF-8" and maybe send 1 json line or many json spplited by "\n"

Like this:

  • One Json

{
"id" 123,
"message": "hellow"
}

  • Many Json

{
"id" 123,
"message": "hellow"
}
{
"id" 123,
"message": "hellow"
}
{
"id" 123,
"message": "hellow"
}
{
"id" 123,
"message": "hellow"
}
{
"id" 123,
"message": "hellow"
}

On my logstash i have these on input and filter:

input {
tcp {
port => 8080
}
}

filter {
json {
source => "message"
}
}

Sometimes the logs are insertd on elastic, sometimes not.
On the logs have a lot of difrent erros.

Error parsing json {:source=>"message", :raw=>"content-length: 280946\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'content': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
"; line: 1, column: 9]>}ntent-length: 280946
Error parsing json {:source=>"message", :raw=>"host: 35.9.13.77:5515\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'host': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
"; line: 1, column: 6]>}st: 35.9.13.77:5515
Error parsing json {:source=>"message", :raw=>"accept: */*\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'accept': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
"; line: 1, column: 8]>}cept: */*
Error parsing json {:source=>"message", :raw=>"user-agent: AHC/2.1\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'user': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
"; line: 1, column: 6]>}er-agent: AHC/2.1
Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>"\r"}

Some help about parse json using logstash with multilines?

Thks

2

Have you tried like this? Are your the "id" 123 field really without : colon? If it's, you have to handle. Correct JSON format is: "id": 123

input {
 tcp {
  port => 8080
  codec => multiline { 
      pattern => "{"
      negate => true
      what => "previous"
      }
 }
}
      
filter {

 mutate { gsub => [ "message", "[\r\n]", "" ]  }
 
 json { source => "message" }
 
}
3

Hi Rios,
Sorry, i wrote wrong, you are correct, the json is "id": 123.

This is the original string

{ "time": "2022-11-06T23:21:54+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "http", "request_uri": "/heapdump", "request_length": "251", "bytes_sent": "208", "tcpinfo_rtt": "142131", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "80", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "42398", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "-", "ssl_cipher": "-", "ssl_session_reused": "-", "ssl_server_name": "-", "request_id": "615af3c3a0527c1582f0c09ddb4212e2", "requestPath": "/heapdump", "requestQuery": "", "configuration": "1633703625" }
{ "time": "2022-11-06T23:22:48+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "https", "request_uri": "/config/", "request_length": "263", "bytes_sent": "208", "tcpinfo_rtt": "143154", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "443", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "41412", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "TLSv1.3", "ssl_cipher": "TLS_AES_256_GCM_SHA384", "ssl_session_reused": ".", "ssl_server_name": "hom.app.com.br", "request_id": "1ab3ccb50e83312722354a51c703a19f", "requestPath": "/config/", "requestQuery": "", "configuration": "1633703625" }
{ "time": "2022-11-06T23:22:44+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "http", "request_uri": "/console.html", "request_length": "418", "bytes_sent": "208", "tcpinfo_rtt": "142426", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "80", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "46478", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "-", "ssl_cipher": "-", "ssl_session_reused": "-", "ssl_server_name": "-", "request_id": "ca1048d658008aa82f49085e3ebb47ac", "requestPath": "console.html", "requestQuery": "", "configuration": "1633703625" }
{ "time": "2022-11-06T23:22:46+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "http", "request_uri": "/redirect.php", "request_length": "377", "bytes_sent": "208", "tcpinfo_rtt": "143804", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (X11; OpenBSD i386) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "80", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "44980", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "-", "ssl_cipher": "-", "ssl_session_reused": "-", "ssl_server_name": "-", "request_id": "fe81bb65f4f3f320e00230ad8be0d74b", "requestPath": " redirect.php", "requestQuery": "/", "configuration": "1633703625" }
{ "time": "2022-11-06T23:22:45+00:00", "client": "1212", "session_id": "-", "stream": "-", "host": "hom.app.com", "request_time": "0.000", "request_method": "GET", "status": "400", "proxy_status": "-", "scheme": "http", "request_uri": "/download.php?file=invoice.pdf", "request_length": "276", "bytes_sent": "208", "tcpinfo_rtt": "142362", "upstream_cache_status": "-", "upstream_status": "-", "upstream_bytes_received": "-", "upstream_connect_time": "-", "upstream_header_time": "-", "upstream_response_time": "-", "upstream_addr": "-", "upstream_bytes_sent": "-", "sent_http_content_type": "application/json", "http_user_agent": "Mozilla/5.0 (X11; OpenBSD i386) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36", "http_referer": "-", "sent_http_x_original_image_size": "-", "server_protocol": "HTTP/1.1", "server_port": "80", "server_addr": "54.23.12.22", "remote_addr": "54.33.92.80", "remote_port": "46568", "waf_attack_family": "-", "waf_attack_action": "-", "waf_learning": "-", "waf_block": "-", "waf_total_processed": "0", "waf_total_blocked": "0", "waf_score": "-", "waf_match": "-", "waf_headers": "-", "country": "Brazil", "state": "Sao Paulo", "asn": "AS16509 AMAZON-02", "ssl_protocol": "-", "ssl_cipher": "-", "ssl_session_reused": "-", "ssl_server_name": "-", "request_id": "82579e372c4f027da194ef4a81b328e6", "requestPath": "/download.php", "requestQuery": "file=/etc/passwd", "configuration": "1633703625" }

Applying your suggested the error changes to bellow.

Error parsing json {:source=>"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 17969\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'POST': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
Error parsing json {:source=>"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 12225\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'POST': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
Error parsing json {:source=>"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 18604\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'POST': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
Error parsing json {:source=>"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 62574\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'POST': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
Error parsing json {:source=>"message", :raw=>"POST / HTTP/1.1\rContent-Type: application/json\rcontent-length: 7675\rhost: 52.45.8.88:8080\raccept: */*\ruser-agent: AHC/2.1\r\r", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'POST': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
4

Your original string is multiline JSON, clean structure.
Are you get these 5 lines in a single message? Then you should use json_lines codec. Try this:

input {
  tcp {
    port => 8080
    codec => "json_lines"
  }
}
filter {
}
output {
    stdout {
        codec => rubydebug{ metadata => true}
    }
}

This codec will decode streamed JSON that is newline delimited. Encoding will emit a single JSON string ending in a @delimiter NOTE: Do not use this codec if your source input is line-oriented JSON, for example, redis or file inputs. Rather, use the json codec. More info: This codec is expecting to receive a stream (string) of newline terminated lines. The file input will produce a line string without a newline. Therefore this codec cannot work with line oriented inputs.

5

Hi

Analysing all requests with tcpdump, i can see incoming with a single line and request with multiline split by \n

Your suggestion was my first test using

input {
  tcp {
    port => 8080
    codec => "json_lines"
  }
}
filter {
split {}
json { source => "message" }
}

I will try apply or new suggestion using json_lines.

thks

6

Remove from the filter and show what is in the message field.

split {}
json { source => "message" }
7

Hi

This is output.

{
  "upstream_bytes_sent" => "10",
          "requestPath" => "/",
        "upstream_addr" => "122.22.19.13:443",
    "waf_attack_action" => "-",
           "ssl_cipher" => "TLS_AES_256",
           "session_id" => "-",
               "scheme" => "https",
"sent_http_content_type" => "image/png",
    "waf_total_blocked" => "0",
         "ssl_protocol" => "TLSv1.3",
         "proxy_status" => "-",
                 "tags" => [
[0] "multiline"
],
          "remote_port" => "46126",
              "message" => "{ \"time\": \"2022-11-08T01:35:11+00:00\", \"client\": \"123123\", \"session_id\": \"-\", \"stream\": \"-\", \"host\": \"demo.com\", \"request_time\": \"0.003\", \"request_method\": \"GET\", \"status\": \"200\", \"proxy_status\": \"-\", \"scheme\": \"https\", \"request_uri\": \"/\", \"request_length\":\n \"202\", \"bytes_sent\": \"8713\", \"tcpinfo_rtt\": \"339618\", \"upstream_cache_status\": \"BYPASS\", \"upstream_status\": \"200\", \"upstream_bytes_received\": \"8756\", \"upstream_connect_time\": \"0.000\", \"upstream_header_time\": \"0.004\", \"upstream_response_time\": \"0.004\", \"upstream_addr\": \"122.22.19.13:443\", \"upstream_bytes_sent\": \"1006\", \"sent_http_content_type\": \"image/png\", \"http_user_agent\": \"Mozilla/5.0 (Linux; Android 10; moto g(8) power lite) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.115 Mobile Safari/537.36\", \"http_referer\": \"https://demo.com/\", \"sent_http_x_original_image_size\": \"-\", \"server_protocol\": \"HTTP/2.0\", \"server_port\": \"443\", \"server_addr\": \"121.11.18.125\", \"remote_addr\": \"222.18.136.25\", \"remote_port\": \"46126\", \"waf_attack_family\": \"-\", \"waf_attack_action\": \"-\", \"waf_learning\": \"-\", \"waf_block\": \"-\", \"waf_total_processed\": \"0\", \"waf_total_blocked\": \"0\", \"waf_score\": \"-\", \"waf_match\": \"-\", \"waf_headers\": \"-\", \"country\": \"Brazil\", \"state\": \"SaoPaulo\", \"asn\": \"AS8167 V tal\", \"ssl_protocol\": \"TLSv1.3\", \"ssl_cipher\": \"TLS_AES_256\", \"ssl_session_reused\": \".\", \"ssl_server_name\": \"demo.com\", \"request_id\": \"33ec66acf1f44f96df900b42dca9d4e4\", \"requestPath\": \"/\", \"requestQuery\": \"\", \"configuration\": \"1610714330\" }",
          "server_addr" => "121.11.18.125",
        "configuration" => "1610714330",
      "ssl_server_name" => "demo.com",
               "stream" => "-",
       "request_length" => "202",
          "remote_addr" => "222.18.136.25",
      "server_protocol" => "HTTP/2.0",
"upstream_response_time" => "0.004",
         "http_referer" => "https://demo.com/",
            "waf_score" => "-",
      "http_user_agent" => "Mozilla/5.0 (Linux; Android 10; moto g(8) power lite) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.115 Mobile Safari/537.36",
         "request_time" => "0.003",
               "client" => "123123",
"upstream_bytes_received" => "8756",
"sent_http_x_original_image_size" => "-",
      "upstream_status" => "200",
           "bytes_sent" => "8713",
            "waf_block" => "-",
              "country" => "Brazil",
                "state" => "SaoPaulo",
         "requestQuery" => "",
   "ssl_session_reused" => ".",
          "request_uri" => "/",
             "@version" => "1",
           "@timestamp" => 2022-11-08T01:36:00.335045685Z,
                  "asn" => "AS8167 V tal",
            "waf_match" => "-",
                "event" => {
"original" => "{ \"time\": \"2022-11-08T01:35:11+00:00\", \"client\": \"123123\", \"session_id\": \"-\", \"stream\": \"-\", \"host\": \"demo.com\", \"request_time\": \"0.003\", \"request_method\": \"GET\", \"status\": \"200\", \"proxy_status\": \"-\", \"scheme\": \"https\", \"request_uri\": \"/\", \"request_length\":\n \"202\", \"bytes_sent\": \"8713\", \"tcpinfo_rtt\": \"339618\", \"upstream_cache_status\": \"BYPASS\", \"upstream_status\": \"200\", \"upstream_bytes_received\": \"8756\", \"upstream_connect_time\": \"0.000\", \"upstream_header_time\": \"0.004\", \"upstream_response_time\": \"0.004\", \"upstream_addr\": \"122.22.19.13:443\", \"upstream_bytes_sent\": \"1006\", \"sent_http_content_type\": \"image/png\", \"http_user_agent\": \"Mozilla/5.0 (Linux; Android 10; moto g(8) power lite) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.115 Mobile Safari/537.36\", \"http_referer\": \"https://demo.com/\", \"sent_http_x_original_image_size\": \"-\", \"server_protocol\": \"HTTP/2.0\", \"server_port\": \"443\", \"server_addr\": \"121.11.18.125\", \"remote_addr\": \"222.18.136.25\", \"remote_port\": \"46126\", \"waf_attack_family\": \"-\", \"waf_attack_action\": \"-\", \"waf_learning\": \"-\", \"waf_block\": \"-\", \"waf_total_processed\": \"0\", \"waf_total_blocked\": \"0\", \"waf_score\": \"-\", \"waf_match\": \"-\", \"waf_headers\": \"-\", \"country\": \"Brazil\", \"state\": \"SaoPaulo\", \"asn\": \"AS8167 V tal\", \"ssl_protocol\": \"TLSv1.3\", \"ssl_cipher\": \"TLS_AES_256\", \"ssl_session_reused\": \".\", \"ssl_server_name\": \"demo.com\", \"request_id\": \"33ec66acf1f44f96df900b42dca9d4e4\", \"requestPath\": \"/\", \"requestQuery\": \"\", \"configuration\": \"1610714330\" }"
},...(more json)
8

And? You have your fields. Now you can only convert to numeric or date types.