HTTP Input - Recording Full JSON Message

[seth.yes (Seth S)]

I've got an HTTP input on Logstash v5.4.0.

I'm trying to record the full JSON message that is posted along with all the fields. I've tried something similar to the following input with no luck. Any ideas on what I'm doing wrong?

input {
  http {
    port => 8080
    type => [ "maps-iei" ]
    add_field => { "raw_message" => "%{message}" }
    add_field => { "iei-type" => "production" }
    id => "maps-iei-prod"
  }
}

But all that comes through for raw_message is the actual string: '%{message}'

[magnusbaeck (Magnus Bäck)]

Works for me with LS 5.4.1:

$ cat test.config 
input {
  http {
    port => 8080
    add_field => { "raw_message" => "%{message}" }
  }
}
output { stdout { codec => rubydebug } }
$ ~/logstash/logstash-5.4.1/bin/logstash -f test.config
Sending Logstash's logs to /home/magnus/logstash/logstash-5.4.1/logs which is now configured via log4j2.properties
[2017-06-11T16:43:08,838][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000}
[2017-06-11T16:43:08,874][INFO ][logstash.pipeline        ] Pipeline main started
[2017-06-11T16:43:08,916][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
        "headers" => {
            "http_accept" => "*/*",
           "content_type" => "application/x-www-form-urlencoded",
           "request_path" => "/foo",
           "http_version" => "HTTP/1.1",
         "request_method" => "POST",
              "http_host" => "localhost:8080",
            "request_uri" => "/foo",
         "content_length" => "14",
        "http_user_agent" => "curl/7.52.1"
    },
     "@timestamp" => 2017-06-11T14:43:30.924Z,
    "raw_message" => "{\"foo\": \"bar\"}",
       "@version" => "1",
           "host" => "127.0.0.1",
        "message" => "{\"foo\": \"bar\"}"
}

(Running curl -XPOST localhost:8080/foo -d '{"foo": "bar"}' in another shell.)

[seth.yes (Seth S)]

@magnusbaeck I'm not really following what you're doing here.. Are you posting the message as: "{\"foo\": \"bar\"}"? Are you purposely setting the raw_message field as the same?

There's a possibility I'm just misinterpreting what you're doing here.

To clarify what I'm trying to accomplish, I'd like to see the full message come in and be indexed as one long field, like a log message would be. So ideally the message would get correctly parsed so I'd be able to see all fields and another field with the entire message in it.

[magnusbaeck (Magnus Bäck)]

Are you posting the message as: "{\"foo\": \"bar\"}"?

Yes. See my curl command.

Are you purposely setting the raw_message field as the same?

That's done with the add_field option in the Logstash configuration.

To clarify what I'm trying to accomplish, I'd like to see the full message come in and be indexed as one long field, like a log message would be. So ideally the message would get correctly parsed so I'd be able to see all fields and another field with the entire message in it.

Right. So keep raw_message around and feed message to a json filter.