Having Issues With JSON Input

Elastic Stack Logstash
1

Hello all,

I've looked up and down StackOverflow and these forums along with the Logstash docs, and I can't for the life of me get this working. I am simply trying to use the message field as my source so that the items in message are used as fields for my index. Here's what I have setup so far:

input {
  udp {
    port => 5045
    type => "moderators_live"
  }
}

filter {
  if [type] == "moderators_live" {
    json {
      source => "message"
    }
  }
}

output {
 if [type] == "moderators_live" {
    stdout { codec => rubydebug }
    elasticsearch {
      hosts => ["http://172.18.0.21:9200"]
      index => "moderators_live-%{+YYYY.MM.dd}"
    }
  }

So that all works fine, BUT it's not using message as the source, so it's showing up like so:

{
    "@timestamp" => 2020-05-14T04:33:01.864Z,
    "level" => "INFO",
    "host" => "apps.mydomain.com",
    "category" => "default",
    "message" => "{\"@timestamp\":\"2020-05-14T04:33:01.803Z\",\"channel.name\": \"testing\" , \"messageId\": \"710348926837981234\", \"author\": \"astuffedtiger\", \"content\": \"Test\", \"attachments\": \"[]\", \"embeds\": \"[]\"}",
    "@version" => 1,
    "type" => "moderators_live"
}

I just want my fields to be those inside message (@timestamp, channel.name, messageId, author, content, attachments, embeds). I've been working on this for a few hours and it's late so it's quite possible I am overlooking something really simple.

Any hints/tips/pointers would be greatly appreciated!

Thanks!

2

If you remove the json filter, what does the [message] field look like in the rubydebug output? Is it possible you have a message field nested inside your message field?

3

Alright, if I remove the filter (I just commented it out):

#filter { 
#  if [type] == "moderators_live" { 
#    json {
#      source => "message"
#    }
#  }
#}

This is what the output looks like:

{
    "host" => "1.2.3.4",
    "message" => "{\"@version\":1,\"@timestamp\":\"2020-05-14T15:17:09.155Z\",\"host\":\"apps.mydomain.com\",\"level\":\"INFO\",\"category\":\"default\",\"message\":\"{\\\"@timestamp\\\":\\\"2020-05-14T15:17:08.924Z\\\",\\\"channel.name\\\": \\\"testing\\\" , \\\"messageId\\\": \\\"710511024612048999\\\", \\\"author\\\": \\\"astuffedtiger\\\", \\\"content\\\": \\\"Hello world!\\\", \\\"attachments\\\": \\\"[]\\\", \\\"embeds\\\": \\\"[]\\\"}\"}",
    "@version" => "1",
    "@timestamp" => 2020-05-14T15:17:09.171Z,
    "type" => "moderators_live"
}

I apologize for not including this before, this is what I am sending:

{
	"@timestamp": "2020-05-14T15:17:09.171Z",
	"channel.name": "testing",
	"messageId": "710511024612048999",
	"author": "astuffedtiger",
	"content": "Hello world!",
	"attachments": "[]",
	"embeds": "[]"
}
4

OK, so you have a message field inside your message field. You need two json filters.

json { source => "message" }
json { source => "message" }
5

That seems to have done the trick, thank you so much @Badger!!! :purple_heart:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.