Json typed log4j data using file input

Elastic Stack Logstash
1

Hi All.

I'm evaluating logstash to replace our river plugin.
Currently, I'm testing below topology.

  1. Create json data which has changed data and write on file using log4j.
    • Using '%m%n' pattern to write only json value.
  2. Read file and send to elastic search using logstash.

We are using index for user clarification and type for elements at elastic search

Below is my json data in log file.
{"Index_Id":"ABCD", "Type":"Type1","_id":"199040",...}

When I test it using stdin with same data , index , document_id and type extract from json data successfully.
But, when I using log4j , any fields aren't extracted and below is log from logstash.

{
"message" => "{"Index_Id":"ABCD", "Type":"ABCD","_id":"199040",...}",
"@version" => "1",
"@timestamp" => "2015-06-01T18:49:20.024Z",
"type" => "%{Type}",
"host" => "dkim",
"path" => "/Users/dkim/search_log/data"
}

Below is my logstash configuration.
input { file {
codec => "json"
type => "%{Type}"
path => "/Users/dkim/search_log/*"
}
}
output {
elasticsearch { host => "127.0.0.1"
index => "%{Index_Id}"
document_id => "%{_id}"
protocol => http
port => 9200
}
stdout { codec => rubydebug }
}

Any body let me know what is problem and how could solve it ?

Thanks
Ducheol

2

Since the type field contains "%{Type}" it seems the file input doesn't allow you to reference fields found in the input when setting the type. Maybe this trips up the codec completely and is the reason why the JSON string isn't expanded? I'd try setting type to a static string to see if that makes any difference. If it doesn't help, try enabling verbose logs with --verbose or even --debug.

By the way:

"message" => "{"Index_Id":"ABCD", "Type":"ABCD","_id":"199040",...}",

I can only assume the missing escaping of the first two pairs of double quotes is just a typo in your post.

3

Thanks Magnus.

--verbose is solve the problem.

One of quotation isn't matched at data, and I haven't recognized until see below verbose message .
JSON parse failure. Falling back to plain-text {:error=>#<JSON::ParserError: unexpected token at ...

BTW, I have one question.
In the document, binding port is recommended for log4j instead of file.
Is there any specific reason to suggest binding port ?

Thanks
Ducheol

4

In the document, binding port is recommended for log4j instead of file.
Is there any specific reason to suggest binding port ?

Sorry, I don't understand this question.