I am trying to create a logging service using the elastic stack.
I have some very large logs that also require filtering to adjust to JSON format, and one of them gives me the following json parse error:
JSON parse error, original data now in message field {:message=>"Unexpected end-of-input: was expecting closing quote for a string value\n at [Source: (StringReader); line: 1, column: 9560693]", :exception=>LogStash::Json::ParserError
The error only occurs on the elastic output, and if i use http output for the exact same file, it outputs correctly and the formatting to json works just fine.
logstash.conf file:
input{
tcp {
port => 5000
codec => json
}
}
filter{
if [system] == 'nostro' {
if [event] == 'state_snapshot' {
json {
source => "state"
target => "current_state"
}
mutate { remove_field => ["state"] }
# Separate state fields
ruby {
code => '
current_state = event.get("current_state")
if current_state.is_a?(Hash)
current_state.each do |key, value|
if value.is_a?(Array)
parsed_arr = value.map { |item| JSON.parse(item) rescue item}
event.set("[current_state][#{key}]", parsed_arr)
elsif key.include?("orders") && value.is_a?(Hash)
parsed_orders = value.transform_values { |json_string| JSON.parse(json_string) rescue json_string }
event.set("[current_state][#{key}]", parsed_orders)
end
end
end
'
}
mutate { rename => { "current_state" => "state" } }
}
}
}
output{
# Condition the output index on the incoming logs 'system' field
if [system] == 'nostro' {
# Send to telegram if log is error
if [log_level] in ['ERROR', 'CRITICAL'] {
http {
url => "http://host.docker.internal:8000/telegram-notification"
http_method => "post"
headers => {
"Content-Type" => "application/json"
}
message => ""
automatic_retries => 1
retry_failed => false
}
}
if [event] == 'state_snapshot' {
http {
url => "http://host.docker.internal:8000/log-to-s3"
http_method => "post"
headers => {
"Content-Type" => "application/json"
}
message => ""
automatic_retries => 2
retry_failed => false
}
}
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "%{system}-%{service}-%{+YYYY.MM.dd}"
}
}
else if [system] == 'riskmanagement' {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "riskmanagement-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "other_index"
}
}
stdout{codec => rubydebug}
}
The error occurs in the system= 'nostro' and event = 'state_snapshot' pipeline.
is there a char limit on the elastic output plugin?
it seems like the parser expects the input to end at this char but it doesn't, therefore it doesn't find the expected closing string char that comes a few chars after the specified one.
Any ideas regarding how to overcome this problem?