Got an Error - exception=>#<ArgumentError: invalid byte sequence in UTF-8>,

Elastic Stack Logstash
1

Kindly find the below error log over here.

Feb 02 06:24:49 logstashbox logstash[1082]: [2019-02-02T06:24:49,357][ERROR][logstash.pipeline ] Error ne_id=>"main", :plugin=>"#LogStash::FilterDelegator:0x5af6ed72", :error=>"invalid byte sequence in UTF-8", :0 run>"}
Feb 02 06:24:49 logstashbox logstash[1082]: [2019-02-02T06:24:49,428][ERROR][logstash.pipeline ] Pipeliipeline_id=>"main", :exception=>#<ArgumentError: invalid byte sequence in UTF-8>, :backtrace=>["org/jruby/Ruby"org/jruby/RubyString.java:1607:in =~'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/ck in add_patterns_from_file'", "org/jruby/RubyIO.java:3290:ineach'", "/usr/share/logstash/vendor/bundle/jru.5/lib/grok-pure.rb:70:in add_patterns_from_file'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstogstash/filters/grok.rb:403:inblock in add_patterns_from_files'", "org/jruby/RubyArray.java:1734:in each'",r/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:399:inadd_patterns_from_fi/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:279:in block in regis.java:1734:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.4/lib/logs block in register'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.-4.0.4/lib/logstash/filters/grok.rb:270:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegat", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:242:inregister_plugin'", "/usr/share/logstash/pipeline.rb:253:in block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/pipeline.rb:253:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:595:in, "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:263:instart_workers'", "/usr/share/logstash/lopeline.rb:200:in run'", "/usr/share/ Feb 02 06:24:49 logstashbox logstash[1082]: logstash/logstash-core/lib/logstash/pipeline.rb:160:inblock in s:0x1b47e310 run>"}
Feb 02 06:24:49 logstashbox logstash[1082]: [2019-02-02T06:24:49,498][ERROR][logstash.agent ] Failedain, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineActionlt: false", :backtrace=>nil}
Feb 02 06:24:50 logstashbox logstash[1082]: [2019-02-02T06:24:50,698][INFO ][logstash.agent ] Succes endpoint {:port=>9600}

It seemed encoding related query but I did not know what to do in development file.

Please help..!

2

What does the configuration look like?

3

Configuration of Logstash :

input {
file {
type => "CiscoASALog"
path => "/var/log/remotehosts/asa.log"
}
}
filter {
if [type] == "CiscoASALog" {
grok {
match => ["message", "%{CISCOTIMESTAMP:timestamp} %{HOSTNAME:hostname}? ?%%{CISCOTAG:ciscotag}: %{GREEDYDATA:cisco_message}"]
}
syslog_pri { }
date {
match => ["timestamp",
"MMM dd HH:mm:ss",
"MMM d HH:mm:ss",
"MMM dd yyyy HH:mm:ss",
"MMM d yyyy HH:mm:ss"
]
timezone => "Australia/Sydney"
}
if "_grokparsefailure" not in [tags] {
mutate {
rename => ["cisco_message", "message"]
remove_field => ["timestamp"]
}
}
grok {
match => [
"message", "%{CISCOFW106001}",
"message", "%{CISCOFW106006_106007_106010}",
"message", "%{CISCOFW106014}",
"message", "%{CISCOFW106015}",
"message", "%{CISCOFW106021}",
"message", "%{CISCOFW106023}",
"message", "%{CISCOFW106100}",
"message", "%{CISCOFW110002}",
"message", "%{CISCOFW302010}",
"message", "%{CISCOFW302013_302014_302015_302016}",
"message", "%{CISCOFW302020_302021}",
"message", "%{CISCOFW305011}",
"message", "%{CISCOFW313001_313004_313008}",
"message", "%{CISCOFW313005}",
"message", "%{CISCOFW402117}",
"message", "%{CISCOFW402119}",
"message", "%{CISCOFW419001}",
"message", "%{CISCOFW419002}",
"message", "%{CISCOFW500004}",
"message", "%{CISCOFW602303_602304}",
"message", "%{CISCOFW710001_710002_710003_710005_710006}",
"message", "%{CISCOFW713172}",
"message", "%{CISCOFW733100}"
]
}
}
}

output {
file {
path => "/var/log/logstash/asafeb.log"
}
}

  • Sample Data:
    Feb 2 07:05:05 shubhamshah.test %ASA-6-305012: Teardown dynamic TCP translation from any:192.168.7.148/57394 to outside:10.0.0.2/57394 duration 1:00:01
    Feb 2 07:03:34 shubhamshah.test %ASA-6-305012: Teardown dynamic TCP translation from any:192.168.7.148/57437 to outside:10.0.0.2/57437 duration 0:00:00
    Feb 2 07:03:36 shubhamshah.test %ASA-6-302014: Teardown TCP connection 2665925 for outside:8.8.8.8/443 to inside:192.168.1.148/57438 duration 0:00:01 bytes 51790 TCP FINs
4

If I am reading the stack correctly it is registering the grok plugin and reading patterns from files. If you enable --log.level debug it should log each pattern as it adds it. It will also log where it is loading patterns from

[DEBUG][logstash.filters.grok    ] Grok patterns path {:paths=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns", "/usr/share/logstash/patterns/*"]}

The pattern that causes the error will be immediately after the last pattern logged, so you will have to find which file that pattern is in and find which would be loaded next. Also, try

file /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/* /usr/share/logstash/patterns/*

For the directories that are shown in your logfile (which may be different, depending on versions).

I am able to reproduce the issue by dropping a binary file into /usr/share/logstash/patterns/

5

Thank you Badger.

Can I drop my all debug logs because I am still not able to find particular pattern which makes trouble ?

Shubham

6

Thank you so much man... Issue got resolved. found anonymous rpm download under /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/* . So I just deleted it and issue got resolved.

Have a great time ahead..!

~Shubham

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.