I'm using a JSON dictionary to convert values to strings, it looks like this:
{
"1.1.0.80.1.*.*": "Motorcycle -> Generic Scooter (Small)",
"1.1.0.80.2.*.*": "Motorcycle -> Generic Sport/Street (Mid-Size)",
"1.1.0.80.3.*.*": "Motorcycle -> Generic Cruiser (Large)",
"1.1.0.80.4.*.*": "Motorcycle -> Generic Dirt Bike",
"1.1.0.80.*.*.*": "Motorcycle",
"1.1.0.81.10.*.*": "Car -> Generic Car",
"1.1.0.81.100.*.*": "Car -> Generic Convertible",
"1.1.0.81.101.*.*": "Car -> Generic Convertible, Mini/Microcar (Tiny)",
etc
}
In my pipeline, I create field Entity, which might look like 1.1.0.80.1.2.4, and then and trying to translate that using the JSON in the following way:
- pipeline.id: entity-state-processing
config.string: |
input { pipeline { address => entitystatelogs } }
filter {
# Checks whether the log has entity daty to parse before attempting to transform it using the dictionary
if [attributes][entityType] {
mutate {
# Creates a new root-level field, Entity, used to store the concatenated 7-digit entity ID
add_field => {
"Entity" => "%{[attributes][entityType][entityKind]}.%{[attributes][entityType][domain]}.%{[attributes][entityType][country]}.%{[attributes][entityType][category]}.%{[attributes][entityType][subcategory]}.%{[attributes][entityType][specific]}.%{[attributes][entityType][extra]}"
}
}
}
# Replaces the spacees with periods
# gsub => [
# "Entity", " ", "-"
# ]
# Uses the Entity field to run the diction against
translate {
regex => true
source => "Entity"
dictionary_path => "/usr/share/logstash/advanced_mappings.json"
refresh_interval => 0
}
}
output {
# Sends parsed logs to elasticsearch
elasticsearch {
hosts => ["${OUTPUT_HOST}"]
user => "${ELASTIC_USER}"
password => "${ELASTIC_PASS}"
index => "{{ .Release.Namespace }}-entity-state-%{+yyyy.MM.dd}"
}
}
This causes my logstash to crash with the following error:
2024/01/18 23:22:11 Setting 'xpack.monitoring.enabled' from environment.
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2024-01-18T23:22:49,518][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2024-01-18T23:22:49,582][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.3.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2024-01-18T23:22:49,584][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Xmx1g, -Xms1g, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2024-01-18T23:22:49,618][INFO ][logstash.settings ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2024-01-18T23:22:49,682][INFO ][logstash.settings ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
[2024-01-18T23:22:50,406][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?]
Not sure how to go about debugging this, as there is no specific error mentioning the translate line of code. Without the translate line, the logstash does not crash, but instead properly generates the Entity field with correct values. Any ideas?