Get initial log from Logstash warning

Elastic Stack Logstash
1

Hi,

I have a rsyslog -> logstash -> Elasticsearch setup and noticed a lot of warnings with tried to parse field [v] as an object, but found a concrete value in Logstash logs, but I cannot find the initial syslog which Logstash received from rsyslog. Is there any way to get syslog because of which logstash threw a warning?

Logstash is running as a service and the version is 7.10.2

Logstash warning:

May 16 08:43:09 logstash-prod logstash[27402]: [2022-05-16T08:43:09,813][WARN ][logstash.outputs.elasticsearch][main][8479a9c2760956f31b7228312a89412d7db93b091cceb8033b031b9671b3efba] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2022.05.16", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x1634e346>], :response=>{"index"=>{"_index"=>"logstash-2022.05.16", "_type"=>"_doc", "_id"=>"orAJzIABf21sMtbdGBw4", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [k.v] tried to parse field [v] as object, but found a concrete value"}}}}

In other words, I don't understand what is wrong with the input syslog, because I don't see it in the warning message.

2

Hi,

could you please share the logstash pipeline config you're using?

3

Sure

input {
    relp {
        host => "0.0.0.0"
        port => "20515"
        type => "syslog"
    }
}

filter {
    dissect {
      mapping => {
        "message" => "%{syslog_ts} %{} %{source_hostname} %{program}: %{source_ip} %{message}"
      }
    }
    json {
      skip_on_invalid_json => true
      source => "message"
      target => "k"
      remove_field => [ "message" ]
    }
  }

output {
    elasticsearch {
      hosts => [ "hostname:port" ]
      user => "admin"
      password => "password"
      template => "/etc/logstash/logstash_template.json"
      template_overwrite => true
      ilm_enabled => false
    }
  }
4

See this answer. Once you have indexed an document in which [k][v] is an object any event in which [k][v] is text will be rejected.

5

Thanks for the answer @Badger. It's possible that from the same source (rsyslog) in some cases v is a string, and in others is an object. Now I understand the problem.

Did I understand correctly that my filter should look like this:

filter {
    dissect {
      mapping => {
        "message" => "%{syslog_ts} %{} %{source_hostname} %{program}: %{source_ip} %{message}"
      }
    }
    json {
      skip_on_invalid_json => true
      source => "message"
      target => "k"
      remove_field => [ "message" ]
    }
    if ! [k.v][v] { mutate { rename { "v" => "kv" } } }
  }

After updating filter I receive this error in Logstash:

May 16 19:32:22 logstash-prod logstash[3113]: [2022-05-16T19:32:22,594][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 13, column 37 (byte 322) after filter {\n    dissect {\n      mapping => {\n        \"message\" => \"%{syslog_ts} %{} %{source_hostname} %{program}: %{source_ip} %{message}\"\n      }\n    }\n    json {\n      skip_on_invalid_json => true\n      source => \"message\"\n      target => \"k\"\n      remove_field => [ \"message\" ]\n    }\n    if ! [k.v][v] { mutate { rename ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
6

The error happens because you need => in rename => { "v" => "kv" }

I do not think if ! [k.v][v] { is the right test. Perhaps

if [k][v] { mutate { rename { "[k][v]" => "kv" } } }`
7

Sorry, but still something is incorrect.

Filter configuration:

filter {
    dissect {
      mapping => {
        "message" => "%{syslog_ts} %{} %{source_hostname} %{program}: %{source_ip} %{message}"
      }
    }
    json {
      skip_on_invalid_json => true
      source => "message"
      target => "k"
      remove_field => [ "message" ]
    }
    if [k][v] { mutate { rename { "[k][v]" => "kv" } } }
  }

Logstash error:

 May 17 05:52:23 logstash-prod logstash[28069]: [2022-05-17T05:52:23,170][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 13, column 33 (byte 318) after filter {\n    dissect {\n      mapping => {\n        \"message\" => \"%{syslog_ts} %{} %{source_hostname} %{program}: %{source_ip} %{message}\"\n      }\n    }\n    json {\n      skip_on_invalid_json => true\n      source => \"message\"\n      target => \"k\"\n      remove_field => [ \"message\" ]\n    }\n    if [k][v] { mutate { rename ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
8

As I said, you need to add a =>.

9

Added "=>" after rename, no "Failed to execute action" errors, but unfortunately, this solution doesn't solve my problem. I still have "...found a concrete value" errors.

Could we please return to the question which I initially asked? How to get input syslog for troubleshooting?

10

If you want to store the events for which Elasticsearch returned an error use a DLQ.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.