Logstash configuration error (wants to check if value is null)

Elastic Stack Logstash
1

Hey everybody,

I have an error in my configuration-file.
I still think that the problem is the check if the value of [_source][machine_name] ist null, but i dont know why?!

Logstash configuration file 
input {
  http {
    port => 31311
	type => "log"
  }
  http {
    port => 31312
	type => "performance"
  }
}

filter {
	mutate {
		 remove_field => [ "headers" ]
	}
}

output { 
	if [type] == "log"{
		if [_source][MachineName] == null{
			elasticsearch {
				index => "log-%{+YYYY.MM.dd}"
			}
		}else{
			elasticsearch{
				index => "unbekannt-%{+YYYY.MM.dd}"
			}
		}
	}else if [type] == "performance" {
		if [_source][machine_name] == null{
			elasticsearch{
				index => "performance-%{+YYYY.MM.dd}"
			}
		}else{
			elasticsearch{
				index => "unbekannt-%{+YYYY.MM.dd}"
			}
		}
	} else{
		elasticsearch{
			index => "unbekannt-%{+YYYY.MM.dd}"
		}
	}
}

Especially the output generates an error.

Here you can see the

Logstash error

[2018-04-12T11:41:07,278][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, ( at line 20, column 36 (byte 236) after output { \n\tif [type] == "log"{\n\t\tif [_source][MachineName] == null", :backtrace=>["C:/..MyPathTo../logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "C:/..MyPathTo../logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "C:/..MyPathTo../logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

I hope anyone can help me!
Thanks a lot,
PolterFox

2

If I understood correctly, you should change the test to

if ![_source][MachineName] {
3

Hey @Badger,
thank you for your reply.

I changed it and it works now. But my problem is now that my input messages at port 31311 (the Log document) are sometimes messages for the port 31312 (performance document). I want to check this case with
if ![_source][MachineName] {
because the log document has the field name: machine_name and the performance document has the field name: MachineName.

My problem now is that the performance documents are also routed into the log index. Do you have an idea how I check this case better ?

Thanks a lot!
Robert

4

Is the field really named [_source][MachineName] rather than just MachineName?

What does an example document look like? Copy/paste from Kibana's JSON tab.

5

Hey @magnusbaeck,

Yes I think so.

Here you can find the documents:
The performance-document is that one with the fieldname [_source][MachineName] and the logging-document has the fieldname [_source][machine_name].

Performance-Document 
{
  "_index": "logstash-2018.04.12",
  "_type": "doc",
  "_id": "...",
  "_version": 1,
  "_score": null,
  "_source": {
    ...
    "MachineName": "MY_FAVORITE_MACHINE_NAME",
    ...
  },
  "fields": {
    "@timestamp": [
      "2018-04-12T06:35:48.656Z"
    ]
  },
  "sort": [
    1523514948656
  ]
}
Logging-Document 
{
  "_index": "logstash-2018.04.12",
  "_type": "doc",
  "_id": "...",
  "_version": 1,
  "_score": null,
  "_source": {
    ...
    "machine_name": "MY_FAVORITE_MACHINE_NAME",
    ...
  },
  "fields": {
    "@timestamp": [
      "2018-04-12T06:35:48.856Z"
    ],
    "timestamp": [
      "2018-04-12T06:35:30.000Z"
    ]
  },
  "sort": [
    1523514948856
  ]
}

Thanks a lot!
Robert

6

As I suspected there is indeed no _source in the field name. What you've pasted above isn't the document itself but the API response from ES. In that JSON document the _source field contains the document. The rest of the fields are metadata.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.