Mutate function not working (Logstash 6.2.4)

Elastic Stack Logstash
1

Hi,

Any mutate function is working on my PC.

In fact, i want use mutate lowercase function in my logstash config, but the function is not working, any data was sent to ES.
I tried to test other mutate function like add_field, remplace... nothing was working.

I have Kibana version 6.2.4, Elasticsearch and Logstash 6.2.4.
I check that Mutate plugin is installed already (With logstash-plugin list) and i used logstash-plugin update also.

Here is my logstash config:

input {
file {
path => "C:/Users/Desktop/test1///Erreurs/*.csv"
start_position => "beginning"
}

}

filter {
ruby {
code => "event.set('fluxname', event.get('path').split('/')[5])"
}
csv {
separator => ","
columns => ["IdSectionExterne__c", "TECH_MAJ__c", "error"]
}
mutate {
lowercase => [ "fluxname" ]
}
}

output {
elasticsearch {
hosts => ["IP:9200"]
index => "test-%{fluxname}"
}
stdout {
codec => rubydebug
}
}

Here is the output logstash config:

image

Could anyone help me to fix the issue? Thx

2

What does an event produced by Logstash look like, i.e. what does your stdout { codec => rubydebug } output produce?

3

In logstash-plain, log like this:

[2018-04-23T10:38:31,818][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"C:/kibana/Logstash/6.2.4/modules/fb_apache/configuration"}
[2018-04-23T10:38:31,858][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/kibana/Logstash/6.2.4/modules/netflow/configuration"}
[2018-04-23T10:38:32,260][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-04-23T10:38:33,256][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-04-23T10:38:35,040][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-04-23T10:38:41,063][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-04-23T10:38:41,657][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://IP:9200/]}}
[2018-04-23T10:38:41,672][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://IP:9200/, :path=>"/"}
[2018-04-23T10:38:51,951][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://IP:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://IP:9200/][Manticore::ConnectTimeout] connect timed out"}
[2018-04-23T10:38:52,014][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-04-23T10:38:52,123][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Template file '' could not be found!", :class=>"ArgumentError", :backtrace=>["C:/kibana/Logstash/6.2.4/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:31:in read_template_file'", "C:/kibana/Logstash/6.2.4/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:inget_template'", "C:/kibana/Logstash/6.2.4/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7:in install_template'", "C:/kibana/Logstash/6.2.4/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:96:ininstall_template'", "C:/kibana/Logstash/6.2.4/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:26:in register'", "C:/kibana/Logstash/6.2.4/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:inregister'", "C:/kibana/Logstash/6.2.4/logstash-core/lib/logstash/output_delegator.rb:42:in register'", "C:/kibana/Logstash/6.2.4/logstash-core/lib/logstash/pipeline.rb:342:inregister_plugin'", "C:/kibana/Logstash/6.2.4/logstash-core/lib/logstash/pipeline.rb:353:in block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "C:/kibana/Logstash/6.2.4/logstash-core/lib/logstash/pipeline.rb:353:in register_plugins'", "C:/kibana/Logstash/6.2.4/logstash-core/lib/logstash/pipeline.rb:730:inmaybe_setup_out_plugins'", "C:/kibana/Logstash/6.2.4/logstash-core/lib/logstash/pipeline.rb:363:in start_workers'", "C:/kibana/Logstash/6.2.4/logstash-core/lib/logstash/pipeline.rb:290:inrun'", "C:/kibana/Logstash/6.2.4/logstash-core/lib/logstash/pipeline.rb:250:in `block in start'"]}
[2018-04-23T10:38:52,185][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-04-23T10:38:54,561][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4640fce2 run>"}
[2018-04-23T10:38:55,201][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}

Thx.

4

Any suggestion about this issue?
Thx.

5

[2018-04-23T10:38:51,951][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://IP:9200/", :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://IP:9200/][Manticore::ConnectTimeout] connect timed out"}

Fix this first. Why isn't Elasticsearch responding?

Also, I'm not interested in the Logstash log. I want to see the events produced by the stdout output.

Any suggestion about this issue?

My suggestion is to have patience and give people reasonable time to respond.

1 Like
6

@magnusbaeck
Sorry, the connection error issue was fixed. This was the old log this morning.

The stdout output was sent to the file logstash-plain that i published.

Thx

7

The stdout output was sent to the file logstash-plain that i published.

Well, there are no Logstash events in what you posted.

8

Yes, i don't understand, the syntax seems like good, but no data sent to ES.
I have often this output that stops on the line "Pipelines runing,: pipelines main":
image

It's a configuration issue?

Thank u

9

Could you maybe deactivate the whole ES output and filters for a moment and run the configuration (only file input and stdout output) manually from the command line? Then we would know if there were events to begin with and what they look like. And then you could activate the filters one by one...

10

When i remove the mutate function, i can see the data sent to ES in the command line,
but when i add the mutate function, any data was sent to ES and i have the output stopped at the line Pipeline runing: pipelines => main as show before.

Thx.

11

Then show us the rubydebug output events that you get without the filter, please...

12

when i remove the mutate function, i get this:

image
image.png1272×126 19.3 KB

Thx.

13

We are still looking for the ruby debug output, something like the following example. If there are events and you have configured the stdout output, there has to be something.

{
"somefield" => "some value",
"anotherfield" => "another value",
"@timestamp" => 2018-04-23T13:44:43.000Z
}

Oh. And the log you posted tells you that there are encoding problems. You should fix your input codec. The file is not UTF-8.

14

debug output, It's not the log file i sent before?
I'm new in ELK, sorry.
You talk about the output like this that I get it in my firefox like this http://localhost:9200/lot1-l5:

image
image.png1266×186 11.4 KB

When i put mutate function , i get somethink like this in firefox:
image

So i think that hte mutate function is not working on my pc.

I'll fixed the encoding issue thank you.

Thx

15

The first picture is your ES index mapping, isn't it?
What we are looking for is completely independent from ES. If logstash is able to receive an event successfully and your stdout output is activated with rubydebug => true, logstash shows you every event with all the fields it has. If that has never worked, there's been something wrong long before you added the mutate filter.

16

Yes, it's my ES index mapping.

In my config logstash, i have set
stdout {
codec => rubydebug
}

How i can see the stdout output? In firefox or in my data log repository C:\kibana\Logstash\6.2.4\logs?

Thx.

17

file {
path => "C:/Users/Desktop/test1///Erreurs/*.csv"
start_position => "beginning"
}

You know that this configuration won't cause the file to be read every time you run Logstash, right? The start_position option only matters the first time an input file is seen. Read what's said about sincedb in the file input documentation.

18

I don't know this. Thank you your remark.

What i do every time when i change the config file is:

Thx

19

You should either delete the sincedb file each time or set the file input's sincedb_path option to "nul".

20


stdout is text that appears in your CMD window when you execute a command there. It's basically the standard way for software to output its response, if there is no graphical user interface. If that is inconvenient for you, you can use a file output with a rubydebug codec instead.