Ohh ok you r getting stdout.
so your file are getting all logs right..? and you want to save only apache logs..??
Please try to be clear about what you want to achieve because i cannot understand your use case.
You want to monitor one log?Multiple logs?You want to filter some?
Give us paths and log examples.
Also since you are turning this post into a logstash and not a fileabeats one you should start a new thread on the logstash subforum.
You have 2 instances.That means 2 different hosts.
You can also use the tags: ["yourtag"],on your filebeat prosector, to mark where your logs are coming from.
So when you are about to filter you can use this tag.
if "instance1" in [tags]{
#filters for instance1
}
if "instance2" in [tags]{
#filters for instance2
}
In the end output to the file you want.
i had changed to logstash.
right now, am getting all the logs in output of the logstash,
now i need to make differentiate the logs
i.e. for apache logs i need filter and put it into apache.log in a logstash instance
and for other services also
So you want both apache logs from instance1 and instance2 to be saved on an apache.log on logstash instance. Do i understand correctly?
yes, exactly
Ok. Since you are sending the logs via filebeat there should be a field that keeps the path called source.
On the source field you will have access to the logs name.
So let's say the paths to apache.log on both instances is /var/log/apache/apache.log then:
output{
if [source]=="/var/log/apache/apache.log" {
file{
path => "/your/file/you/want/to/save"
}
}
if [source]=="/var/lib/mysql/mysql.log" {
file{
path => "/your/other/file"
}
}
}
If you don't wanna use source you can do the tagging thing i mentioned on my last reply and do the same thing but checking in if "yourtag" in [tags], and yourtag could be apache or mysql.
its not working
i tried it in filter also
input {
beats {
port => 5044
}
}
FILTER HERE
filter{
if "/var/log/apache2/*.log" in ["source"]
{
mutate {
add_tag => [ "apache_logs" ]
}
}
}
OUTPUT HERE
output {
stdout {
codec => rubydebug
}
}
If this is what you written it's normal to fail.
Is your log named *.log?
This is the wildcard you used to capture it but it has a name and that is what exists on the source.
If you wanna use the wildcard and avoid multiple checks then:
filter{
if [source] =~ /\/var\/log\/apache\/.+\.log/ {
#do your stuff
mutate{
add_tag => ["apache"]
}
}
}
output{
if "apache" in [tags]{
file => "/your/path"
}
}
The regex i used should work i think. Otherwise fix the regex or use the files name and not the wildcard.
# INPUT HERE
input {
beats {
port => 5044
}
}
# FILTER HERE
filter{
if [source] ="/var/log/apache2/error.log"
{
mutate {
add_tag => [ "apache_logs" ]
}
}
}
# OUTPUT HERE
output {
if "apache" in [tags]
{
file {
path => "/home/ubuntu/apache.log"
}
}
}
}
its not working
We are overly stretching this out.
On what you posted you got a typo. It's if [source] == not =.
What is being written to file?
Give me an output example.
Although with this typo logstash should not start.
sorry,
i had put it "==" also
getting error
[ERROR] 2018-04-13 11:39:20.377 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, in, not , ==, !=, <=, >=, <, >, =~, !~, and, or, xor, nand, { at line 9, column 14 (byte 86) after filter{\n\tif [source] ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
# INPUT HERE
input {
beats {
port => 5044
}
}
# FILTER HERE
filter{
if [source]=="/var/log/apache2/error.log"
{
mutate {
add_tag => [ "apache_logs" ]
}
}
}
# OUTPUT HERE
output {
file {
path => "/home/ubuntu/apache.log"
codec => line { format => "rubydebug" }
}
}
finally, I solved out,
thank you guys ur patience
@sancroth
Results:
{"tags":["beats_input_codec_plain_applied"],"source":"/var/log/dpkg.log","message":"2018-
04-13 11:58:02 status half-configured apache2:amd64 2.4.18-2ubuntu3.5","prospector":
{"type":"log"},"beat":{"hostname":"ip-192-168-2-196","version":"6.2.3","name":"ip-192-168-
2-196"},"offset":56124,"@version":"1","@timestamp":"2018-0 13T11:58:04.567Z","host":"ip-192-168-2-196"}
{"tags":["beats_input_codec_plain_applied"],"source":"/var/log/dpkg.log","message":"2018
-04-13 11:58:03 startup packages configure","prospector":
{"type":"log"},"offset":56240,"beat":{"version":"6.2.3","hostname":"ip-192-168-2-
196","name":"ip-192-168-2-196"},"@version":"1","@timestamp":"2018-04-You want to add :
if "apache_logs" in [tags]{ <- this is the tag you added. Compare with what you have do not blindly copy paste
file { path=> "" }
}
in your output otherwise when you add more logs they will end up in the same file.You need a way to separate them and that's why we added the add_tag.
@sancroth
in output "if command is not working"
so am trying to create "pipeline" for each and every service or logs
is it fine?
# INPUT HERE
input {
beats {
port => 5044
}
}
# FILTER HERE
filter{
if [source]=="/var/log/apache2/error.log"
{
mutate {
add_tag => [ "apache_logs" ]
}
}
}
# OUTPUT HERE
output {
if "apache_logs" in [tags] {
file {
path => "/home/ubuntu/apache.log"
}
}
#if "mysql" in [tags] { <- after you do the tagging etc. wont be there by magic.
#other file
#}
}
This is what you need. And then add more checks for other logs based on some field so you can separate your outputs.
As for your total logs i recommend going with the regex i gave you so you can capture multiple apache logs from different sources otherwise you will need to create a chain of "or"s.
Also it is not working for a reason.
If you see your output no "apache_logs" exists. -> {"tags":["beats_input_codec_plain_applied"],"source":"/var/log/dpkg.log" . Beats have tagged the event but it's not an apache one.And no "apache_logs" tag is there.
This is becouse the events we see do not come from apache(or that patch you are sawing us so far).
They come from : /var/log/dpkg.log
So they never enter that filter of yours. And with my code they don't go anywhere since i never checked for them.
its not working
Based on these events i already told you that you will not see anything with the configs i posted.
I only see dpkg.log in there(ubuntu package manager logs) and no apache logs.
I think you are not receiving apache logs at all and thus no filtering is happening, no tag is created and thus no output exists.
To make sure your overall setup is working change only your output to :
output {
if "apache_logs" in [tags] {
file {
path => "/home/ubuntu/apache.log"
}else{
file{
path => "/another/log/file" # <- make sure this exists and logstash can write to it
}
}
}
Monitor what is being written to each file.
It might be a grok_parse_failure that ends up to not adding the tag after failing or you just dont receive the apache logs which i think is your current case.
sorry man,
# INPUT HERE
input {
beats {
port => 5044
}
}
# FILTER HERE
filter{
if [source]=="/var/log/apache2/error.log"
{
mutate {
add_tag => [ "apache_logs" ]
}
}
}
# OUTPUT HERE
output {
if "apache_logs" in [tags] {
file {
path => "/home/ubuntu/apache.log"
}
}
i had done this, still not working
Result:
[ERROR] 2018-04-16 09:21:23.251 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, else, if, ", ', } at line 23, column 1 (byte 336) after output {\n if "apache_logs" in [tags] {\n file {\n path => "/home/ubuntu/apache.log"\n }\n}\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}