# Filebeat vs Logstash

**URL:** <https://discuss.elastic.co/t/filebeat-vs-logstash/127485>\
**Category:** Logstash\
**Created:** [April 10, 2018, 12:57pm UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485 "2018-04-10T12:57:05Z")\
**Posts on this page:** 20\
**Page:** 4

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [April 13, 2018, 6:24am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/61 "2018-04-13T06:24:18Z")

</div>

Ohh ok you r getting stdout.

so your file are getting all logs right..? and you want to save only apache logs..??

---

<div class="post-metadata">

**Author:** ![sancroth](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sancroth](https://discuss.elastic.co/u/sancroth)\
**Post date:** [April 13, 2018, 7:27am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/62 "2018-04-13T07:27:36Z")

</div>

Please try to be clear about what you want to achieve because i cannot understand your use case.  
You want to monitor one log?Multiple logs?You want to filter some?  
Give us paths and log examples.  
Also since you are turning this post into a logstash and not a fileabeats one you should start a new thread on the logstash subforum.

---

<div class="post-metadata">

**Author:** ![sancroth](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sancroth](https://discuss.elastic.co/u/sancroth)\
**Post date:** [April 13, 2018, 7:38am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/63 "2018-04-13T07:38:41Z")

</div>

> [@jawad846](#):
>
> I had 2 instances running with apache and i had configured filebeat on both, and i need to filter only apache and mysql logs separately from both the instances, and save to other instance (where logstash is running) separately in a different file.
> 
> eg:
> 
> apache logs from inst. 1 ====\>
> 
> ---------------------------------------------save both logs to logstash inst. in a single file-- apache.log
> 
> apache logs from inst. 2 ====\>

You have 2 instances.That means 2 different hosts.  
You can also use the `tags: ["yourtag"]`,on your filebeat prosector, to mark where your logs are coming from.  
So when you are about to filter you can use this tag.

```
if "instance1" in [tags]{
    #filters for instance1
}
if "instance2" in [tags]{
    #filters for instance2
}

```

In the end output to the file you want.

---

<div class="post-metadata">

**Author:** ![jawad846](https://avatars.discourse-cdn.com/v4/letter/j/edb3f5/32.png) [@jawad846](https://discuss.elastic.co/u/jawad846)\
**Post date:** [April 13, 2018, 8:40am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/64 "2018-04-13T08:40:58Z")

</div>

i had changed to logstash.

right now, am getting all the logs in output of the logstash,

now i need to make differentiate the logs

i.e. for apache logs i need filter and put it into apache.log in a logstash instance  
and for other services also

@sancroth  
@harshbajaj16

---

<div class="post-metadata">

**Author:** ![sancroth](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sancroth](https://discuss.elastic.co/u/sancroth)\
**Post date:** [April 13, 2018, 8:42am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/65 "2018-04-13T08:42:47Z")

</div>

So you want both apache logs from instance1 and instance2 to be saved on an apache.log on logstash instance. Do i understand correctly?

---

<div class="post-metadata">

**Author:** ![jawad846](https://avatars.discourse-cdn.com/v4/letter/j/edb3f5/32.png) [@jawad846](https://discuss.elastic.co/u/jawad846)\
**Post date:** [April 13, 2018, 8:44am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/66 "2018-04-13T08:44:07Z")

</div>

yes, exactly

---

<div class="post-metadata">

**Author:** ![sancroth](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sancroth](https://discuss.elastic.co/u/sancroth)\
**Post date:** [April 13, 2018, 8:49am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/67 "2018-04-13T08:49:54Z")

</div>

Ok. Since you are sending the logs via filebeat there should be a field that keeps the path called source.  
On the source field you will have access to the logs name.  
So let's say the paths to apache.log on both instances is /var/log/apache/apache.log then:

```
output{
  if [source]=="/var/log/apache/apache.log" {
    file{
      path => "/your/file/you/want/to/save"
    }
  }
  if [source]=="/var/lib/mysql/mysql.log" {
    file{
      path => "/your/other/file"
    }
  }
}

```

If you don't wanna use source you can do the tagging thing i mentioned on my last reply and do the same thing but checking in if "yourtag" in [tags], and yourtag could be apache or mysql.

---

<div class="post-metadata">

**Author:** ![jawad846](https://avatars.discourse-cdn.com/v4/letter/j/edb3f5/32.png) [@jawad846](https://discuss.elastic.co/u/jawad846)\
**Post date:** [April 13, 2018, 10:47am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/68 "2018-04-13T10:47:56Z")

</div>

> [@sancroth](#):
>
> output{  
> if [source]=="/var/log/apache/apache.log" {  
> file{  
> path =\> "/your/file/you/want/to/save"  
> }  
> }  
> if [source]=="/var/lib/mysql/mysql.log" {  
> file{  
> path =\> "/your/other/file"  
> }  
> }  
> }

its not working

i tried it in filter also

input {  
beats {  
port =\> 5044  
}  
}

# FILTER HERE

filter{  
if "/var/log/apache2/\*.log" in ["source"]  
{  
mutate {  
add\_tag =\> ["apache\_logs"]  
}  
}  
}

# OUTPUT HERE

output {  
stdout {  
codec =\> rubydebug  
}  
}

---

<div class="post-metadata">

**Author:** ![sancroth](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sancroth](https://discuss.elastic.co/u/sancroth)\
**Post date:** [April 13, 2018, 11:11am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/69 "2018-04-13T11:11:26Z")

</div>

If this is what you written it's normal to fail.  
Is your log named \*.log?  
This is the wildcard you used to capture it but it has a name and that is what exists on the source.

If you wanna use the wildcard and avoid multiple checks then:

```
filter{
if [source] =~ /\/var\/log\/apache\/.+\.log/ {
  #do your stuff
mutate{
add_tag => ["apache"]
}
}
}
output{
if "apache" in [tags]{
file => "/your/path"
}
}

```

The regex i used should work i think. Otherwise fix the regex or use the files name and not the wildcard.

---

<div class="post-metadata">

**Author:** ![jawad846](https://avatars.discourse-cdn.com/v4/letter/j/edb3f5/32.png) [@jawad846](https://discuss.elastic.co/u/jawad846)\
**Post date:** [April 13, 2018, 11:38am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/70 "2018-04-13T11:38:09Z")

</div>

> [@sancroth](#):
>
> filter{  
> if [source] =~ //var/log/apache/.+.log/ {  
> #do your stuff  
> mutate{  
> add\_tag =\> ["apache"]  
> }  
> }  
> }  
> output{  
> if "apache" in [tags]{  
> file =\> "/your/path"  
> }  
> }

```
# INPUT HERE
input {
    beats {
            port => 5044
    }
}
# FILTER HERE
filter{
    if [source] ="/var/log/apache2/error.log"
    {
    mutate {
             add_tag => ["apache_logs"]
     }
    }
}
# OUTPUT HERE
output {
    if "apache" in [tags]
    {
            file {
           path => "/home/ubuntu/apache.log"
   }
          }
   }
}

```

its not working

---

<div class="post-metadata">

**Author:** ![sancroth](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sancroth](https://discuss.elastic.co/u/sancroth)\
**Post date:** [April 13, 2018, 11:42am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/71 "2018-04-13T11:42:54Z")

</div>

We are overly stretching this out.  
On what you posted you got a typo. It's if [source] == not =.  
What is being written to file?  
Give me an output example.  
Although with this typo logstash should not start.

---

<div class="post-metadata">

**Author:** ![jawad846](https://avatars.discourse-cdn.com/v4/letter/j/edb3f5/32.png) [@jawad846](https://discuss.elastic.co/u/jawad846)\
**Post date:** [April 13, 2018, 11:49am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/72 "2018-04-13T11:49:52Z")

</div>

sorry,

i had put it "==" also

getting error

[ERROR] 2018-04-13 11:39:20.377 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, in, not , ==, !=, \<=, \>=, \<, \>, =~, !~, and, or, xor, nand, { at line 9, column 14 (byte 86) after filter{\n\tif [source] ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in`compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in`map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in`initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in`execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in`with\_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in`each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in`block in converge\_state\_and\_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in`converge\_state\_and\_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in`block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

---

<div class="post-metadata">

**Author:** ![jawad846](https://avatars.discourse-cdn.com/v4/letter/j/edb3f5/32.png) [@jawad846](https://discuss.elastic.co/u/jawad846)\
**Post date:** [April 13, 2018, 12:03pm UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/73 "2018-04-13T12:03:41Z")

</div>

```
# INPUT HERE
input {
    beats {
            port => 5044
    }
}
# FILTER HERE
filter{
    if [source]=="/var/log/apache2/error.log"
    {
    mutate {
             add_tag => ["apache_logs"]
     }
    }
}
# OUTPUT HERE
output {
            file {
           path => "/home/ubuntu/apache.log"
            codec => line { format => "rubydebug" }
   }
}

```

finally, I solved out,

thank you guys ur patience  
@sancroth

Results:

```
{"tags":["beats_input_codec_plain_applied"],"source":"/var/log/dpkg.log","message":"2018-            
04-13 11:58:02 status half-configured apache2:amd64 2.4.18-2ubuntu3.5","prospector": 
{"type":"log"},"beat":{"hostname":"ip-192-168-2-196","version":"6.2.3","name":"ip-192-168- 
2-196"},"offset":56124,"@version":"1","@timestamp":"2018-0 13T11:58:04.567Z","host":"ip-192-168-2-196"}
{"tags":["beats_input_codec_plain_applied"],"source":"/var/log/dpkg.log","message":"2018 
-04-13 11:58:03 startup packages configure","prospector": 
{"type":"log"},"offset":56240,"beat":{"version":"6.2.3","hostname":"ip-192-168-2- 
196","name":"ip-192-168-2-196"},"@version":"1","@timestamp":"2018-04-
```

---

<div class="post-metadata">

**Author:** ![sancroth](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sancroth](https://discuss.elastic.co/u/sancroth)\
**Post date:** [April 13, 2018, 12:06pm UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/74 "2018-04-13T12:06:26Z")

</div>

You want to add :

```
if "apache_logs" in [tags]{ <- this is the tag you added. Compare with what you have do not blindly copy paste
   file { path=> "" }
} 

```

in your output otherwise when you add more logs they will end up in the same file.You need a way to separate them and that's why we added the add\_tag.

---

<div class="post-metadata">

**Author:** ![jawad846](https://avatars.discourse-cdn.com/v4/letter/j/edb3f5/32.png) [@jawad846](https://discuss.elastic.co/u/jawad846)\
**Post date:** [April 13, 2018, 12:09pm UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/75 "2018-04-13T12:09:11Z")

</div>

@sancroth  
in output "if command is not working"

so am trying to create "pipeline" for each and every service or logs

is it fine?

---

<div class="post-metadata">

**Author:** ![sancroth](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sancroth](https://discuss.elastic.co/u/sancroth)\
**Post date:** [April 13, 2018, 12:14pm UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/76 "2018-04-13T12:14:45Z")

</div>

```
# INPUT HERE
input {
    beats {
            port => 5044
    }
}
# FILTER HERE
filter{
    if [source]=="/var/log/apache2/error.log"
    {
    mutate {
             add_tag => ["apache_logs"]
     }
    }
}
# OUTPUT HERE
output {
   if "apache_logs" in [tags] {
            file {
           path => "/home/ubuntu/apache.log"
   }
  }
  #if "mysql" in [tags] { <- after you do the tagging etc. wont be there by magic.
  #other file
  #}
}

```

This is what you need. And then add more checks for other logs based on some field so you can separate your outputs.  
As for your total logs i recommend going with the regex i gave you so you can capture multiple apache logs from different sources otherwise you will need to create a chain of "or"s.

---

<div class="post-metadata">

**Author:** ![sancroth](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sancroth](https://discuss.elastic.co/u/sancroth)\
**Post date:** [April 13, 2018, 12:17pm UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/77 "2018-04-13T12:17:14Z")

</div>

Also it is not working for a reason.  
If you see your output no "apache\_logs" exists. -\> `{"tags":["beats_input_codec_plain_applied"],"source":"/var/log/dpkg.log"` . Beats have tagged the event but it's not an apache one.And no "apache\_logs" tag is there.  
This is becouse the events we see do not come from apache(or that patch you are sawing us so far).  
They come from : /var/log/dpkg.log  
So they never enter that filter of yours. And with my code they don't go anywhere since i never checked for them.

---

<div class="post-metadata">

**Author:** ![jawad846](https://avatars.discourse-cdn.com/v4/letter/j/edb3f5/32.png) [@jawad846](https://discuss.elastic.co/u/jawad846)\
**Post date:** [April 16, 2018, 8:59am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/78 "2018-04-16T08:59:20Z")

</div>

> [@sancroth](#):
>
> # INPUT HERE
> 
> input {  
> beats {  
> port =\> 5044  
> }  
> }
> 
> # FILTER HERE
> 
> filter{  
> if [source]=="/var/log/apache2/error.log"  
> {  
> mutate {  
> add\_tag =\> ["apache\_logs"]  
> }  
> }  
> }
> 
> # OUTPUT HERE
> 
> output {  
> if "apache\_logs" in [tags] {  
> file {  
> path =\> "/home/ubuntu/apache.log"  
> }  
> }  
> #if "mysql" in [tags] { \<- after you do the tagging etc. wont be there by magic.  
> #other file  
> #}  
> }

its not working

---

<div class="post-metadata">

**Author:** ![sancroth](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sancroth](https://discuss.elastic.co/u/sancroth)\
**Post date:** [April 16, 2018, 9:16am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/79 "2018-04-16T09:16:17Z")

</div>

> [@jawad846](#):
>
> {"tags":["beats\_input\_codec\_plain\_applied"],"source":"/var/log/dpkg.log","message":"2018-  
> 04-13 11:58:02 status half-configured apache2:amd64 2.4.18-2ubuntu3.5","prospector":  
> {"type":"log"},"beat":{"hostname":"ip-192-168-2-196","version":"6.2.3","name":"ip-192-168-  
> 2-196"},"offset":56124,"@version":"1","@timestamp":"2018-0 13T11:58:04.567Z","host":"ip-192-168-2-196"}  
> {"tags":["beats\_input\_codec\_plain\_applied"],"source":"/var/log/dpkg.log","message":"2018  
> -04-13 11:58:03 startup packages configure","prospector":  
> {"type":"log"},"offset":56240,"beat":{"version":"6.2.3","hostname":"ip-192-168-2-  
> 196","name":"ip-192-168-2-196"},"@version":"1","@timestamp":"2018-04-

Based on these events i already told you that you will not see anything with the configs i posted.  
I only see dpkg.log in there(ubuntu package manager logs) and no apache logs.  
I think you are not receiving apache logs at all and thus no filtering is happening, no tag is created and thus no output exists.

To make sure your overall setup is working change only your output to :

output {  
if "apache\_logs" in [tags] {  
file {  
path =\> "/home/ubuntu/apache.log"  
}else{  
file{  
path =\> "/another/log/file" # \<- make sure this exists and logstash can write to it  
}  
}  
}

Monitor what is being written to each file.  
It might be a grok\_parse\_failure that ends up to not adding the tag after failing or you just dont receive the apache logs which i think is your current case.

---

<div class="post-metadata">

**Author:** ![jawad846](https://avatars.discourse-cdn.com/v4/letter/j/edb3f5/32.png) [@jawad846](https://discuss.elastic.co/u/jawad846)\
**Post date:** [April 16, 2018, 9:20am UTC](https://discuss.elastic.co/t/filebeat-vs-logstash/127485/80 "2018-04-16T09:20:28Z")

</div>

sorry man,

```
# INPUT HERE
input {
beats {
        port => 5044
    }
}
# FILTER HERE
filter{
if [source]=="/var/log/apache2/error.log"
{
mutate {
         add_tag => ["apache_logs"]
 }
    }
}
# OUTPUT HERE
output {
   if "apache_logs" in [tags] {
        file {
       path => "/home/ubuntu/apache.log"
  }
}

```

i had done this, still not working

Result:

[ERROR] 2018-04-16 09:21:23.251 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, else, if, ", ', } at line 23, column 1 (byte 336) after output {\n if "apache\_logs" in [tags] {\n file {\n path =\> "/home/ubuntu/apache.log"\n }\n}\n", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in`compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in`map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in`initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in`execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in`with\_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in`each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in`block in converge\_state\_and\_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in`converge\_state\_and\_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in`block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

[Previous page](https://discuss.elastic.co/t/filebeat-vs-logstash/127485.md?page=3)

[Next page](https://discuss.elastic.co/t/filebeat-vs-logstash/127485.md?page=5)