# Filebeat not sending logs to logstash

**URL:** <https://discuss.elastic.co/t/filebeat-not-sending-logs-to-logstash/127372>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 9, 2018, 7:15pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-logs-to-logstash/127372 "2018-04-09T19:15:18Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [April 9, 2018, 7:15pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-logs-to-logstash/127372/1 "2018-04-09T19:15:18Z")

</div>

elk.home: /tmp/ELK

filebeat.prospectors:

- input\_type: log  
paths:

- input\_type: log  
paths:

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.

# Multiple outputs may be used.

#-------------------------- Logstash output ------------------------------

output.logstash:

# Array of hosts to connect to.

hosts: ["10.169.95.54:5044"]

#================================ Logging =====================================

# Sets log level. The default log level is info.

# Available log levels are: critical, error, warning, info, debug

#logging.level: debug

# At debug level, you can selectively enable logging only for some components.

# To enable all selectors use ["\*"]. Examples of other selectors are "beat",

# "publish", "service".

Logstash.conf

input {  
beats {  
port =\> 5044  
}  
}  
filter {  
if [fields][log\_type] == "sesvg1" {  
csv {  
separator =\> ","  
columns =\> ["nodeName","APIName","Value","Share","Total"]  
}  
}  
else if [fields][log\_type] == "sesvg2" {  
csv {  
separator =\> ","  
columns =\> ["nodeName","APIName","Value","Share","Total"]  
}  
}

mutate {  
convert =\> { "nodeName" =\> "string" }  
convert =\> { "APIName" =\> "string" }  
convert =\> { "Value" =\> "string" }  
convert =\> { "Share" =\> "float" }  
convert =\> { "Total" =\> "integer" }  
}  
}

output {  
if [fields][log\_type] == "sesvg1" {  
elasticsearch {  
action =\> "index"  
hosts =\> "[http://localhost:9200](http://localhost:9200)"  
index =\> "sesvg1newbeat1"  
}  
} else if [fields][log\_type] == "sesvg2" {  
elasticsearch {  
action =\> "index"  
hosts =\> "[http://localhost:9200](http://localhost:9200)"  
index =\> "sesvg2newbeat2"  
}  
}  
}

here are configurations and I got see any indexes getting created in kibana I have tried with single log file i am abele to process and see the data getting populated

---

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [April 9, 2018, 7:20pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-logs-to-logstash/127372/2 "2018-04-09T19:20:52Z")

</div>

additional fields are commented

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [April 10, 2018, 6:19am UTC](https://discuss.elastic.co/t/filebeat-not-sending-logs-to-logstash/127372/3 "2018-04-10T06:19:12Z")

</div>

can you pls share the logs of filebeat ???

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [April 10, 2018, 6:30am UTC](https://discuss.elastic.co/t/filebeat-not-sending-logs-to-logstash/127372/4 "2018-04-10T06:30:54Z")

</div>

> [@sandeepnarla22322](#):
>
> ${elk.home}/LogStashOutputFormatted/SES\_VG1/\*.csv
> 
> fields:
> 
> log\_type: sesvg1
> 
> input\_type: log
> 
> paths:
> 
> ${elk.home}/LogStashOutputFormatted/SES\_VG2/\*.csv
> 
> fields:
> 
> log\_type: sesvg2

Hi,

you should use below format for log files ( Each - is a prospector).

> paths:  
> - /var/log/_.log  
> - /var/log/elasticsearch/_  
> - ${LOG}/filebeat  
> - c:\programdata\elasticsearch\logs\*

Thanks,  
Harsh Bajaj

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [April 11, 2018, 2:12pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-logs-to-logstash/127372/5 "2018-04-11T14:12:27Z")

</div>

@sandeepnarla22322 Hello, it will also help to see the log from filebeat, if you start filebeat with `./filebeat -v -e -d "*" -c yourconfig.yml`, the log should tell us what filebeat see and if it can connect to Logstash.

---

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [April 11, 2018, 5:50pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-logs-to-logstash/127372/6 "2018-04-11T17:50:16Z")

</div>

> [@pierhugues](#):
>
> ./filebeat -v -e -d "\*" -c yourconfig.yml

Thanks Pier,

I have a scenario where I have to process logs from 100 prod nodes at the same time and logs will be rotated every five minutes

I am trying test this scenario by dumping a new file from a new file into the folder every five minutes to see if it is processing the logs but I don't see my logs being processed

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common

# options. The filebeat.full.yml file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

# 

# You can find the full configuration reference here:

# [Filebeat Reference | Elastic](https://www.elastic.co/guide/en/beats/filebeat/index.html)

#=========================== Filebeat prospectors =============================

elk.home: /tmp/ELK

filebeat.prospectors:

- input\_type: log  
ignore\_older: 10m  
scan\_frequency: 10s
# kafka log files prospector
paths:
  - ${elk.home}/LogStashOutput/SES\_VG1/\*.csv  
fields:  
logtype: sesvg1

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.

# Multiple outputs may be used.

#-------------------------- Logstash output ------------------------------

output.logstash:

# Array of hosts to connect to.

hosts: ["10.169.95.54:5044"]

#================================ Logging =====================================

# Sets log level. The default log level is info.

# Available log levels are: critical, error, warning, info, debug

#logging.level: debug

# At debug level, you can selectively enable logging only for some components.

# To enable all selectors use ["\*"]. Examples of other selectors are "beat",

# "publish", "service".

#logging.selectors: ["\*"]

Here the config files  
my sample logs node1.csv and I have dumped node2.csv after the fifth minute

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [April 12, 2018, 1:48pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-logs-to-logstash/127372/7 "2018-04-12T13:48:01Z")

</div>

Thanks for including your configuration, but I want to see the output of the command:

`./filebeat -v -e -d "*" -c yourconfig.yml`

The command above should dump to STDOUT what Filebeat is doing this will help us diagnose the problem.

---

<div class="post-metadata">

**Author:** ![sandeepnarla22322](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\
**Post date:** [April 25, 2018, 3:55pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-logs-to-logstash/127372/8 "2018-04-25T15:55:51Z")

</div>

thanks it fixed

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 23, 2018, 3:55pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-logs-to-logstash/127372/9 "2018-05-23T15:55:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.