# Help reviewing filter/input/ouput Packetbeat nor working with LS

**URL:** <https://discuss.elastic.co/t/help-reviewing-filter-input-ouput-packetbeat-nor-working-with-ls/62489>\
**Category:** Logstash\
**Created:** [October 7, 2016, 1:09pm UTC](https://discuss.elastic.co/t/help-reviewing-filter-input-ouput-packetbeat-nor-working-with-ls/62489 "2016-10-07T13:09:11Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kernel\_Panic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kernel_panic/32/12294_2.png) [@Kernel\_Panic](https://discuss.elastic.co/u/Kernel_Panic)\
**Post date:** [October 7, 2016, 1:09pm UTC](https://discuss.elastic.co/t/help-reviewing-filter-input-ouput-packetbeat-nor-working-with-ls/62489/1 "2016-10-07T13:09:11Z")

</div>

Hi there guys,

I've been having this issue [Packetbeat index not created and no info from Kibana](https://discuss.elastic.co/t/packetbeat-index-not-created-and-no-info-from-kibana/62197/10) , I cannot ingest the data sent by packetbeat into elasticsearch through logstash, I configure the plugin to connect directly to ES and it works, so I need you assistance.

Apart from that, maybe I'm hitting my head against the wall unnecessary, is there any advantage configuring the plugin to connect directly to ES instead of LS?

I've configured filebeat and topbeat and are working through LS, this is my configuration:

00-log.conf ( for sensu , was trying to configure metrics, still no luck )  
input {  
tcp {  
port =\> 5514  
codec =\> "json"  
type =\> "sensu-logs"  
}  
}

01-beats-input.conf ( for the beats )  
input {  
beats {  
port =\> 5044  
congestion\_threshold =\> "60"  
ssl =\> true  
ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt"  
ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key"  
}  
}

10-syslog-filter.conf

filter {  
if [type] == "syslog" {  
grok {  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" }  
add\_field =\> ["received\_at", "%{@timestamp}"]  
add\_field =\> ["received\_from", "%{host}"]  
}  
syslog\_pri { }  
date {  
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]  
}  
}  
}

11-sensu-filter.conf  
filter {  
if [type] == "sensu" {  
date {  
match =\> ["[check][issued]", "UNIX" ]  
}  
mutate {  
remove\_field =\> ["host", "[client][handlers]", "[check][handlers]", "[check][history]", "[client][keepalive][handler]", "[client][keepalive][refresh]", "[client][keepalive][thresholds][critical]", "[client][keepalive][thresholds][warning]", "[client][subscriptions]", "[client][address]" ]  
}  
}  
}

filter {  
mutate {  
add\_field =\> { "event\_id" =\> "%{[client][name]}_%{[check][name]}_%{[check][status]}" }  
}

throttle {  
after\_count =\> 1  
period =\> 86400  
key =\> "%{event\_id}"  
add\_tag =\> "throttled"  
}  
}

filter {  
grok {  
match =\> { "message" =\> "%{DATA:metric} %{DATA:value} %{INT:unixtime}" }  
}  
}

20-packetbeat-output.conf  
output {

# For debugging, remove later.

stdout { codec =\> rubydebug { metadata =\> true } }

# If you need a conditional on the output you could use a tag. Don't use

# type because it will be set to dns or http.

if "packetbeat" in [tags] {  
elasticsearch {  
hosts =\> "localhost:9200"  
manage\_template =\> false  
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
document\_type =\> "%{[@metadata][type]}"  
}  
}  
}

30-elasticsearch-output.conf  
output {  
elasticsearch {  
hosts =\> "localhost:9200"  
sniffing =\> true  
manage\_template =\> false  
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
document\_type =\> "%{[@metadata][type]}"  
}  
}

Version numbers:  
logstash-2.2.4-1.noarch  
elasticsearch-2.4.0-1.noarch

Any help/advice really appreciated!!

Thanks for your time and support  
Best regards

---

<div class="post-metadata">

**Author:** ![Kernel\_Panic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kernel_panic/32/12294_2.png) [@Kernel\_Panic](https://discuss.elastic.co/u/Kernel_Panic)\
**Post date:** [October 10, 2016, 12:48pm UTC](https://discuss.elastic.co/t/help-reviewing-filter-input-ouput-packetbeat-nor-working-with-ls/62489/2 "2016-10-10T12:48:03Z")

</div>

Guys, any help/advice on this?

Thanks you very much for your time and support  
Regards

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 13, 2016, 5:47am UTC](https://discuss.elastic.co/t/help-reviewing-filter-input-ouput-packetbeat-nor-working-with-ls/62489/3 "2016-10-13T05:47:10Z")

</div>

Are Packetbeat events being fed to stdout (which should be the case given the stdout output)? What does an example event look like?

---

<div class="post-metadata">

**Author:** ![Kernel\_Panic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kernel_panic/32/12294_2.png) [@Kernel\_Panic](https://discuss.elastic.co/u/Kernel_Panic)\
**Post date:** [October 14, 2016, 2:03pm UTC](https://discuss.elastic.co/t/help-reviewing-filter-input-ouput-packetbeat-nor-working-with-ls/62489/4 "2016-10-14T14:03:01Z")

</div>

Hi, this is the output I get when configured to connect directly to ES.

timestamp October 13th 2016, 13:33:01.071  
t\_id AVe\_UaV5XF1rq7trfaeB  
t\_index packetbeat-2016.10.13  
#\_score  
t\_type http  
tbeat.hostname servername  
[tbeat.name](http://tbeat.name) servername  
#bytes\_in 815  
#bytes\_out 208  
tclient\_ip x.x.x.x  
#client\_port 42,125  
tclient\_proc  
tclient\_server  
#count 1  
tdirection in  
#http.code 200  
#http.content\_length 72  
thttp.phrase OK  
thttp.request\_headers.accept application/json, text/plain, _/_  
thttp.request\_headers.accept-language es-AR,es;q=0.8,en-US;q=0.5,en;q=0.3  
thttp.request\_headers.connection Keep-Alive  
thttp.request\_headers.cookie uchiwa\_theme=uchiwa-default; BIGipServerpool\_servername01\_80=2177738944.20480.0000; uchiwa\_toastrSettings=%7B%22positionClass%22%3A%22toast-bottom-right%22%2C%22preventOpenDuplicates%22%3Atrue%2C%22timeOut%22%3A7500%7D; hideSilenced=false; hideClientsSilenced=false; hideOccurrences=false  
thttp.request\_headers.host x.x.x.x  
thttp.request\_headers.referer [http://servername/](http://servername/)  
thttp.request\_headers.user-agent Mozilla/5.0 (Windows NT 10.0; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0  
?http.request\_headers.via 1.1 servername  
?http.request\_headers.x-forwarded-for x.x.x.x,x.x.x.x,x.x.x  
?http.request\_headers.x-forwarded-host servername, servername  
?http.request\_headers.x-forwarded-server servername, servername  
thttp.response\_headers.connection close  
thttp.response\_headers.content-length 72  
thttp.response\_headers.content-type text/plain; charset=utf-8  
thttp.response\_headers.date Thu, 13 Oct 2016 18:33:01 GMT  
tip x.x.x.x  
tmethod GET  
tparams  
tpath /health  
#port 80  
tproc  
tquery GET /health  
treal\_ip x.x.x.x  
#responsetime 0  
tserver  
tstatus OK  
?tags servername-tag-inconfig-file  
ttype http

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 15, 2016, 3:49pm UTC](https://discuss.elastic.co/t/help-reviewing-filter-input-ouput-packetbeat-nor-working-with-ls/62489/5 "2016-10-15T15:49:26Z")

</div>

Trying again:

- Is there anything in the Packetbeat log that indicates any problems connecting to Logstash?
- Are Packetbeat events being fed to Logstash's stdout (which should be the case given the stdout output)?
- If yes, what does an example event produced by the stdout output look like?

---

<div class="post-metadata">

**Author:** ![Kernel\_Panic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kernel_panic/32/12294_2.png) [@Kernel\_Panic](https://discuss.elastic.co/u/Kernel_Panic)\
**Post date:** [October 20, 2016, 3:53pm UTC](https://discuss.elastic.co/t/help-reviewing-filter-input-ouput-packetbeat-nor-working-with-ls/62489/6 "2016-10-20T15:53:55Z")

</div>

Hi Magnus, sorry for the delay, had some issued with the VM and had to start it over, this is a new fresh installation.

```
Logstash stuff:
input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => ["received_at", "%{@timestamp}"]
      add_field => ["received_from", "%{host}"]
    }
    syslog_pri { }
    date {
      match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
    }
  }
}

output {
  elasticsearch {
    hosts => ["z77s-daem04.zebra.lan"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

```

Packet Beat configuration:

```
interfaces:
  device: eth0
protocols:
  http:
    ports: [80]
    send_all_headers: true
    split_coookie: true
    real_ip_header: "X-Forwarded-For"
output:
  logstash:
    hosts: ["server:5044"]
    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
  file:
    path: "/var/log/packetbeat"
    filename: packetbeat.log
    number_of_files: 7
shipper:
  name: servername
  ignore_outgoing: true
  refresh_topology_freq: 60
  topology_expire: 120
  queue_size: 1000
  geoip:
    paths:
      - "/usr/share/GeoIP/GeoLiteCity.dat"
logging:
  to_syslog: true
  to_files: true
  files:
    path: /var/log/packetbeat
    name: packetbeat.log

    keepfiles: 7
  selectors: ["*"]
  level: debug

```

No index is created:

```
yellow open filebeat-2016.10.16 5 1 10688 0 4.7mb 4.7mb
yellow open filebeat-2016.10.17 5 1 16288 0 6.5mb 6.5mb
yellow open filebeat-2016.10.18 5 1 16255 0 6.6mb 6.6mb
yellow open filebeat-2016.10.19 5 1 12616 0 5.4mb 5.4mb
yellow open .kibana 1 1 103 0 88.2kb 88.2kb
yellow open filebeat-2016.10.14 5 1 696 0 833.7kb 833.7kb
yellow open filebeat-2016.10.15 5 1 1647 0 1.6mb 1.6mb
yellow open filebeat-2016.10.20 5 1 156814 0 39.3mb 39.3mb
yellow open topbeat-2016.10.20 5 1 9312 0 3.4mb 3.4mb

```

From the logs:

```
2016-10-20T10:36:55-05:00 DBG Init a MongoDB protocol parser
2016-10-20T10:36:55-05:00 DBG Local IP addresses: [127.0.0.1 x.x.x.x]
2016-10-20T10:36:55-05:00 DBG tcp%!(EXTRA string=Port map: %v, map[uint16]protos.Protocol=map[80:http])

2016-10-20T10:36:55-05:00 DBG Initializing sniffer
2016-10-20T10:36:55-05:00 DBG BPF filter: tcp port 80
2016-10-20T10:36:55-05:00 DBG Sniffer type: pcap device: eth0
2016-10-20T10:36:55-05:00 DBG Layer type: Ethernet
2016-10-20T10:36:55-05:00 INFO packetbeat sucessfully setup. Start running
2016-10-20T10:53:43-05:00 DBG Interrupted
2016-10-20T10:53:43-05:00 DBG Interrupted
2016-10-20T10:53:44-05:00 DBG Interrupted
2016-10-20T10:53:44-05:00 DBG Interrupted
2016-10-20T10:53:45-05:00 DBG Interrupted
2016-10-20T10:53:45-05:00 DBG Interrupted
2016-10-20T10:53:46-05:00 DBG Interrupted
2016-10-20T10:53:46-05:00 DBG Interrupted

```

Can you please shed some lights on this?  
Thank you very much for your time and support  
Regared

---

<div class="post-metadata">

**Author:** ![Kernel\_Panic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kernel_panic/32/12294_2.png) [@Kernel\_Panic](https://discuss.elastic.co/u/Kernel_Panic)\
**Post date:** [October 20, 2016, 4:04pm UTC](https://discuss.elastic.co/t/help-reviewing-filter-input-ouput-packetbeat-nor-working-with-ls/62489/7 "2016-10-20T16:04:21Z")

</div>

I do not want to waste your time, but reading this [https://www.elastic.co/guide/en/beats/packetbeat/5.0/faq.html](https://www.elastic.co/guide/en/beats/packetbeat/5.0/faq.html) makes any difference to point the configuration directly to ES ? doing it that way it works.

Thanks  
Regards

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 21, 2016, 5:49am UTC](https://discuss.elastic.co/t/help-reviewing-filter-input-ouput-packetbeat-nor-working-with-ls/62489/8 "2016-10-21T05:49:35Z")

</div>

Please edit your post and make sure the configuration is formatted as preformatted text (there's a toolbar button for that) and that the indentation looks exactly like in your actual configuration. Indentation is important in YAML files and if you don't post your file exactly like it is we might be unable to spot errors.

---

<div class="post-metadata">

**Author:** ![Kernel\_Panic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kernel_panic/32/12294_2.png) [@Kernel\_Panic](https://discuss.elastic.co/u/Kernel_Panic)\
**Post date:** [October 25, 2016, 6:24pm UTC](https://discuss.elastic.co/t/help-reviewing-filter-input-ouput-packetbeat-nor-working-with-ls/62489/9 "2016-10-25T18:24:37Z")

</div>

Hi Magnus, I preformated the text!, as I said before, I configured it to connect directlly to ES ( filebeat and topbeat are configured to connect to logstash ), but packetbeat still doesn't work.

Thanks for your time and support  
Regards

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:32am UTC](https://discuss.elastic.co/t/help-reviewing-filter-input-ouput-packetbeat-nor-working-with-ls/62489/10 "2017-07-06T04:32:50Z")

</div>