# Filebeat Modules and Parsing, with Logstash - Cisco Related

**URL:** <https://discuss.elastic.co/t/filebeat-modules-and-parsing-with-logstash-cisco-related/246027>\
**Category:** Beats\
**Tags:** beats-module, filebeat\
**Created:** [August 23, 2020, 5:01pm UTC](https://discuss.elastic.co/t/filebeat-modules-and-parsing-with-logstash-cisco-related/246027 "2020-08-23T17:01:04Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![mgotechlock](https://avatars.discourse-cdn.com/v4/letter/m/dc4da7/32.png) [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\
**Post date:** [August 23, 2020, 5:01pm UTC](https://discuss.elastic.co/t/filebeat-modules-and-parsing-with-logstash-cisco-related/246027/1 "2020-08-23T17:01:04Z")

</div>

From what I can glean from [https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html](https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html) and [Problems parsing Cisco ASA logs using filebeat](https://discuss.elastic.co/t/problems-parsing-cisco-asa-logs-using-filebeat/195326/3), using the filebeat cisco module with logstash successfully looks like this: Correct me if I am wrong.

1. You must load the filebeat cisco ingent pipelines from a filebeat system direct to elasticsearch, using filebeat setup --pipelines --modules cisco. For this step, you likely have to break your existing logging from that system in order to do this one-time configuration. Afterward, you can revert back to your previous filebeat config.
2. Configure filebeat to output to logstash. Configure logstash input as "beats" to read in.
3. Then, on your logstash server, Set the `pipeline` option to `%{[@metadata][pipeline]}` in the output stanza. This setting configures Logstash to select the correct ingest pipeline based on metadata passed in the event.

Questions/Comments:

1. So....All the parsing of a filebeat "module" seems to be done either at the logstash or elasticsearch side. It's not quite clear which but certainly does not appear to be at the filebeat side. This begs the question of "whats the point of calling this a filebeat module, when filebeat doesnt do any of the work?"
2. This seems to REQUIRE that logstash output to elasticsearch, right? What if that is not the use case, and instead we want to output somewhere else first, like S3 storage? Are we just out of luck and cannot use these modules?
3. Why is this so overly complicated? Why isnt it just a setting you enable in one place and dont have to jump thru all these hoops?

---

<div class="post-metadata">

**Author:** ![Travis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/travis/32/54079_2.png) [@Travis](https://discuss.elastic.co/u/Travis)\
**Post date:** [September 14, 2020, 8:11am UTC](https://discuss.elastic.co/t/filebeat-modules-and-parsing-with-logstash-cisco-related/246027/2 "2020-09-14T08:11:23Z")

</div>

+1. I have the same questions.

@mgotechlock did you succeed to make it works ? On my side I didn't using this configuration :

Filbeat =\> Logstash (only act as a simple inpout/ouput gateway) =\> Kafka =\> Logstash (for event transformation) =\> Elasticsearch

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 14, 2020, 8:20am UTC](https://discuss.elastic.co/t/filebeat-modules-and-parsing-with-logstash-cisco-related/246027/3 "2020-09-14T08:20:33Z")

</div>

Please do not ping people not already involved in the thread.

---

<div class="post-metadata">

**Author:** ![Travis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/travis/32/54079_2.png) [@Travis](https://discuss.elastic.co/u/Travis)\
**Post date:** [September 14, 2020, 8:23am UTC](https://discuss.elastic.co/t/filebeat-modules-and-parsing-with-logstash-cisco-related/246027/4 "2020-09-14T08:23:59Z")

</div>

Sorry Christian I fixed it

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 14, 2020, 8:37am UTC](https://discuss.elastic.co/t/filebeat-modules-and-parsing-with-logstash-cisco-related/246027/5 "2020-09-14T08:37:49Z")

</div>

As far as I know most processing takes place in Elasticsearch ingest pipelines which means you need to send the data to Elasticsearch in order to use the modules.

---

<div class="post-metadata">

**Author:** ![Travis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/travis/32/54079_2.png) [@Travis](https://discuss.elastic.co/u/Travis)\
**Post date:** [September 15, 2020, 10:12am UTC](https://discuss.elastic.co/t/filebeat-modules-and-parsing-with-logstash-cisco-related/246027/6 "2020-09-15T10:12:13Z")

</div>

Thank you Christian.

I solved my issue (Filbeat =\> Logstash (only act as a simple input/ouput gateway) =\> Kafka =\> Logstash (for event transformation) =\> Elasticsearch), put it here if it can help :

- On the logstash which act as a simple input/ouput gateway :

Filebeat.conf :

```
input {
  beats {
    port => 5044
  }
}

filter {
  mutate {
 #copy metadata to keep them in the transformation logstash as metadata are not sent in the output
     copy => { "@metadata" => "metacopy" }
  }
}

```

Kafka.conf :

```
output {
  kafka 
    topic_id => ["kafka"]
    bootstrap_servers => "sever1:9092,server2:9092,server3:9092"
    codec => json
  }
}

```

- On the logstash for events transformation :

main\_pipeline.conf :

```
input {
  kafka {
    topics => ["kafka"]
    bootstrap_servers => "sever1:9092,server2:9092,server3:9092"
    consumer_threads => 2
    codec => json
    decorate_events => true
  }
}

filter
{
}

output {
  if [metacopy][pipeline] {
    elasticsearch {
      hosts => "https://elastic:9200"
      manage_template => false
      index => "%{[metacopy][beat]}-%{[metacopy][version]}-%{+YYYY.MM.dd}"
      pipeline => "%{[metacopy][pipeline]}"
      user => "elastic"
      password => "password"
    }
  } else {
    elasticsearch {
      hosts => "https://elastic:9200"
      manage_template => false
      index => "other"
      user => "elastic"
      password => "password"
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 13, 2020, 12:12pm UTC](https://discuss.elastic.co/t/filebeat-modules-and-parsing-with-logstash-cisco-related/246027/7 "2020-10-13T12:12:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.