# Sending Squid Logs with Filebeat and manipulate with logstash

**URL:** <https://discuss.elastic.co/t/sending-squid-logs-with-filebeat-and-manipulate-with-logstash/167298>\
**Category:** Logstash\
**Created:** [February 6, 2019, 2:06pm UTC](https://discuss.elastic.co/t/sending-squid-logs-with-filebeat-and-manipulate-with-logstash/167298 "2019-02-06T14:06:16Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sdvincent](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@sdvincent](https://discuss.elastic.co/u/sdvincent)\
**Post date:** [February 6, 2019, 2:06pm UTC](https://discuss.elastic.co/t/sending-squid-logs-with-filebeat-and-manipulate-with-logstash/167298/1 "2019-02-06T14:06:17Z")

</div>

Hello,

This might be a newbish question but I haven't been able to find what I've been looking for elsewhere online. I am trying to learn how to send logs that don't have pre-built modules to logstash and have logstash do some formatting and create indexes that I can then send to elasticsearch. I am trying to collect squid logs, mark them as such, and send them to logstash for this formatting with filebeat.

My issue is, it seems like there are no issues with filebeat on the squid server but I'm not seeing anything pass through logstash to elasticsearch. I'm assuming I am tagging the logs wrong and/or I my logstash filter isn't picking them up.

Here is my configuration files:

filebeat.yml on squid server:

```auto
filebeat.inputs:
    - type: log
      enabled: true

      paths:
        - '/var/log/squid/access.log'

      exclude_files: ['.gz$']

      fields:
        type: 'squid'

    output.logstash:
      hosts: ["10.4.4.15:5044"]

```

On ELK server..

02-beats-input.conf

```auto
input {
  beats {
    port => 5044
  }
}

```

12-squid-filter.conf

```auto
filter {
      if [type] == "log" {
        if [type] == "squid" {
          mutate {
            add_field => { "hey_this_works" => "yay" }
          }
        }
      }
    }

```

And for output..I'm troubleshooting by sending locally to a file but will change once I get it working.  
30-elasticsearch-output.conf

```auto
   output {
       file {
            codec => "plain"
            path => "/var/logs/logs-%{+YYYY-MM-dd}.txt"
        }
    }

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 6, 2019, 2:44pm UTC](https://discuss.elastic.co/t/sending-squid-logs-with-filebeat-and-manipulate-with-logstash/167298/2 "2019-02-06T14:44:37Z")

</div>

> [@sdvincent](#):
>
> if [type] == "log" { if [type] == "squid" {

It cannot be both, so this will never execute the mutate.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 6, 2019, 3:00pm UTC](https://discuss.elastic.co/t/sending-squid-logs-with-filebeat-and-manipulate-with-logstash/167298/3 "2019-02-06T15:00:08Z")

</div>

If you are new to Logstash and looking to parse Squid logs, [this introductory blog post](https://www.elastic.co/blog/a-practical-introduction-to-logstash) might be useful.

---

<div class="post-metadata">

**Author:** ![sdvincent](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@sdvincent](https://discuss.elastic.co/u/sdvincent)\
**Post date:** [February 6, 2019, 3:02pm UTC](https://discuss.elastic.co/t/sending-squid-logs-with-filebeat-and-manipulate-with-logstash/167298/4 "2019-02-06T15:02:09Z")

</div>

I figured that was the main culprit. How would I correctly tag this data and have logstash act on it?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 6, 2019, 3:21pm UTC](https://discuss.elastic.co/t/sending-squid-logs-with-filebeat-and-manipulate-with-logstash/167298/5 "2019-02-06T15:21:16Z")

</div>

Unless you are setting fields\_under\_root in filebeat adding a type field will get you an field called "[fields][type]". If you decide to set fields\_under\_root I would suggest _not_ calling the field [type](https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html). Use doctype instead.

```
filter {
    if [fields][type] == "squid" {
        mutate {
            add_field => { "hey_this_works" => "yay" }
        }
    }
}
```

---

<div class="post-metadata">

**Author:** ![sdvincent](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@sdvincent](https://discuss.elastic.co/u/sdvincent)\
**Post date:** [February 7, 2019, 4:40am UTC](https://discuss.elastic.co/t/sending-squid-logs-with-filebeat-and-manipulate-with-logstash/167298/6 "2019-02-07T04:40:29Z")

</div>

Thank you for the post. I wish I stumbled across that last night.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 7, 2019, 4:40am UTC](https://discuss.elastic.co/t/sending-squid-logs-with-filebeat-and-manipulate-with-logstash/167298/7 "2019-03-07T04:40:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.