# Logstash s3 elb config into existing ELK stack

**URL:** <https://discuss.elastic.co/t/logstash-s3-elb-config-into-existing-elk-stack/52914>\
**Category:** Logstash\
**Created:** [June 15, 2016, 7:30pm UTC](https://discuss.elastic.co/t/logstash-s3-elb-config-into-existing-elk-stack/52914 "2016-06-15T19:30:09Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![meathouse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/meathouse/32/10343_2.png) [@meathouse](https://discuss.elastic.co/u/meathouse)\
**Post date:** [June 15, 2016, 7:30pm UTC](https://discuss.elastic.co/t/logstash-s3-elb-config-into-existing-elk-stack/52914/1 "2016-06-15T19:30:09Z")

</div>

I recently set up an ELK stack in AWS with 1 elasticsearch master, and 1 data node. Running fine for a few weeks with filebeat and some basic auth log and syslog filters.

Now I'm trying to add my ELB logs from s3. I have the plugin configured and I can see s3 logs landing in logstash's tmp dir. So I believe that part is fine. I have 2 input config files. Here is the one for s3 bucket:

```
input {
  s3 {
    type => "elb"
    bucket => "loadbalancer-name-comapany.net"
    region => "us-east-1"
  }
}

```

Note I didn't put any credentials in because my EC2 instance is using IAM roles for access to s3.

Next I made a filter. It's one of 3 filter config files I have. I made it based on other examples I found lying about the internet. logstash configtest tells me that the config is OK. logstash also stops/start fine when I have this config file in place. Here it is:

```
filter {
    if [type] == "elb" {
        grok {
            match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:loadbalancer} %{IP:client_ip}:%{NUMBER:client_port:int} %{IP:backend_ip}:%{NUMBER:backend_port:int} %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{NUMBER:elb_status_code:int} %{NUMBER:backend_status_code:int} %{NUMBER:received_bytes:int} %{NUMBER:sent_bytes:int} %{QS:request}"]
        }
        date {
            match => ["timestamp", "ISO8601"]
        }
        # Add geolocalization attributes based on ip.
        geoip {
            source => "ip"
        }
   }
}

```

The part I'm stuck on is if I need a new output block for this or not. My existing config file (I only have 1) with an output block looks like this

```
output {
  elasticsearch {
    hosts => ["10.0.1.60:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

```

I don't think that will handle the new ELB logs from s3... or will it? If it won't do I edit the existing? Add a new one? My symptom is that I don't see any of these new logs available in Kibana. Maybe I need to define the index for the new elb logs somehow? Sorry if this is a noob question, but I am a bit new to ELK. Thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 16, 2016, 3:28am UTC](https://discuss.elastic.co/t/logstash-s3-elb-config-into-existing-elk-stack/52914/2 "2016-06-16T03:28:14Z")

</div>

Given you won't have `[@metadata][beat]` in the data from that input you will probably want to define a different output.

---

<div class="post-metadata">

**Author:** ![meathouse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/meathouse/32/10343_2.png) [@meathouse](https://discuss.elastic.co/u/meathouse)\
**Post date:** [June 28, 2016, 11:19pm UTC](https://discuss.elastic.co/t/logstash-s3-elb-config-into-existing-elk-stack/52914/3 "2016-06-28T23:19:11Z")

</div>

Thanks @warkolm you were right. I blindly copied the index from another config file and it wasn't the right one to use. I changed the index to this:  
`index => "elb-%{+YYYY.MM.dd}"`  
Then gave the services a restart. After a few minutes I was able to start seeing the data show up in Kibana UI. Now on to kibana dashboards...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:50am UTC](https://discuss.elastic.co/t/logstash-s3-elb-config-into-existing-elk-stack/52914/4 "2017-07-06T04:50:37Z")

</div>