# \*beat -\> logstash -\> logstash

**URL:** <https://discuss.elastic.co/t/beat-logstash-logstash/63924>\
**Category:** Logstash\
**Created:** [October 25, 2016, 7:09pm UTC](https://discuss.elastic.co/t/beat-logstash-logstash/63924 "2016-10-25T19:09:43Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![tbmorris78](https://avatars.discourse-cdn.com/v4/letter/t/65b543/32.png) [@tbmorris78](https://discuss.elastic.co/u/tbmorris78)\
**Post date:** [October 25, 2016, 7:09pm UTC](https://discuss.elastic.co/t/beat-logstash-logstash/63924/1 "2016-10-25T19:09:44Z")

</div>

I have been playing with ELK stack for a month or so and seem to have it working quite well at a single site, but I am running into issues of how to move forward.

I have several sites with logstash that I would like to feed into another logstash/elasticsearch at a central site. This would allow logs to be maintained at a site for maybe 30 days with no backups and then logs maintained at a central location for a longer period of time before they are archived.

I don't believe that a elasticsearch cluster is what I am looking for, so I am playing with logstash outputs. Currently, at each site, I have logstash configured to output to both the local elasticsearch and lumberjack. That lumberjack output is being received at my central site in what appears to be the proper format (took me a little bit to realize that I needed codec =\> "json" to get it to do that). The problem is that it is not appearing in elasticsearch at my central site, only the local beats inputs are showing up.

Has anybody set something up like this before and can they assist me?

Thanks!

Terry

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 26, 2016, 10:41am UTC](https://discuss.elastic.co/t/beat-logstash-logstash/63924/2 "2016-10-26T10:41:49Z")

</div>

Providing some of your configs would he helpful 🙂

---

<div class="post-metadata">

**Author:** ![tbmorris78](https://avatars.discourse-cdn.com/v4/letter/t/65b543/32.png) [@tbmorris78](https://discuss.elastic.co/u/tbmorris78)\
**Post date:** [October 26, 2016, 12:13pm UTC](https://discuss.elastic.co/t/beat-logstash-logstash/63924/3 "2016-10-26T12:13:23Z")

</div>

Sorry about that. Here are the relevant files.

Upstream server:

```
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/path/to/my/crt"
    ssl_key => "/path/to/my/key"
  }

  lumberjack {
    port => 2400
    codec => "json"
    type => "downstream"
    congestion_threshold => 40
    ssl_certificate => "/path/to/my/crt"
    ssl_key => "/path/to/my/key"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => ["received_at", "%{@timestamp}"]
      add_field => ["received_from", "%{host}"]
    }
    syslog_pri { }
    date {
      match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
    }
  }

  if [type] == "downstream" {
    json {
      source => "message"
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

```

Downstream server:

```
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/path/to/my/crt"
    ssl_key => "/path/to/my/key"
  }

}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => ["received_at", "%{@timestamp}"]
      add_field => ["received_from", "%{host}"]
    }
    syslog_pri { }
    date {
      match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
    }
  }
}

output {
  lumberjack {
    hosts => ["my.upstream.server"]
    port => 2400
    codec => "json"
    ssl_certificate => "/path/to/my/crt"
  }

  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
```

---

<div class="post-metadata">

**Author:** ![tbmorris78](https://avatars.discourse-cdn.com/v4/letter/t/65b543/32.png) [@tbmorris78](https://discuss.elastic.co/u/tbmorris78)\
**Post date:** [October 31, 2016, 5:18pm UTC](https://discuss.elastic.co/t/beat-logstash-logstash/63924/4 "2016-10-31T17:18:48Z")

</div>

Does anybody have any idea where I am going wrong?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 31, 2016, 10:43pm UTC](https://discuss.elastic.co/t/beat-logstash-logstash/63924/5 "2016-10-31T22:43:16Z")

</div>

What if you add a `stdout`?

---

<div class="post-metadata">

**Author:** ![tbmorris78](https://avatars.discourse-cdn.com/v4/letter/t/65b543/32.png) [@tbmorris78](https://discuss.elastic.co/u/tbmorris78)\
**Post date:** [November 2, 2016, 2:05am UTC](https://discuss.elastic.co/t/beat-logstash-logstash/63924/6 "2016-11-02T02:05:08Z")

</div>

I have added a stdout previously. I compared entires that were added (topbeat going directly to the upstream server) with entries that were not added (topbeat coming from the downstream server) and they were identical. I see no reason why it wouldn't work.

---

<div class="post-metadata">

**Author:** ![tbmorris78](https://avatars.discourse-cdn.com/v4/letter/t/65b543/32.png) [@tbmorris78](https://discuss.elastic.co/u/tbmorris78)\
**Post date:** [March 31, 2017, 7:12pm UTC](https://discuss.elastic.co/t/beat-logstash-logstash/63924/7 "2017-03-31T19:12:11Z")

</div>

Bump

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:27am UTC](https://discuss.elastic.co/t/beat-logstash-logstash/63924/8 "2017-07-06T04:27:23Z")

</div>