# Non-Production Logstash Instance Accepts Filebeat Connection, but not Production **URL:** \ **Category:** Logstash\ **Created:** [July 11, 2019, 8:26pm UTC](https://discuss.elastic.co/t/non-production-logstash-instance-accepts-filebeat-connection-but-not-production/190106 "2019-07-11T20:26:14Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [July 11, 2019, 8:26pm UTC](https://discuss.elastic.co/t/non-production-logstash-instance-accepts-filebeat-connection-but-not-production/190106/1 "2019-07-11T20:26:14Z")
I'm stumped. I have two ELK stacks running - one in production, one in non-production. Each uses the identical config. I have numerous beats reporting into each. When attempting to connect a new filebeat to the production instance of Logstash, I am receiving the following errors: ``` 2019-07-11T15:07:33.473-0500 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://logstash.example.com:5044)): dial tcp 10.21.23.200:5044: connectex: No connection could be made because the target machine actively refused it. 2019-07-11T15:07:41.984-0500 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://logstash.example.com:5044)): dial tcp 10.21.23.200:5044: connectex: No connection could be made because the target machine actively refused it. ``` However, when I connect to the non-production instance, it connects as expected. The `beats.conf` (excluding filters) is: ``` input { beats { port => 5044 } } filter { # Insert filter here... } output { elasticsearch { hosts => ["localhost:9200"] manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" # document_type => "%{[@metadata][type]}" ssl => true cacert => "/etc/elasticsearch/root-ca.pem" user => "logstash" password => "logstash" } } ``` What am I missing? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 11, 2019, 9:40pm UTC](https://discuss.elastic.co/t/non-production-logstash-instance-accepts-filebeat-connection-but-not-production/190106/2 "2019-07-11T21:40:49Z")
> [@DougR](#): > > What am I missing? A firewall blocking the connection? ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [July 11, 2019, 10:38pm UTC](https://discuss.elastic.co/t/non-production-logstash-instance-accepts-filebeat-connection-but-not-production/190106/3 "2019-07-11T22:38:06Z")
Ya - given that they're on separate subnets, I was thinking a possible firewall rule, but I have other beats on the same subnets reporting in as expected. I'll check again though. ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [July 12, 2019, 2:54pm UTC](https://discuss.elastic.co/t/non-production-logstash-instance-accepts-filebeat-connection-but-not-production/190106/4 "2019-07-12T14:54:06Z")
Firewall rules are OK. I started to experience this with another beat as well, which helped me to run this down more easily. It appears that I wrote a bad filter rule, which is being applied to the logs generated from that beat. The filter survived a `logstash -t` check, but triggers a `FATAL` error when it is triggered. I was wrong about the configs being identical - I hadn't added it to my non-prod instance yet. The rule is as follows: ``` grok { match => { "message" => [ "^%{DATESTAMP:[@metadata][_timestamp]}\s+%{TZ:[@metadata][_timezone]}>\s+%{LOGLEVEL:log.level}.*$" ] } tag_on_failure => [] } date { # Match line (below) appears to trigger the issue. match => ["[@metadata][_timestamp] [@metadata][_timezone]", "MM/dd/yyyy HH:mm:ss.SSS ZZZ"] target => "@timestamp" tag_on_failure => [] } ``` The error being generated is: ``` [2019-07-12T09:42:53,836][FATAL][logstash.runner] An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `[@metadata][_timestamp] [@metadata][_timezone]`, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:235)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:748)"]} ``` I'll attempt to fix this and post back, once I've determined that this is(n't) the issue. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 12, 2019, 3:29pm UTC](https://discuss.elastic.co/t/non-production-logstash-instance-accepts-filebeat-connection-but-not-production/190106/5 "2019-07-12T15:29:16Z")
> [@DougR](#): > > "[@metadata][\_timestamp] [@metadata][\_timezone]" I think you will need to do a mutate+add\_field with %{[fieldname]} sprintf references before the date filter. ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [July 15, 2019, 3:02pm UTC](https://discuss.elastic.co/t/non-production-logstash-instance-accepts-filebeat-connection-but-not-production/190106/6 "2019-07-15T15:02:24Z")
Good catch - I determined the same thing, and that's almost exactly what I did to resolve it. But rather than `add_field`, I did: ``` mutate { update => { "[@metadata][_timestamp] => "%{[@metadata][_timestamp]} %{[@metadata][_timezone]}" }} ``` Same result, I just didn't like the idea of adding yet **another** field, when it was going to be dropped anyway with the `[@metadata][...]` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 12, 2019, 3:02pm UTC](https://discuss.elastic.co/t/non-production-logstash-instance-accepts-filebeat-connection-but-not-production/190106/7 "2019-08-12T15:02:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.