We have the following Filebeat 6.1.3 conf:
filebeat.prospectors: - type: log paths: - /usr/share/filebeat/taifunlogs/server.log output.logstash: hosts: ["xxxxxx.com:443"] bulk_max_size: 999999999 logging.level: debug logging.to_files: false logging.to_syslog: false loggins.metrice.enabled: false logging.files: path: /var/log/filebeat name: filebeat keepfiles: 7 permissions: 0644 ssl.verification_mode: none
and Logstash 6.1.3 pipeline:
input {
http {
port => "5044"
}
}Add your filters / logstash plugins configuration here
output {
elasticsearch {
hosts => ["https://376d1d6ba0b54ecf9ace06b3b8ddd4db.elasticsearch.xxxx.com"]
user => "es3iv3ARd9U6egMrg5"
password => "GeaUxFT5zmMJqUZT"
ssl => true
ssl_certificate_verification => true
codec => "plain"
workers => 1
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
manage_template => true
template_name => "logstash"
template_overwrite => true
}
}
We also configured a TCP routing from 443 to 5044, in fact we can run a test wget from the filebeat machine to the Logstash url (443) and the message is received correctly.
Unfortunately with this configuration this is what we get from filbeat (trying to inject a 173M message):
filebeat_1 | "message": "[#|2019-08-25T18:04:08.335+0200|FINE|Payara 4.1|ch.ergon.taifun.sql.DistributedWorksetSql|_ThreadID=263;_ThreadName=allegro/incrementalUpdateConsumer-managedThreadFactory-Thread-5;_TimeMillis=1566749048335;_LevelValue=500;ClassName=ch.ergon.taifun.base.jooq.impl.LoggingExecuteListener;MethodName=logFine;|Query with params : select CE_LOCK.LOCK_NAME, CE_LOCK.SUB_LOCK_NAME, CE_LOCK.LOCKING_FUNCTION, CE_LOCK.HOSTNAME, CE_LOCK.USERW, CE_LOCK.DATEW, CE_LOCK.USERI, CE_LOCK.DATEI from CE_LOCK where CE_LOCK.LOCK_NAME = 'REPORTING_SYNC_LOCK' for update|#]",
filebeat_1 | "prospector": {
filebeat_1 | "type": "log"
filebeat_1 | },
filebeat_1 | "beat": {
filebeat_1 | "name": "filebeat",
filebeat_1 | "hostname": "filebeat",
filebeat_1 | "version": "6.1.3"
filebeat_1 | }
filebeat_1 | }
filebeat_1 | 2019/08/26 09:11:33.420534 processor.go:275: DBG [publish] Publish event: {
filebeat_1 | "@timestamp": "2019-08-26T09:11:33.420Z",
filebeat_1 | "@metadata": {
filebeat_1 | "beat": "filebeat",
filebeat_1 | "type": "doc",
filebeat_1 | "version": "6.1.3"
filebeat_1 | },
filebeat_1 | "source": "/usr/share/filebeat/taifunlogs/server.log",
filebeat_1 | "offset": 2375631,
filebeat_1 | "message": "[#|2019-08-25T18:04:08.356+0200|INFO|Payara 4.1|ch.ergon.taifun.commonentities.lock.LockService|_ThreadID=263;_ThreadName=allegro/incrementalUpdateConsumer-managedThreadFactory-Thread-5;_TimeMillis=1566749048356;_LevelValue=800;|Acquired partial locks unsafely: REPORTING_INC_RUNNING(d-alg-be-11)|#]",
filebeat_1 | "prospector": {
filebeat_1 | "type": "log"
filebeat_1 | },
filebeat_1 | "beat": {
filebeat_1 | "version": "6.1.3",
filebeat_1 | "name": "filebeat",
filebeat_1 | "hostname": "filebeat"
filebeat_1 | }
filebeat_1 | }
filebeat_1 | 2019/08/26 09:11:33.420615 processor.go:275: DBG [publish] Publish event: {
filebeat_1 | "@timestamp": "2019-08-26T09:11:33.420Z",
filebeat_1 | "@metadata": {
filebeat_1 | "beat": "filebeat",
filebeat_1 | "type": "doc",
filebeat_1 | "version": "6.1.3"
filebeat_1 | },
filebeat_1 | 2019/08/26 09:12:05.648135 output.go:92: ERR Failed to publish events: client is not connected
filebeat_1 | 2019/08/26 09:12:05.648197 async.go:94: DBG [logstash] connect
filebeat_1 | 2019/08/26 09:12:05.648619 logger.go:22: INFO retryer: send unwait-signal to consumer
filebeat_1 | 2019/08/26 09:12:05.648644 logger.go:22: INFO done
filebeat_1 | 2019/08/26 09:12:05.648655 logger.go:22: INFO retryer: send wait signal to consumer
filebeat_1 | 2019/08/26 09:12:05.648665 logger.go:22: INFO done
filebeat_1 | 2019/08/26 09:12:05.654675 logger.go:22: INFO retryer: send unwait-signal to consumer
filebeat_1 | 2019/08/26 09:12:05.654698 logger.go:22: INFO done
filebeat_1 | 2019/08/26 09:12:05.658828 async.go:142: DBG [logstash] 295 events out of 295 events sent to logstash host xxxxxxxxx.com:443. Continue sending
filebeat_1 | 2019/08/26 09:12:05.660761 client.go:201: DBG [transport] handle error: EOF
filebeat_1 | 2019/08/26 09:12:05.660799 client.go:114: DBG [transport] closing
filebeat_1 | 2019/08/26 09:12:05.660869 async.go:235: ERR Failed to publish events caused by: EOF
filebeat_1 | 2019/08/26 09:12:05.689668 async.go:142: DBG [logstash] 1973 events out of 1973 events sent to logstash host xxxxxxxx.com:443. Continue sending
filebeat_1 | 2019/08/26 09:12:05.689696 async.go:99: DBG [logstash] close connection
filebeat_1 | 2019/08/26 09:12:05.689705 async.go:99: DBG [logstash] close connection
filebeat_1 | 2019/08/26 09:12:05.689724 async.go:235: ERR Failed to publish events caused by: client is not connected
filebeat_1 | 2019/08/26 09:12:05.689739 logger.go:22: INFO retryer: send wait signal to consumer