Connect logstash to elasticsearch

Elastic Stack Logstash
1

Hello!

I can see in logstash logs that it is not able to connect to elasticsearch to provide data
The ELK documentation says:

Copy the self-signed CA certificate from the Elasticsearch config/certs directory.
Save it to a location that Logstash can access, such as config/certs on the Logstash instance.

I checked Elasticsearch /config/certs directory and there are 3 files there:
http_ca.crt http.p12 transport.p12
As indicated in Docs created config/certs directory and copied files from elasticsearch to the /config/certs dir in logstash directory

changed my logstash.conf to use crt from elasticsearch like:

input {

	beats {
		port => 5044
		ssl => true
		ssl_certificate => "/etc/ssl/logstash_frwrd.crt"
		ssl_key => "/etc/ssl/logstash-forwarder.key"
		}
}

###################################################################################################
filter {
	if [type] == "syslog" {
				grok {
					match=>{ "message" => "%{SYSLOGLINE}" }
                                     }
				date {
					match => [ "timestamp", "MMM d HH:mm:ss",  "MMM dd HH:mm:ss" ]
	     			     }
			      }
	}
##################################################################################################
output {
	elasticsearch {
			hosts=> ["https://localhost:9200"]
			cacert => '/etc/logstash/config/certs/http_ca.crt'
			index=>"%{[@metadata][beat]}-%{+YYYY.MM.dd}"
			#user => "elastic"
			#password => "M+tKhvNwqIQgI3JOX-b+"
			#index=>"%{[@metadata][beat]}-%{[@metadata][version]}"
		      }

	stdout {
		codec => rubydebug
       	       }
       }

not able to start logstash after changes were applied. It used to work before but was not able to send data to elasticsearch

What I'm doing wrong?

2

that's the error message I can see in logstash log though I have

-rw-r----- 1 elasticsearch elasticsearch  1915 Dec  1 14:18 http_ca.crt
-rw-r----- 1 elasticsearch elasticsearch 10077 Dec  1 14:18 http.p12
-rw-r----- 1 elasticsearch elasticsearch  5822 Dec  1 14:18 transport.p12

[2022-12-01T14:47:05,497][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Errno::EACCES: Permission denied - /etc/logstash/config/certs/http_ca.crt>, :backtrace=>["org/jruby/RubyIO.java:1227:in `sysopen'", "org/jruby/RubyFile.java:362:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "org/jruby/RubyIO.java:1146:in `open'", "org/jruby/RubyKernel.java:287:in `open'", "/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/open-uri.rb:37:in `open'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:704:in `setup_trust_store'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:692:in `ssl_socket_factory_from_options'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:454:in `pool_builder'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:462:in `pool'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/manticore-0.9.1-java/lib/manticore/client.rb:227:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:26:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:329:in `build_adapter'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:345:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:39:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch.rb:296:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:234:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:600:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:246:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x49704f59 run>"}
3

after some changes made - changed the owner of crt file to logstash like this

[root@logstash conf.d]# ls -l /etc/logstash/certs/
total 24
-rw-r----- 1 logstash logstash  1915 Dec  1 14:18 http_ca.crt
-rw-r----- 1 logstash logstash 10077 Dec  1 14:18 http.p12
-rw-r----- 1 logstash logstash  5822 Dec  1 14:18 transport.p12

now I have this error message in logstash log:

[root@logstash conf.d]# tail -f /var/log/logstash/logstash-plain.log  | grep error
[2022-12-01T16:52:01,889][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'https://localhost:9200/'"}

my logstash.conf looks like this now:

input {

	beats {
		port => 5044
		ssl => true
		ssl_certificate => "/etc/ssl/logstash_frwrd.crt"
		ssl_key => "/etc/ssl/logstash-forwarder.key"
		}
}

###################################################################################################
filter {
	if [type] == "syslog" {
				grok {
					match=>{ "message" => "%{SYSLOGLINE}" }
                                     }
				date {
					match => [ "timestamp", "MMM d HH:mm:ss",  "MMM dd HH:mm:ss" ]
	     			     }
			      }
	}
##################################################################################################
output {
	elasticsearch {
			hosts => ["https://localhost:9200"]
			#user => "elastic"
			#password => "M+tKhvNwqIQgI3JOX-b+"
			#index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
			cacert => '/etc/logstash/certs/http_ca.crt'
			#ssl => true
			#index=>"%{[@metadata][beat]}-%{[@metadata][version]}"
		      }

	stdout {
		#codec => rubydebug
       	       }
       }
4

This is an authentication error, you need to provide the username and password.

Your logstash config has the user and password commented, you need to check if those are the correct credentials and uncomment it.

5

I thought it is 2 different ways to connect to Elasticsearch, but OK I have added username and pass to configuration and looks like the 401 problem not in the logs any more but I still have some exception which privents me from seeing logs in kibana I think

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 3257000008003243000215bc785eec7d6993e3485e773e8f96fbbe6fc8d57254cff890645996c5d6cc541f3351d03ddd41f7ec76b035d1215be92a45db5621c9d5d53bf43b0822388365b92208ee0502780344001f854f43107f654a4acbb20e4b96edea9a379e72cbb632f377e53f5392f24b08a1ff8790e07c267ee0db33e2f9e6ec523444455294b6acb425e599dc37a49e21f73b7d75f8ab624bfc60467cd3327d53343e1347c4f445439cd8533222a62fb644fff525110df185e58cc59678455ccf76e6a221ea9d7e4711dfb444f39ccc7dd1f84c9c9b333872b2fc41eeab921f6e89e4f282cc886b4e5fd89668889a35d2a5b1aab42763b5df562766bf3db264b93d31d5b13622236d321e8b2d3138763219f54d4b19b6f5a1a2b7554551da7a6f326a0f648d0c263d495735134e6fea9c43c3e034e0f5d2f42f4443ec5e996e77ea9c779f3bee7947eac0516f5aa2339978c4170de94d4b9c11cf33cfa1455fc118e39ed45125e5530cbf3fbf5c042de69ae8f92e3167f01d64ecc10ff18d953a12fccb85e3051f1b3b73dfb4e7c4b5bf4a2cd19898538fb444fb5234be22ca43a5236bf0114d125be2c4922463a2cb963119488635960d75a41864a44a06d107b231d107698729863a52558348a4670c4613d9188fa0e32644970c43938643c3d4877d439e287d8328b226b6a21f9615a5238b9fb6c49939861392a4b674afad0cdb27bdb672bfad9e882db1afb4fb6a5b92da2727f0f65087e313c30f8d4dbc65bae30bdb27637fe142b75eebda0b4d155ba21374d84be2cec95434c45e47963a525b9635a943a6834e74dcd8b108fbca7b8e4b62704eedf9e25a6c899753d39f38ee4c34c43199fb8eb784d8013e828fdd816e3567f6f4b568882eb12e02b486df4be6fee3a7f861f0856f18d44ca5674e6455d2872651279a6a2a03b5474692d2eb2bea6444c4376f02e2fd7f8484eb5ce20d3ada40a99578db6651d43be14072b80fce3bf5f4d6f300c8667c16914d565b85e8b9c4c9e79dc7ee397e8e9f12f78ab858ee285207b094cd4d0065f0d3c1a91d30e25686e4666a0a8c57a2a1e922c218282024fcc62d038b3210da0a7428668a3d6d9981f889ebf8ced899e22fd1dfc6b2dcc2bf42ae6c502a0cbe54868ddb65566b03b6bf596159903c72ede0968d948d9f4348f8bbe6d898b09581b481ad2c45bdbb0b7b6ae1c797c4357d7b7e8e9fbef67c323330660965d81f74e44e8f0b2975a4c304e4a855d5ec339bb021b27dcad0c2e1e8960d7b1ed56878fd268484ff699cacf011e897620e24f7fa4b16b4c4d67b0bd725733f85af419ac7c12085d44d883ffe828c9f3e7a821f3973fc78ec63798865cd9075a33fc49f3cbb8715499170c48c6c575b9f7b21cd4063e135c1f2f45473cbfebd653feb1b3a7e87c1f06f46e873ef35c7f07c1e2cf5e16db82a18ae425160bdb754b1a881d915aa2810a522a763a717b68565070e40f15752c834590d0c33a2d257aa86c45f0ea61578eccc66e6dcc2537b4e0c7cf7f1e3672f4e1f9d7cf4e0b87b3583c0f4d5767a590bbb8ee31f772d72d59d999797c4edd2ca551bdec7ae83c7aee95dd0513d3617be83bd4b32f65df2e24a3976897fe9c02f62d7ea4caf669de9d531fd78977e3cf9aef7cabcc4eec5f908ffdac2263e7e78f2f147c764fee293a79d4f9e7dd8d6818519c54d9a14be0521e14ff63c29a803797d52a0b9febee913034b323eb974ed29b5782cf50d493594e1c9a39cae88dd3c01f4b7a6e412ea50d355d97c8dab3835e1dd3c10a2424b0d8948b03dc5cb28ec507e7e2b42c29f37cecf5052fb4a6549a5fc3cbd6fe06bc73d6f5fcb72dbe3cbbc6d395818c89f6c0396e134ca2e65c0e712a3b92ea0270ebba194d83759e3299a84e4f673c4caa827a99e4bd16f4348f85a73148d4259f2dff6a3af58e10ede02f614ab05f407cbe5e82505096b01acbdd899e04bfb7a66ce0d2c757a6a27bf369d3f7bb99d8604d390b88732660611639a98ad94d03a7656149bcc3ebf1d21e11bcd7113bc06504f7b73b5355b3686dc1e2ed19b6f0d239660b32e1bacae48684aafea6cf3ccbf4b268e4bb04b2e1d375895b8749dd194ccbc161e5f90f14b7ce1fb9746b7fbca7e6977ae3b8e7b9e3b9389ec21d12eee60564d8ea148f1bb7dffc8b5d2ef4048f8fa0eac743ffb6a232bd5747d69b27ce6fb0e9e992f09f6162ec1fe85e9e3d7ce025f9857f017c153d3275ee4ae9d5c80e59b69d9849ee87d10d1c45be9a40c137ab10d40f9534d10bd7aa6bf374c6799997e2742c27f34c74e06ff702e3ae8695505f791e9be24ae67e0a376fb4ea0b5c46ae1a377deb98327ae33c363673eb1cf31fc480b1f1d1fdfc11699988ba98f3de2833eb772d901e08504509603f03986bc62a067073791c178abc8482d377b87053f4b657d1f0e14eb4080cdfe5aeb772124fc6573e4053cc3d924baeaeda44590a66ec86218cf837d007e223e64444cbec070e61fbdfb6ea4faf16a4f0b1f7dfef377f0dcf1ed31fc717a7a07db7328789bbeedcccd293840c29874b9b831e52fc87c77b3fbacb24e68bda4ec8edfac41743abb83daebaa81deb4acc74d21e39ea4babefd196282cdabdeba3a11d7079b6c393ff38fbefce53bf895e9ceedf9790b1f3d78700713d775dc163efaf83490006ccf2ea76446e67e1013df7fff0e5ecc5fce9d57f38e48e7abdf8390f05fcd996a7cfa4b05d220a93e74687035f0595af3cfc4167e66cf88813f746d7c9f8c3156b03430fa9ad15761bd1446b7aa18d064ca86ec96a2dba2e8566314eb372ae8bc18043db877aba7092f1ef60605278994c0df8b90f08fcd11989d2d7c244fade2ebd764491e2e8597
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1213) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1280) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	... 17 more

and I have this lines when logstash starts

[2022-12-02T14:32:52,423][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.5.2) {:es_version=>8}
[2022-12-02T14:32:52,463][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-12-02T14:32:52,567][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-12-02T14:32:52,571][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-12-02T14:32:52,597][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-12-02T14:32:52,608][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2022-12-02T14:32:53,326][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x64db7fa4 run>"}
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.