Two pipelines for single filebeat input | ELK version (6.5.4)

I am trying to create multiple pipelines first-pipeline and second-pipeline in logstash listening to beats events from the same port but getting an error saying the address in use for the second pipeline and the reason why I am using two pipelines is that I want pipeline.workers: 1 only for selected indexes in which sequence of log matters.
ERROR:

Pipeline_id:second-pipeline

Plugin: <LogStash::Inputs::Beats host=>"127.0.0.1", port=>5044, id=>"7c07a66c7959c1734f6aead8ca456bc7c3b086aafb7b5bd4882ee45e0f3c9fc5", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_4d22b75f-e478-4fbc-b5fe-27ae02ac486b", enable_metric=>true, charset=>"UTF-8">, ssl=>false, add_hostname=>true, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>8>
Error: Address already in use
Exception: Java::JavaNet::BindException
Stack: sun.nio.ch.Net.bind0(Native Method)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:438)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:430)
sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:225)
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:128)
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:558)
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1283)
io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:501)
io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:486)
io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:989)
io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:254)
io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:364)
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:163)
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:403)
io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:463)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:858)
io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30)
java.lang.Thread.run(java/lang/Thread.java:750)

pipelines.yml

- pipeline.id: first-pipeline
  path.config: "/Users/gyrao/Documents/ELK/logstash-6.5.4/config/pipelines/api-address.config"
  pipeline.batch.size: 1

- pipeline.id: second-pipeline
  path.config: "/Users/gyrao/Documents/ELK/logstash-6.5.4/config/pipelines/my-config.config"
  pipeline.workers: 1
  pipeline.batch.size: 1
  queue.type: persisted
  path.queue: "/Users/gyrao/Documents/ELK/logstash-6.5.4/config/queue"

my-config.config

input {
	beats {
		host => "127.0.0.1"
		port => 5044
	}
}
filter {
	
}
output {
	
}

api-address.config

input {
	beats {
		host => "127.0.0.1"
		port => 5044
	}
}
filter {
	
}
output {
  
}

You can not have multiple input plugins listening on the same port. What you can do is set up a single dedicated input pipeline that contains the input plugin and the logic required to determine which pipeline that should process the event. This pipeline can then send vents to one of your pipelines through pipeline-to-pipeline communication . It seems this feature was Beta in 6.5 so I would recommend you upgrade to at least version 6.8 to ensure you benefit from any early bugfixes.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.