Logstash don't execute my two Pipelines file

Elastic Stack Logstash
1

i thease have two pipeline in the logstash pipeline folder
logstash.conf :
input {
beats {
port => 5066
}
}

filter {
  if [cloud][account][id] == "941682856883" {
    mutate {
      add_field => { "ambiente" => "desenvolvimento" }
    }
  }

  if [cloud][account][id] == "040046441916" {
    mutate {
      add_field => { "ambiente" => "producao/qualidade" }
    }
  }
}

output {
  elasticsearch {
    hosts => "https://elasticsearch:9200"
    index => "metric"
    user => "elastic"
    password => "aCFly1CMvLJJ6gVHyOo4"
    cacert => "/usr/share/logstash/logstash.crt.pem"
    ssl_certificate_verification => false
  }
}

pro.conf :
input {
beats {
port => 5066
}
}

filter {
  if [agent][type] == "filebeat" {
    drop { }
  }
  json {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => "https://elasticsearch:9200"
    index => "prod"
    user => "elastic"
    password => "aCFly1CMvLJJ6gVHyOo4"
    cacert => "/usr/share/logstash/logstash.crt.pem"
    ssl_certificate_verification => false
  }
}

But I think logstash is mixing these two files because the filters in pro.conf works in logstash.conf. and I can't separate these two index files.

My logstash is running in Docker containers and the pipelines files running automatically.

Can anyone help me?!

2

Both your configs are using the same port, you can't do that, you need to use different ports.

What errors do you have in the logstash logs?

Also, what are you using to send logs? It is Metricbeat? Winlogbeat?

3

For the first index I'm collecting logs from metricbeat, and in the second index I'm collecting logs from file .log (with filebeat).

my logstash logs show me this:

[2021-05-28T15:22:13,225][INFO ][org.logstash.beats.Server][main][331a32649800ecc5378a481c17366f1715339dc0c7bf49b83aa2e9ec6bee3dfc] Starting server on port: 5066
[2021-05-28T15:22:19,239][ERROR][logstash.javapipeline    ][main][331a32649800ecc5378a481c17366f1715339dc0c7bf49b83aa2e9ec6bee3dfc] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Beats port=>5066, id=>"331a32649800ecc5378a481c17366f1715339dc0c7bf49b83aa2e9ec6bee3dfc", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_1044ce16-f3db-402c-b66e-a501f28d67e8", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, add_hostname=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>1>
  Error: Address already in use
  Exception: Java::JavaNet::BindException
  Stack: sun.nio.ch.Net.bind0(Native Method)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:455)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:447)
sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:227)
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:134)
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:550)
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1334)
io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:506)
io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:491)
io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:973)
io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:248)
io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:356)
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:164)
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:472)
io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:500)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(io/netty/util/internal/ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30)
java.lang.Thread.run(java/lang/Thread.java:834)
4

As I said, you are using the same port on both pipelines, you can't use the same port, you need to change the port in one of your pipelines.

This is the error you are getting because you are using the same port on both inputs.

  Error: Address already in use
  Exception: Java::JavaNet::BindException

Change the port in one of your pipelines and try again.

Also, your second pipeline that saves in the index prod will drop everything it receives from filebeat, is this what you want?

5

yeah, now I see the point. As I was receiving everything at one port, I decided to separate the ones that came from filebeat and metricbeat using the tag drop. in the second pipeline is suppose to be:

if [agent][type] == "metricbeat" {
    drop { }
  }

and in the first, is suppose to be:

if [agent][type] == "filebeat" {
    drop { }
  }

but now I will put them to come in different port to correct my mistake. thanks for helping me. I am very grateful to have this community available. thanks!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.