[Help] Multiple pipelines with N logs from filebeat and N servers

Elastic Stack Logstash
1

I have this structure:

                                  __________    __________       ______________
                                 |same name |  |same name |     |   same name  |
                                 |----------|  |----------|     |--------------|
Machine1 > fileabeat with 3 log (| logA.txt |, | logB.log | and |logC.something|)
Machine2 > fileabeat with 3 log (| logA.txt |, | logB.log | and |logC.something|)
Machine3 > fileabeat with 3 log (| logA.txt |, | logB.log | and |logC.something|)
...                              |          |  |          |     |              |
Machine6 > fileabeat with 3 log (| logA.txt |, | logB.log | and |logC.something|)
                                 |----------|  |----------|     |--------------|

The output filebeat is logstash on port 5044 for all filebeat.

My logstash are on other machine and this machine have too elasticsearch and kibana.
On logstash I made three .conf using beat input on 5044 port with grok filter made by my self and as output is elasticsearch.
My logstash.yml have this config:

  pipeline.batch.size: 125
  pipeline.batch.delay: 5
- pipeline.id: dbjus
  pipeline.workers: 1
  path.config: "/path/to/logA.conf"
- pipeline.id: tasks
  pipeline.workers: 1
  path.config: "/path/to/logB.log"
- pipeline.id: info_log
  pipeline.workers: 1
  path.config: "/path/to/logC.something"
  http.host: "127.0.0.1"
  http.port: 9600

And my pipeline.yml have this:

 - pipeline.id: logA
   path.config: "/path/to/logA.conf"
 - pipeline.id: logB
   path.config: "/path/to/logB.conf"
 - pipeline.id: logC
   path.config: "/path/to/logC.conf"

Before this configuration I tested each .conf with only one machine and worked fine. Kibana show me the logs. But when use the configuration above does not work! Kibana do not show the logs.

Probably I do something wrong.

PS.: Sorry for my English. I'm not a speaker of English.

2

Multiple pipelines can not have inputs using the same port so I would recommend creating a single input pipeline where you have a beats input listening to port 5044. In this pipeline you can then look at the data and determine which pipeline should process it before sending it to the correct pipeline using pipeline-to-pipeline communication.

3

[I hope this is my final edit! =D]

I think understood. I will use the distributor patter with some ifs with conf.string. But the example in documentation use [type], I need use the path log. What I need put instead of [type] to use the path log?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.