Filebeat -> Logstash -> ElasticSearch with multiple outputs

Elastic Stack Elasticsearch
1

Hi team,
I am trying to send output to two different index by taking two different inputs in tcp and beats. I am able to send logs to elasticsearch which are coming from filebeat but I am not able to send the logs to logstash index which are coming from a different source in form of tcp. Below is the configuration for the same. Am I doing something wrong?

input {
    tcp {
        port => 9600
        tags => ["tcp"]
        type => "tcp"
    }
    beats {
        port => 5044
        tags => ["beats"]
        type => "beats"
    }
}

filter {}

output {
        if "beats" in [tags] {
                elasticsearch {
                        hosts => ["localhost:9200"]
                        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
                        }
                }
       if "tcp" in [tags] {
                stdout {
                        codec => json
                }
                elasticsearch {
                        hosts => ["localhost:9200"]
                        index => "logstash-%{+YYYY.MM.dd}"
                        }
                }
        }
2

Do you have data coming in through TCP? Do you get output to stdout? Are there any filters that could be causing problems?

Document types are being deprecated in Elasticsearch so I would recommend not setting the type in the inputs as this could cause problems, at least down the line.

3

Hi @Christian_Dahlqvist

No data is coming in through TCP. So, no output to stdout. I am confused why it's not coming. Sending you the part of the configuration file from where I am sending the logs to logstash:

         "logstash": {
             "level": "DEBUG",
             "class": "logstash_async.handler.AsynchronousLogstashHandler",
             "transport": "logstash_async.transport.TcpTransport",
             "formatter": "simple",
             "host": "Logstash host",
             "port": 9600,
             "version": 1,
             "message_type": "python-logstash",  # "type" field in logstash message; default value: "logstash"
             "fqdn": False,  # fully qualified domain name; default value: false
             "extra_prefix": "dev",
             "tags": ["tcp"],
             "database_path": "{}/logstash.db".format(BASE_DIR),
         }

There are no filter set as of now.

Which process would you recommend instead of Document type? I tried other options too, but nothing is working. Can you please help?

4

Hey,

It would be helpful if someone can guide us

5

If no data is coming in on TCP I assume you need to check the systems sending that data and check progress step by step. I am not sure how much we can help with that.

6

We are getting data in beats but not in tcp.

What might be the possible problem?

7

Check the Logstash logs at startup to see if there is any issue with the use of port 9600. It might also be worth changing it to something else and see if that helps as well.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.