Mapping input and output in configuration

Elastic Stack Logstash
1

I have bumped into very weird thing. "internet" says that I can't just map input and output of logstash config because of single pipeline or whatever. So I have multiple RabbitMQ queues that I want to read and created different indexes per different queues and I can't just do this basic thing without adding some extra unnecessary fields (tags) to my logs messages just to be able to filter by them in output part of the config.

If you guys are doing something with logstash it should be the first thing to do. Who needs other features if one just can't work with multiple log sources without voodoo dancing?

2

We can help you find a way that doesn't require voodoo dancing, but I'm afraid your question was lost among all the ranting.

3

Thanks for replying. Ranting is the result of the lost day for figuring out something that you can't expect at all. And I think I am not the only one:)

Anyway, in a nutshell all my logs are sent to different Rabbit queues. I have rabbitmq input plugin and I need to create different indexes when reading different queues. Sounds more than easy but all my logs go to all my indexes as logstash configs merge. So how can i match
queue => "postman" in input to index => "postman-%{+YYYY-MM-dd}" in output
and
queue => "hubex" to index => "hubex-%{+YYYY-MM-dd}"

4

First, in the input you should consider using metadata fields such as [@metadata][queue] => "postman" . Metadata fields are not passed to Elasticsearch, so they are handy to use to control processing in the Logstash pipeline without adding unnecessary values to the data.

In the output it is as easy as...

output {
    elasticsearch {
        index => "%{[@metadata][queue]}-%{+YYYY.MM.dd}"
        # plus whatever option you need to connect to elasticsearch
    }
}
5

Something killed my Kibana. In chrome console I have loading default index pattern with error message Request to Elasticsearch failed: "Request Timeout after 30000ms"

is there any way to reset Kibana config?

6

is there any way to reset Kibana config?

Are you sure your Elasticsearch is okay? In my experience that kind of timeout can happen when ES is hosed.

7

In addition to what @rcowart said, the rabbitmq input can optionally save various message metadata as metadata fields so depending on your situation you might be able to use one of those fields out of the box (or mutate the field slightly, then use it). See the docs for details.

8

Thanks for reply. index => "%{[@metadata][rabbitmq_properties][routing-key]}-%{+YYYY.MM.dd}" kind of solved the problem. I am surprised there is no queue property, so I have to name the routing key like the queue but it is OK for my case.

Ideally I would like to read any queue for some exchange and create index based on the routing-key or the queue name but if I leave the queue name empty in plugin config, it just creates the queue with random name as per the documentation but not per my expectation :slight_smile:

If you have any idea how to read any queue (so that not to modify the config each time the new queue is created in Rabbit) please advise.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.