Can't use two ports and 2 types logs using beats

Elastic Stack Beats
1

Hi All,
I have configured two conf files using the followings:

input {
beats {
port => 5044
type => "logbeat_general"
}
}

input {
beats {
port => 5045
type => "openlogtst"
}
}

And configured the output conf as follow:
output {
if [type] == "openlogtst" {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "open_log_tst-%{+YYYY.MM.dd}"
document_type => "sivanbeat-openstash"
}
} else {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "logstash-%{+YYYY.MM.dd}"
document_type => "filebeat-appstash"
}
}
}

The idea was to be able to log different logs using beats on different servers with the type separate them and I can do analyzing on each as I want to.

I do not know why But I can only log the filebeat-appstash and even using Port 5045.

can someone assist me with that so i will be able tro understand what i did wrong and how I can fix it?

thank You

2

Do you have monitoring installed so you can check if both input plugins are receiving data?

3

I have beats install on both and I can see that i am getting traffic on both ports
using sudo lsof -i :5044 and 5045 - I can see that ESTABLISHMENT on both when I send the logs.

The problem is that I am getting all the logs under the same type and i can NOT filter them out and use a template for each individual logging.

There is nothing wrong with the communication - this is pure logic configuration issue that i would be happy to resolve as soon as possible so I would be testing more machines and see how that ELK is stashing logs by indexes and patterns.

I would be happy if someone can assist me on that
Thank you

4

Do the documents end up in the correct index? Are you using the default Beats mapping templates?

5

As I have mentioned above:
file one located at conf.d called 02-beats.conf -
input {
beats {
port => 5044
type => "logbeat_general"
}
}
and second file 03-beats.conf-
input {
beats {
port => 5045
type => "openlogtst"
}
}
That I want to get from both ports and have each one with its own type. as mentioned above.
and i have the out like this:
output {
if [type] == "openlogtst" {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "open_log_tst-%{+YYYY.MM.dd}"
document_type => "sivanbeat-openstash"
}
} else {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "logstash-%{+YYYY.MM.dd}"
document_type => "filebeat-appstash"
}
}
}
I wanted to have the output to elasticsearch with the type and index - i.e.
if [type] == "openlogtst" then
index => "open_log_tst-%{+YYYY.MM.dd}"
document_type => "sivanbeat-openstash"
otherwise the default is taking place.
But at this point the logs I get marked as _type: filebeat-appstash and _index: logstash-%{+YYYY.MM.dd} - It looks like the type in the input files is being ignored and If it did NOT had default at all I would not be getting any logs at all.
Any idea why I can't control the logs input using beats the way i do it? I want to be able to modulate inputs for each system and target each log-stash with its own indexing pattern.
I would be happy to get help on how to do that the correct way and make it work.

Thank You

6

Which version are you using?

7

Please properly format logs and config files using the </>-button.

The type field can not be set via the input. See the beats plugin documentation.

You can either tag events in in the input or have the type (or other custom field) being set in filebeat itself.

Beats use the @metadata field to pass information like index and other information to logstash. The @metadata field is dropped by all outputs.

You can easily simplify your output configuration by storing and re-using the index name in @metadata.

Elasticsearch is removing support for _type. That's why beats and logstash set the document type to docs by default. You can still have a type field without setting the document_type.

8

version 5.6.2 - would be upgrading to version 6.0 when I understand how? :slight_smile:

9

OK. Will do from now on.
I have setup the document_type on the yml of the filebeat on the server and that I got the type correctly and output the correct index.
In order to undersatnd it more - Each server that send beats to ELK need to be pre-configured with the document_type before hand? The control of the indexing type is on the side of the SERVER and not the ELK, correct?

In order to change the type I should change the YML file on the SERVER senduing the logs and I ca not do that from within the ELK itself.

I would be happy if I can get the following to work - I want to have three indexes seperated using type on each.
One is coming using Filebeats the second is coming using http (API calls) and the last should be the default .

I have tried with
if [type]=="filebeat"{.....} else if [type]=="apicall"{......} else {......}
I could not catch the http_input_plugin type when I configured it in the input config file.
How can I do that?

Thanks,
Arye

10

You can use tags or type in the http input.

For debugging, it's sometimes helpful to replace the output section with

output {
  stdout {
    codec => rubydebug
  }
}

and test with a few events of one or the other kind only. This get's you an idea about fields being actually available.

Using ruby one can even add some debug output in between filters:

filter {
  ...
ruby {
            init => "require 'json'"
            code => "puts 'myfilter'; puts JSON.pretty_generate(event); puts '=' * 80"
}
 ...
}

This will print:

myfilter
{....} # <- json encoded event
================================================================================

to console.

11

This topic was automatically closed after 21 days. New replies are no longer allowed.