How to create multiple Indices of elasticsearch using logstash configuration

Elastic Stack Logstash
1

Hello everyone,

I am trying to create different indices of elasticsearch using logstash but i am not able to do this.can anyone help me out with this.How we can create and send multiple different index to elasticsearch.

Thanks in advance.

2

What have you tried so far? What are your inputs?

3

Hi Christian Dahlqvist,

Thanks for replying,

Now I can able to create multiple indices.

4

Hi Christian,
I created two different droplets in Digitalocen and install nginx and filebeat on those two droplets.I configured my filebeat.yml file as below:

VM1 filebeat.yml:_

filebeat.prospectors:

  • type: log
    paths:
    • /var/log/nginx/.log
      filebeat.config.modules:
      path: ${path.config}/modules.d/      .yml
      reload.enabled: true

setup.template.settings:
index.number_of_shards: 3

output.logstash:

The Logstash hosts

hosts: ["134.209.153.109:5045"]

VM2 filebeat.yml:_

filebeat.prospectors:

  • type: log
    paths:
    • /var/log/nginx/.log
      filebeat.config.modules:
      path: ${path.config}/modules.d/      .yml
      reload.enabled: true

setup.template.settings:
index.number_of_shards: 3

output.logstash:

The Logstash hosts

hosts: ["134.209.153.109:5044"]

My Logstash vm1.conf file:

input {
beats {
port => 5045
}
}
filter {
if [fileset][module] == "nginx" {
if [fileset][name] == "access" {
grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}""] }
remove_field => "message"
}
output {
elasticsearch {
hosts => ["134.209.153.109:9200"]
manage_template => false
index => "vm1_nginx-1"
}
}

My Logstash vm2.conf file:

input {
beats {
port => 5044
}
}
filter {
if [fileset][module] == "nginx" {
if [fileset][name] == "access" {
grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][referrer]}" "%{DATA:[nginx][access][agent]}""] }
remove_field => "message"
}
output {
elasticsearch {
hosts => ["134.209.153.109:9200"]
manage_template => false
index => "vm2_nginx-1"
}
}

I configured my file and whhen i run logstash i am getting the error like below

[FATAL] 2020-03-09 05:19:25.972 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

[ERROR] 2020-03-09 05:19:25.988 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Can you please help me out how to solve the problem.Can you help me out what is the meaning " If you wish to run multiple instances, you must change the "path.data" setting"

Thanks in advance

5

There is no need to have multiple Logstash confogurations to do what you are you are requesting. You can even have just a single beats input.

Assuming you have a field (or combination of fields) in the logs that determine which index they are supposed to go to you can use conditionals based on this to control the flow like this (you need to change field references and values):

if [host][name] == "vm1" {
    elasticsearch {
        ...
        index => "vm1_nginx-1"
        ...
    }
} else {
    elasticsearch {
        ...
        index => "vm2_nginx-1"
        ...
    }
}

An even easier way might be to use the elasticsearch output plugin directly to write to different indices based on data in the event.

I see that you are separating data into different indices based on the source. This is generally not necessary as a single index can handle vary large amounts of data. What is driving this requirement?

6

Hi ,

Thanks for reply,

My task is to create different index for my project.My project using technologies like php,Laravel and Nginx,Mysql. So i need to get logs for each one and have to create separate index for it like for Laravel separate index and for Mysql separate index and so on.Because of this i am creating multiple conf files.Can you suggest me the best way this could be done.

7

Did you look at the resources I linked to?

8

Yes i saw those resources and now i am creating single logstash.conf and having multiple elasticsearch outputs .Let me check and I will keep you post.

9

Hi have a doubt should i fill any data in [host][name] fields ??

10

Look at the events you have got in Elasticsearch so you can see the fields that are populated. If f nothing has reached Elasticsearch instead add a stipulation output filter with a rubydebug codec so you can see what is being processed.

11

Ok I that is fine.I want to know should i write any data in

12

sorry for above comment by mistake it has sent>>

if [host][name] == "vm1" {
elasticsearch {

above if condition [host] and [name] fields are there should i have to write my host ip or anything??

13

Did you read both links I provided? If so, what in the examples given there is not clear?

14

Hi I went through those topics and i understood that thanks for that

15

Can you tell me why i am getting error like below

INFO ] 2020-03-09 10:13:50.206 [[main]<beats] Server - Starting server on port: 5044

[ERROR] 2020-03-09 10:13:56.428 [[main]<beats] javapipeline - A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Beats port=>5044, id=>"80e52ac0ad4b661509cb2f6834a82e7666f97274a7d23997b34b29cb7e49cce4", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_55210164-c165-4c19-bdc6-25897016d396", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, add_hostname=>false, ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>2>
Error: Address already in use
Exception: Java::JavaNet::BindException

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.