Is it possible to create multiple index with multiple folder name

guhi_rockky · December 17, 2022, 3:35pm

Am monitoring logs files in different folders. I have 3 folders, folder1,folder2,folder3. Is it possible to Create different index name with name of folders using logstash.?

Rios · December 17, 2022, 4:49pm

Yes, it is possible. Use gsub to extract folder name, then add in [@metadata][dir]
The best is to provide with dir name samples.

output {
   elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "indexname_%{[@metadata][dir]}"
   } 
}

Other possibly is to set with IFs which is more strict and more usable in some cases.

output {
 if ([@metadata][dir]=="dir1"){
   elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "indexname_%{[@metadata][dir]}"
   } 
 }
 else  if ([@metadata][dir]=="dir2"){
    elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "indexname_%{[@metadata][dir]}"
   } 
 }
 else  if ([@metadata][dir]=="dir3"){
    elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "indexname_%{[@metadata][dir]}"
   } 
 }

Badger · December 17, 2022, 4:53pm

You can use grok to extract a directory name from a path. Examples are here and here. Depending on your versions and ECS compatibility the path of the file will likely be in [path] or, more recently [log][file][path]

guhi_rockky · December 17, 2022, 4:57pm

Hi buddy, thanks for your reply.

My directories structure

/apps/app1/logs/SystemOut.log
apps/app2/logs/SystemOut.log
apps/app3/logs/SystemOut.log

I tired with following pipeline. But its not working

input {
file {
path => "/folder1/logs/server1/SystemOut.log"
start_position => "beginning"
}
}

filter {
if [path] == "/folder1/logs/server1/SystemOut.log" {
grok {
match => { "message" => [
"[%{DATA:timestamp1}]%{SPACE}%{WORD:value1}%{SPACE}%{WORD:value2}%{SPACE}%{WORD:value3}%{SPACE}%{TIMESTAMP_ISO8601:timestamp2}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{WORD:value4}:%{NUMBER:value5}%{SPACE}-%{SPACE}%{GREEDYDATA:value6}"
] }
}
}

}

output {
if [path] == "/folder1/logs/server1/SystemOut.log" {
opensearch {
hosts => ["https://url:443"]
index => "ee"
user => "in"
password => "dm3"
}
}

system · December 17, 2022, 4:57pm

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

guhi_rockky · December 17, 2022, 4:58pm

Hi buddy, thanks for your reply.

My directories structure

/apps/app1/logs/SystemOut.log
apps/app2/logs/SystemOut.log
apps/app3/logs/SystemOut.log

I tired with following pipeline. But its not working

input {
file {
path => "/folder1/logs/server1/SystemOut.log"
start_position => "beginning"
}
}

filter {
if [path] == "/folder1/logs/server1/SystemOut.log" {
grok {
match => { "message" => [
"[%{DATA:timestamp1}]%{SPACE}%{WORD:value1}%{SPACE}%{WORD:value2}%{SPACE}%{WORD:value3}%{SPACE}%{TIMESTAMP_ISO8601:timestamp2}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{WORD:value4}:%{NUMBER:value5}%{SPACE}-%{SPACE}%{GREEDYDATA:value6}"
] }
}
}

}

output {
if [path] == "/folder1/logs/server1/SystemOut.log" {
opensearch {
hosts => ["https://url:443"]
index => "dev-logs"
user => "in"
password => "dm3"
}
}

guhi_rockky · December 17, 2022, 5:08pm

Orelse Is it possible to add tag in new field with folder name???

Rios · December 17, 2022, 5:41pm

This has been working on my simplified version with files as output. Try something like this:

input {
  file {
   path => "/apps/app*/logs/SystemOut.log"
   start_position => beginning
   sincedb_path => "/dev/null" # change to keep records or do not use NUL in prod
   mode => "tail"
  }
}

filter {

  grok {  match => { "[log][file][path]" => "%{GREEDYDATA:dir}/%{DATA:[@metadata][dest]}/%{DATA}/%{GREEDYDATA}" } }

}

output {
 stdout {codec => rubydebug { metadata => true} } # test to see results, latter remove or comment

    elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "indexname_%{[@metadata][dest]}"
   }
}

guhi_rockky · December 17, 2022, 5:47pm

Thanks rios, let me try this and update here.
One more question in this.
Is it possible to add tag or extract a new field with folder name.?
If you give any example for this i will try that one also.

Rios · December 17, 2022, 5:53pm

Yes, but I don't see point of tag because every tag will be recorded in index, except if you want 1 single index with extra tag or field to know the source, for example app1 directory.
Save the space in big data use @metadata

      mutate {
        add_tag => [ "%{[@metadata][dest]}" ]
      }

guhi_rockky · December 17, 2022, 6:02pm

Might be i have added wrong question, right now i have one field name "path" ( /apps/folder1/logs). My question is, can we add that "folder1" name in new filed???

Rios · December 17, 2022, 6:12pm

mutate {
        add_field => {
          "%{[@metadata][dest]}" => "%{[@metadata][dest]}" # v1
          "source_%{[@metadata][dest]}" => "%{[@metadata][dest]}" # v2
        }
}

or do not use temp [@metadata][dest], use just source: %{DATA:source}

guhi_rockky · December 17, 2022, 6:13pm

Let me try rios, thanks mate.

Rios · December 17, 2022, 6:16pm

Also you can use this to avoid grok, rarely rare cases.

guhi_rockky · December 18, 2022, 5:00am

may i try like this ??

input {
file {
path => "/folder1/logs/server1/SystemOut.log"
start_position => "beginning"
sincedb_path => "nul"
}
}

filter {
if [path] == "/folder1/logs/server1/SystemOut.log" {
grok {
mutate { copy => { "[log][file][path]" => "[@metadata][path]"}}
mutate { split => { "[@metadata][path]" => "/" } }
mutate { add_field => { "folder1" => "%{[@metadata][path][4]}"}}
}
}

}

output {
opensearch {
hosts => ["https://url:443"]
index => "dev-logs"
user => ""
password => ""
}
}

Rios · December 18, 2022, 9:05am

This will not work because it is [log][file][path], not just path
if [log][file][path] == "/folder1/logs/server1/SystemOut.log"
But...since you have change the position in path, should use this:

input {
 file {
 path => "/folder1/logs/server1/SystemOut.log"
 start_position => "beginning"
 sincedb_path => "nul"
 }
}

filter {

  grok {  match => { "[log][file][path]" => "%{GREEDYDATA:dir}/%{DATA:[@metadata][dest]}/%{GREEDYDATA}" } }

  mutate { add_field => { "folder1" => "%{[@metadata][dest]}"}}

}

output {
opensearch {
hosts => ["https://url:443"]
index => "dev-logs"
}
}

guhi_rockky · December 18, 2022, 10:24am

Thanks buddy, its working fine.

Rios · December 18, 2022, 11:43am

Good.

"NUL" on Windows or "/dev/null" on Linux it to avoid tracking. Set a file, to keep record which lines has been read otherwise you might have duplicated documents. Set something like this:
sincedb_path => "/path/file.db"
Remove dir from grok, it's useless, I left from old code.
grok { match => { "[log][file][path]" => "%{GREEDYDATA}/%{DATA:[@metadata][dest]}/%{GREEDYDATA}" } }
Debugger not need, remove the line stdout {codec => rubydebug...

guhi_rockky · December 18, 2022, 2:43pm

Sure rios, let me do like this

system · January 15, 2023, 2:43pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to create multiple indexs with multiple input in logstash Logstash	8	2881	March 19, 2021
How to handle multiple inputs with Logstash to different indices Logstash	19	19218	July 6, 2017
What is the easiest way to create indexes for different nginx logs in logstash Logstash	5	131	June 27, 2024
How to create multiple index from file beat log input in log-stash config Beats filebeat	9	2988	May 6, 2019
Output data in multiple indices based on fields Logstash	3	2919	January 15, 2018

Is it possible to create multiple index with multiple folder name

Related Topics