Logtash adding filename as index name

Elastic Stack Logstash
1

Hi,
I want the "index" name to be my filename when uploading to elastic via Logtash, Searched widely but unable to get a working solution. Need help how we can use the grok to set the index name as per filename

I tried the below configs but not working

CONFIG: 1
Logtash.conf

input {
file {
path => "C:/logtash/*"
start_position => "beginning"
}
}

filter {
# Drop Elasticsearch Bulk API control lines
if ([message] =~ "{"index") {
drop {}
}

json {
    source => "message"
    remove_field => "message"
}
grok {
    match => [
        "source", "C:\\logtash\\%{DATA:myindex}.json"
  ]
}

}

output {
elasticsearch {
hosts => "localhost:9200"
document_type => "pcap_file"
manage_template => false
index => "%{[myindex]}"
}
}

CONFIG-2
input {
file {
path => "C:/logtash/*"
start_position => "beginning"
}
}

filter {
# Drop Elasticsearch Bulk API control lines
if ([message] =~ "{"index") {
drop {}
}

json {
    source => "message"
    remove_field => "message"
}
grok {
    match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.json"]
}

}

output {
elasticsearch {
hosts => "localhost:9200"
document_type => "pcap_file"
manage_template => false
index => filename
}
}

2 
grok { match => [ "path", "/(?<filename>[^/]+).json" ] }

will pull out the filename, then you can reference it using 'index => "%{filename}"'.

3

You have not explained the use case, but creating a large number of small indexes is not an efficient way to do things.

1 Like
5

Thank you, this worked!!, our usecase is similar to https://www.elastic.co/blog/analyzing-network-packets-with-wireshark-elasticsearch-and-kibana

we want to group the packets based on the packet filename and categorize

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.