Unable to index a file in elasticsearch


(נבו ויקסלבאום) #1

i'm using elasticsearch 5.6.0 with logstash and filebeats.

i'm trying to index my files by the pattern "filebeat-" and then the source file name.
when i used a simple string like "hello" the file got indexed correctly. when i tried to add the filename to the index non of it was received in elasticsearch discovery and iwasn't able to find the index.

here is my logstash configfile:
input {
beats {
port => 5044
}
}
filter {
grok {
break_on_match => false
match => { "message" => "some pattern to match file context"
"source" => "some pattern to match filename.log$" "
}
}
mutate { add_field => { "[@metadata][filename]" => "%{filename}" } }
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][filename]}"
document_type => "%{[@metadata][type]}"
user => "elastic"
password => "1234"
}
}

when i used output to stdout with ruby debug i saw that "[@metadata][filename]" had the right name in it.
and the data was parsed as i wanted. only the index is the problem

hope anyone can help.


(Imma) #2

What's your filename? Elastic index names have some restrictions, and that might be an issue.


(נבו ויקסלבאום) #3

my file name is in the format of:

"aaa.bbb.d0.ccc.f1-ddd.eee.csv".

can you see the problem? maybe the dots?


(Imma) #4

Yes, I think you can't use dots or certain special characters. Try changing the index name, no dots.

It'd be easier to debug the problem if you can create the index directly on elastic search via the API (https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html), and see if it gets created or what's the error you get. Then move to logstash.


(David Pilato) #5

Are you sure you want to create one index per file?


(נבו ויקסלבאום) #6

this was very helpful.
i tried my index with the API and saw the error that prevented me from indexing my files. it was a lowercase problem.

thank you very much for your help


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.