Logstash doesn't parse new files in directory

Logstash doesn't parse new logs files in directory
Here's my config

input {

  file{
    path => "/home/user/Documents/bdu/*.json"
    sincedb_path => "/dev/null"
    type => "bdu"
    codec => "json"
 }


}

output {

    if [type] == "bdu"{
	opensearch{  
	hosts => ["http://*****"]
	index => "psdoi-bdu-%{+YYYY.MM.dd}"
	document_id => "%{ID}"
	action => "update"
	doc_as_upsert => true	
        }
      }

stdout { codec => rubydebug }
}

i need it to send every new file in bdu directory to the opensearch, but it works only after changing some files

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

Can you try to use "read" mode and start from "beginning"?

Documentation

2 Likes

I don't see any need for type => "bdu" and remove from output

input {
  file {
    path => "/home/user/Documents/bdu/*.json" # is it documents or Documents?
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
  }
}
output {
stdout { codec => rubydebug }
}

First start with debug, then go to opensearch.If this is multiline JSON, you have to use the multiline codec, check the sample.

that's really works, thank you

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.