Logstash on Unix can't read Windows created Files

Elastic Stack Logstash
1

Running Logstash as a service in a Linux Machine. Trying to send files from a mounted folder location (files created in Windows Environment). It seems that Logstash can't parse any files that were created in Windows, but if I create a new file of the same type, and copy/paste the data that was in the Windows file, it works.

I opened Notepad++, and the file that Logstash CAN read is EOL of Unix (LF). The file that Logstash CANNOT read is EOL of Windows (CR LF).

Any work around for this issue?

I am running ELK on a VM (Ubuntu Guest, and WIndows Host). The mounted folder is mounted via VirtualBox's Shared Folders.

2

I doubt you've diagnosed the problem correctly. What, exactly, did you try?

3

@magnusbaeck I have tried this:

  1. Pointed Logstash to files that are in a shared folder for my VM. Logstash did not parse into ES.
  2. Created a new file (using Ubuntu), copied and pasted the data from one of the existing files in the shared folder. Logstash did parse into ES.
  3. Created a copy of the file created in Ubuntu and pasted that into the shared folder. Logstash saw the new file, and parsed it into ES.
4

Pointed Logstash to files that are in a shared folder for my VM. Logstash did not parse into ES.

What did the Logstash configuration look like? If you start Logstash with increased loglevel, are there any clues (like whether Logstash is in any way considering the files)?

5 
input {
	file {
	    type => "myType"
	    path => "/media/sf_folder/*.json"
	    start_position => "beginning"
	    sincedb_path => "/dev/null"
	    codec => "json"
	}
}

filter {
	json {
		source => "message"
	}
}

output {
	stdout { codec => rubydebug }

	elasticsearch{
		hosts => ["localhost:9200"]
		index => "testindex"
	}
}

It also doesn't work when I changed the "path" to directly point to the Windows-created file.

6

Please answer my second question as well.

7

I can't find any clues. As per my config, Logstash should be considering any file that ends in ".json" in that directory.

When Logstash is running, when I create a file within Ubuntu with the exact same data in it, and place it in the path I set, Logstash will read it. But when I create a file within Windows with the exact same data in it, and place it in the path I set, Logstash will not read it. Very strange issue.

I have given the necessary permissions for Logstash to read files from that directory (necessary for Shared Folders in VirtualBox), and it can read Unix created files that are copy/pasted from Windows.

8

I can't find any clues.

No, but perhaps we can if you post the log.

9

/var/log/logstash/logstash-plain.log:

[2018-05-08T09:00:12,522][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2018-05-08T09:00:12,531][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"}
[2018-05-08T09:00:12,544][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-05-08T09:00:12,544][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-05-08T09:00:12,549][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-05-08T09:00:12,550][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-05-08T09:00:12,561][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]}
[2018-05-08T09:00:12,572][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
[2018-05-08T09:00:12,574][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2018-05-08T09:00:12,583][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"}
[2018-05-08T09:00:12,588][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-05-08T09:00:12,589][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-05-08T09:00:12,593][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-05-08T09:00:12,595][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-05-08T09:00:12,606][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]}
[2018-05-08T09:00:12,615][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
[2018-05-08T09:00:12,615][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2018-05-08T09:00:12,625][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"}
[2018-05-08T09:00:12,637][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-05-08T09:00:12,637][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-05-08T09:00:12,645][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-05-08T09:00:12,647][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-05-08T09:00:12,664][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]}
[2018-05-08T09:00:12,977][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1cb80b3a@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}
[2018-05-08T09:00:13,003][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}

This is after pointing Logstash to read the file directly. I checked Kibana, the document was not ingested into Elasticsearch.

10

Issue resolved. The files were not formatted correctly in a way that Logstash didn't know the EOL.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.