I have the following logstash conf file: input { file { path => "C:\Dashboard\Elmah\*.xml" start_position => "beginning" type => "error" codec => multiline { pattern => "^<\?error .*\>" negate => true what => "previous" } sincedb_path =>"C:\Da…

If you have already run LS on these files before then it's likely to be a sincedb issue.

Thanks Mark. I am a elasticsearch newbie so please forgive me if my questions are very basic. I have added a sincedb_path => "C:\Dashboard\Elmah" to the file plugin but I still do not see any documents in elasticsearch. How do we debug these kinds of issues?

Have you tried to process these files with LS before?

I have not. This is my first attempt to parse xml files into LogStash so it can be stored in ElasticSearch. My understanding is that this should be a common usecase.. Would you have any additional pointers on what could be wrong? Before this, I was able to successfully parse and log files into …

I suspect it could be a problem of the multiline codec not emitting the first message because it's waiting for the start of the second message but the second message never comes since the file only contains a single message. The file/multiline combination just isn't very good at slurping a whole fi…

Thanks Magnus. In this case, what could you suggest? If nothing else works, I will write a .Net client to insert the data into ElasticSearch directly. I was trying to avoid writing a program to process millions of xml files, and also keep track of which files, folders have been parsed.

The main problem here is the multiline filter, so if you could just get the contents of your XML files on a single line you'll be okay. That should be easy as long as the files aren't too big (not sure if there are any problematic line length limitations). You could e.g. write a small piece of code …

Magnus Back: For debugging purposes, I have kept just one file in the input "C:\Dashboard\Elmah" folder. The data in that file is on just one line. LogStash still does not parse and insert the data into elasticsearch. Would you have any further inputs on what could be wrong? Sample file: <error er…

Make sure you have start_position => beginning, and keep in mind that this only matters for unseen files. You may have to delete the associated sincedb file or create a new file that's covered by the input's filename pattern. Also, keep ES out of the picture for now. Make sure Logstash is reading t…

[image] magnusbaeck: d that this only matters for unseen files. You may have to delete the associated sincedb file or create a new file that's covered by the input's filename pattern. Also, keep ES out of the picture for now. Make sure Logstash is reading the file(s) and producing well-formed m…

Parsing XML files using LogStash

Elastic Stack Logstash

AjitGoel (Ajit Goel) July 17, 2015, 8:38pm 11

It seems that LogStash is logging the "Failed to install template: Connection refused: connect" error when it is trying to process the xml files. Please see the output log when I run the logstash command with the --debug parameter.
https://www.dropbox.com/s/g7g1154uvf9fr1f/outputlog2.txt?dl=0

Any inputs on what could be wrong?

Topic		Replies	Views
Help with logstash and XML Logstash	5	1202	July 6, 2017
Unable to load xml file in logstash Logstash	5	3414	November 21, 2018
Parsing XML files with Logstash in Windows Logstash	6	3129	July 6, 2017
Not able to parse xml Logstash	4	291	September 17, 2018
Log file parsing and ingesting into ES Logstash	15	1739	October 6, 2020

Parsing XML files using LogStash

Related topics