# Using Logstash to read from a log file and then output each log lines to Kafka topic, cannot understand why it is not putting file content to kafka topic specified in conf file **URL:** \ **Category:** Logstash\ **Created:** [August 25, 2024, 10:31am UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499 "2024-08-25T10:31:22Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![pranchalm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pranchalm/32/138226_2.png) [@pranchalm](https://discuss.elastic.co/u/pranchalm)\ **Post date:** [August 25, 2024, 10:31am UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/1 "2024-08-25T10:31:22Z")
Details of the issue-: logstash version used-: 8.15.0 Kafka broker version - : 3.8.0- up at port 9092 Machine-: Windows Following is my logstash.conf-: input { file { path =\> "C:\Users\LENOVO\Downloads\ms1\logs\xelerate\_system.log" start\_position =\> "beginning" sincedb\_path =\> "nul" } } output{ kafka { bootstrap\_servers =\> "localhost:9092" topic\_id =\> "app-logs" } } i run the follwing command to start the logstash process-: .\bin\logstash.bat -f C:\Users\LENOVO\Downloads\logstash-8.15.0-windows-x86\_64\logstash-8.15.0\config\logstash.conf After running this , i get no errors on the console or anything, and when i check my kafka topic, i dont see any messgaes/data in the topic as mentioned in conf file. I cannot figure out where i m going wrong, Pls help out. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [August 25, 2024, 1:12pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/2 "2024-08-25T13:12:39Z")
The same issue [here](https://discuss.elastic.co/t/logstash-seems-to-be-working-but-no-indices-in-kibana/364444/3), you should change slashes to backslashes: `path => "C:/Users/LENOVO/Downloads/ms1/logs/xelerate_system.log"` ---
**Author:** ![pranchalm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pranchalm/32/138226_2.png) [@pranchalm](https://discuss.elastic.co/u/pranchalm)\ **Post date:** [August 25, 2024, 2:03pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/3 "2024-08-25T14:03:22Z")
@Rios , tried it didnt work out, currently i have removed output to kafka and just wanted to see my logs atleast get read via logstash and just print on console, but even that is not happening, pipeline gets started without any errors, but no outputs are printed on console. here is my revised conf file-: for checking if logstash is able to read file or not`` and print contents on console input { file { path =\> "C:\Users\LENOVO\Downloads\logstash-8.15.0-windows-x86\_64\apachemax.log" start\_position =\> "beginning" sincedb\_path =\> "NULL" } } output { stdout { codec =\> rubydebug } } here i am simply trying to output on console, but thats not happening too, attaching the log file too, so anyone can check the contents, may be thats causing the issue(i dont think so). ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [August 25, 2024, 2:11pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/4 "2024-08-25T14:11:51Z")
> [@pranchalm](#): > > sincedb\_path =\> "NULL" This needs to be **NUL** in windows, not **null** or **NULL**. Also, what does your file looks like? Does it have just one line? ---
**Author:** ![pranchalm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pranchalm/32/138226_2.png) [@pranchalm](https://discuss.elastic.co/u/pranchalm)\ **Post date:** [August 25, 2024, 2:20pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/5 "2024-08-25T14:20:21Z")
Thanks for responding @leandrojmp, corrected the NUL part as suggested but still no outputs , here is a sample of my log file content-: ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/6/367de65ee627ac3064da25299a95d28c3190778c.png) still no output on console. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [August 25, 2024, 2:29pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/6 "2024-08-25T14:29:51Z")
How do you ran it? As a service or from command line/PShell? Might be your LS user doesn't have rights to read. Again, change to: `path => "C:/Users/LENOVO/Downloads/logstash-8.15.0-windows-x86_64/apachemax.log"` `sincedb_path => "NUL"` Run it as: `c:\path\logstash\bin\logstash.bat -f c:\path\logstash\conf.d\name.conf --path.settings c:\path\logstash\config ` ---
**Author:** ![pranchalm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pranchalm/32/138226_2.png) [@pranchalm](https://discuss.elastic.co/u/pranchalm)\ **Post date:** [August 25, 2024, 3:46pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/7 "2024-08-25T15:46:31Z")
Hi @Rios tried the same-: still no output on console-: here is a screenshot of the latest run-: used the same command as suggested-: ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/b/4b67711eab40b71b4387d5ecec27d50aab0d83ad.png) ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [August 25, 2024, 4:08pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/8 "2024-08-25T16:08:56Z")
Run cmd as administrator and make sure that: - path has the forwarding slashes - / - sincedb\_path is "NUL" and start\_position is "beginning" - apachemax.log exist in the input-path directory ---
**Author:** ![pranchalm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pranchalm/32/138226_2.png) [@pranchalm](https://discuss.elastic.co/u/pranchalm)\ **Post date:** [August 25, 2024, 4:43pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/9 "2024-08-25T16:43:55Z")
Hi @Rios , i tried the following steps, still issue persists. Any more advises? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 25, 2024, 5:00pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/10 "2024-08-25T17:00:47Z")
Enable log.level trace as described [here](https://discuss.elastic.co/t/logstash-wildcards-regex-not-working/164204/2). It's very verbose, but it will tell you what the file input is seeing. ---
**Author:** ![pranchalm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pranchalm/32/138226_2.png) [@pranchalm](https://discuss.elastic.co/u/pranchalm)\ **Post date:** [August 25, 2024, 7:14pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/11 "2024-08-25T19:14:14Z")
[2024-08-26T00:39:07,979][INFO][logstash.runner] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000` [2024-08-26T00:39:08,063][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2024-08-26T00:39:12,420][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600, :ssl\_enabled=\>false} [2024-08-26T00:39:12,898][INFO][org.reflections.Reflections] Reflections took 562 ms to scan 1 urls, producing 138 keys and 481 values [2024-08-26T00:39:14,466][INFO][logstash.javapipeline] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise. [2024-08-26T00:39:14,549][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50, "pipeline.max\_inflight"=\>500, "pipeline.sources"=\>["D:/Users/LENOVO/Desktop/logstash-8.15.0-windows-x86\_64/logstash-8.15.0/logstash.conf"], :thread=\>"#\"} [2024-08-26T00:39:16,547][INFO][logstash.javapipeline][main] Pipeline Java execution initialization time {"seconds"=\>1.99} [2024-08-26T00:39:16,628][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=\>"main"} [2024-08-26T00:39:16,649][INFO][filewatch.observingtail][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] START, creating Discoverer, Watch with file and sincedb collections [2024-08-26T00:39:16,713][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>} [2024-08-26T00:39:32,812][DEBUG][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] writing sincedb (delta since last write = 15) [2024-08-26T00:39:32,832][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] sincedb\_write: NULL (time = 2024-08-26 00:39:32 +0530) [2024-08-26T00:39:32,834][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] non\_atomic\_write: {:time=\>2024-08-26 00:39:32.812 +0530} [2024-08-26T00:39:45,983][TRACE][filewatch.discoverer][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] discover\_files {:count=\>0} [2024-08-26T00:39:47,986][DEBUG][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] writing sincedb (delta since last write = 15) [2024-08-26T00:39:47,989][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] sincedb\_write: NULL (time = 2024-08-26 00:39:47 +0530) [2024-08-26T00:39:47,994][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] non\_atomic\_write: {:time=\>2024-08-26 00:39:47.986 +0530} [2024-08-26T00:40:01,029][TRACE][filewatch.discoverer][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] discover\_files {:count=\>0} [2024-08-26T00:40:02,033][DEBUG][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] writing sincedb (delta since last write = 15) [2024-08-26T00:40:02,035][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] sincedb\_write: NULL (time = 2024-08-26 00:40:02 +0530) [2024-08-26T00:40:02,038][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] non\_atomic\_write: {:time=\>2024-08-26 00:40:02.033 +0530} [2024-08-26T00:40:16,115][TRACE][filewatch.discoverer][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] discover\_files {:count=\>0} [2024-08-26T00:40:17,125][DEBUG][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] writing sincedb (delta since last write = 15) [2024-08-26T00:40:17,129][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] sincedb\_write: NULL (time = 2024-08-26 00:40:17 +0530) [2024-08-26T00:40:17,133][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] non\_atomic\_write: {:time=\>2024-08-26 00:40:17.119 +0530} Updated the log level via API, for filewatch here are the trace logs...any insights? i tried with a csv file, i was able to read and output to console, then similarly i tried it with .log file, the above are the logs for that, it didnt print anything on the console, while doing for the csv i added a filter with a seperator too, a,so worked fine. but with .log files, no filter used, it simply cant output to console, whatever the log content is.? ---
**Author:** ![pranchalm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pranchalm/32/138226_2.png) [@pranchalm](https://discuss.elastic.co/u/pranchalm)\ **Post date:** [August 25, 2024, 7:17pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/12 "2024-08-25T19:17:19Z")
@Badger , please let me know, i hope this will be helpful. ---
**Author:** ![pranchalm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pranchalm/32/138226_2.png) [@pranchalm](https://discuss.elastic.co/u/pranchalm)\ **Post date:** [August 25, 2024, 7:26pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/13 "2024-08-25T19:26:53Z")
here is a screenshot of logs after TRACE enabled-: ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/4/64bf7551c3575d6edc26d34d351c387b7f100200.png) ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [August 25, 2024, 7:32pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/14 "2024-08-25T19:32:42Z")
LS cannot find any file. You haven't run cmd as administrator ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 25, 2024, 8:34pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/15 "2024-08-25T20:34:13Z")
@pranchalm Please do not post screen shots of logs... Text please. `since_db` still shows `NULL` which is NOT correct. so Logstash is probably not reading your only 1 file because it was probably already read once. The logs show 0 files discovered so path may not be correct. Put a simple path to a couple files... Properly set @Rios Said clearly ```auto path => "C:/Users/LENOVO/Downloads/logstash-8.15.0-windows-x86_64/apachemax.log" sincedb_path => "NUL" ``` Or try a simple path with multiple files... `C:/tmp/*.log` Try again ---
**Author:** ![pranchalm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pranchalm/32/138226_2.png) [@pranchalm](https://discuss.elastic.co/u/pranchalm)\ **Post date:** [August 26, 2024, 6:06pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/16 "2024-08-26T18:06:14Z")
Tried again, no luck-: here is the conf file-: input { file { path =\> "C:/applogs/apachemax.log" start\_position =\> "beginning" sincedb\_path =\> "NUL" } } output { stdout { codec =\> rubydebug } } ran as administrator, enabled TRACE logs via API call, the path to log file is also very simple. i ran the following command from my base logstash folder to start the process, still no output of any sort on console, .\bin\logstash.bat -f C:\logstash-8.15.0\logstash.conf TRACE logs are same showing count=\>0 and repeating the same thing as in the previous screenshot shared, Any more stuff to try, @stephenb , @Rios , please let me know..thanks for the help guys. ---
**Author:** ![pranchalm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pranchalm/32/138226_2.png) [@pranchalm](https://discuss.elastic.co/u/pranchalm)\ **Post date:** [August 26, 2024, 6:17pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/17 "2024-08-26T18:17:00Z")
Guys i tried by changing the extension of the apachemax.log to apachemax.txt, then i am getting the desired output, data going to console now. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [August 26, 2024, 6:49pm UTC](https://discuss.elastic.co/t/using-logstash-to-read-from-a-log-file-and-then-output-each-log-lines-to-kafka-topic-cannot-understand-why-it-is-not-putting-file-content-to-kafka-topic-specified-in-conf-file/365499/18 "2024-08-26T18:49:57Z")
It doesn't make sense that extension make issues. Anyway you make it. LS can read almost every character set except ancient symbols from pyramids or alien communications. However that is not submitted on GitH, yet.