Using Logstash to read from a log file and then output each log lines to Kafka topic, cannot understand why it is not putting file content to kafka topic specified in conf file

Elastic Stack Logstash
1

Details of the issue-:
logstash version used-: 8.15.0
Kafka broker version - : 3.8.0- up at port 9092
Machine-: Windows
Following is my logstash.conf-:

input {
file {
path => "C:\Users\LENOVO\Downloads\ms1\logs\xelerate_system.log"
start_position => "beginning"
sincedb_path => "nul"
}
}

output{
kafka {
bootstrap_servers => "localhost:9092"
topic_id => "app-logs"
}
}

i run the follwing command to start the logstash process-:

.\bin\logstash.bat -f C:\Users\LENOVO\Downloads\logstash-8.15.0-windows-x86_64\logstash-8.15.0\config\logstash.conf
After running this , i get no errors on the console or anything, and when i check my kafka topic, i dont see any messgaes/data in the topic as mentioned in conf file. I cannot figure out where i m going wrong, Pls help out.

2

The same issue here, you should change slashes to backslashes:

path => "C:/Users/LENOVO/Downloads/ms1/logs/xelerate_system.log"

3

@Rios , tried it didnt work out, currently i have removed output to kafka and just wanted to see my logs atleast get read via logstash and just print on console, but even that is not happening, pipeline gets started without any errors, but no outputs are printed on console.

here is my revised conf file-: for checking if logstash is able to read file or not`` and print contents on console

input {
file {
path => "C:\Users\LENOVO\Downloads\logstash-8.15.0-windows-x86_64\apachemax.log"
start_position => "beginning"
sincedb_path => "NULL"
}
}

output {
stdout {
codec => rubydebug
}
}

here i am simply trying to output on console, but thats not happening too, attaching the log file too, so anyone can check the contents, may be thats causing the issue(i dont think so).

4

This needs to be NUL in windows, not null or NULL.

Also, what does your file looks like? Does it have just one line?

1 Like
5

Thanks for responding @leandrojmp, corrected the NUL part as suggested but still no outputs , here is a sample of my log file content-:

image
image1362×695 40.5 KB

still no output on console.

6

How do you ran it? As a service or from command line/PShell?
Might be your LS user doesn't have rights to read.
Again, change to:
path => "C:/Users/LENOVO/Downloads/logstash-8.15.0-windows-x86_64/apachemax.log"
sincedb_path => "NUL"

Run it as:
c:\path\logstash\bin\logstash.bat -f c:\path\logstash\conf.d\name.conf --path.settings c:\path\logstash\config

7

Hi @Rios tried the same-: still no output on console-: here is a screenshot of the latest run-: used the same command as suggested-:

image
image1366×709 42 KB

8

Run cmd as administrator and make sure that:

  • path has the forwarding slashes - /
  • sincedb_path is "NUL" and start_position is "beginning"
  • apachemax.log exist in the input-path directory
9

Hi @Rios , i tried the following steps, still issue persists. Any more advises?

10

Enable log.level trace as described here. It's very verbose, but it will tell you what the file input is seeing.

11

[2024-08-26T00:39:07,979][INFO ][logstash.runner ] Jackson default value override logstash.jackson.stream-read-constraints.max-number-length configured to 10000
[2024-08-26T00:39:08,063][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2024-08-26T00:39:12,420][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-08-26T00:39:12,898][INFO ][org.reflections.Reflections] Reflections took 562 ms to scan 1 urls, producing 138 keys and 481 values
[2024-08-26T00:39:14,466][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
[2024-08-26T00:39:14,549][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["D:/Users/LENOVO/Desktop/logstash-8.15.0-windows-x86_64/logstash-8.15.0/logstash.conf"], :thread=>"#<Thread:0x13e7ccf4 D:/Users/LENOVO/Desktop/logstash-8.15.0-windows-x86_64/logstash-8.15.0/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-08-26T00:39:16,547][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.99}
[2024-08-26T00:39:16,628][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2024-08-26T00:39:16,649][INFO ][filewatch.observingtail ][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] START, creating Discoverer, Watch with file and sincedb collections
[2024-08-26T00:39:16,713][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2024-08-26T00:39:32,812][DEBUG][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] writing sincedb (delta since last write = 15)
[2024-08-26T00:39:32,832][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] sincedb_write: NULL (time = 2024-08-26 00:39:32 +0530)
[2024-08-26T00:39:32,834][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] non_atomic_write: {:time=>2024-08-26 00:39:32.812 +0530}
[2024-08-26T00:39:45,983][TRACE][filewatch.discoverer ][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] discover_files {:count=>0}
[2024-08-26T00:39:47,986][DEBUG][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] writing sincedb (delta since last write = 15)
[2024-08-26T00:39:47,989][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] sincedb_write: NULL (time = 2024-08-26 00:39:47 +0530)
[2024-08-26T00:39:47,994][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] non_atomic_write: {:time=>2024-08-26 00:39:47.986 +0530}
[2024-08-26T00:40:01,029][TRACE][filewatch.discoverer ][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] discover_files {:count=>0}
[2024-08-26T00:40:02,033][DEBUG][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] writing sincedb (delta since last write = 15)
[2024-08-26T00:40:02,035][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] sincedb_write: NULL (time = 2024-08-26 00:40:02 +0530)
[2024-08-26T00:40:02,038][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] non_atomic_write: {:time=>2024-08-26 00:40:02.033 +0530}
[2024-08-26T00:40:16,115][TRACE][filewatch.discoverer ][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] discover_files {:count=>0}
[2024-08-26T00:40:17,125][DEBUG][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] writing sincedb (delta since last write = 15)
[2024-08-26T00:40:17,129][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] sincedb_write: NULL (time = 2024-08-26 00:40:17 +0530)
[2024-08-26T00:40:17,133][TRACE][filewatch.sincedbcollection][main][d3bd0640be7e413b63347fd1ce27d6c40e345044c799afbb3928bad816a6bda8] non_atomic_write: {:time=>2024-08-26 00:40:17.119 +0530}

Updated the log level via API, for filewatch
here are the trace logs...any insights? i tried with a csv file, i was able to read and output to console, then similarly i tried it with .log file, the above are the logs for that, it didnt print anything on the console, while doing for the csv i added a filter with a seperator too, a,so worked fine. but with .log files, no filter used, it simply cant output to console, whatever the log content is.?

12

@Badger , please let me know, i hope this will be helpful.

13

here is a screenshot of logs after TRACE enabled-:

image
image1364×721 49 KB

14

LS cannot find any file. You haven't run cmd as administrator

15

@pranchalm
Please do not post screen shots of logs... Text please.

since_db still shows NULL which is NOT correct.

so Logstash is probably not reading your only 1 file because it was probably already read once.

The logs show 0 files discovered so path may not be correct.

Put a simple path to a couple files... Properly set

@Rios Said clearly

path => "C:/Users/LENOVO/Downloads/logstash-8.15.0-windows-x86_64/apachemax.log"
sincedb_path => "NUL"

Or try a simple path with multiple files... C:/tmp/*.log

Try again

16

Tried again, no luck-:

here is the conf file-:

input {
file {
path => "C:/applogs/apachemax.log"
start_position => "beginning"
sincedb_path => "NUL"
}
}

output {
stdout {
codec => rubydebug
}
}

ran as administrator, enabled TRACE logs via API call, the path to log file is also very simple.

i ran the following command from my base logstash folder to start the process, still no output of any sort on console,

.\bin\logstash.bat -f C:\logstash-8.15.0\logstash.conf

TRACE logs are same showing count=>0 and repeating the same thing as in the previous screenshot shared, Any more stuff to try, @stephenb , @Rios , please let me know..thanks for the help guys.

17

Guys i tried by changing the extension of the apachemax.log to apachemax.txt, then i am getting the desired output, data going to console now.

18

It doesn't make sense that extension make issues. Anyway you make it.

LS can read almost every character set except ancient symbols from pyramids or alien communications. However that is not submitted on GitH, yet.