Logstash doesnot ingest data when started as a service

Elastic Stack Logstash
1

Hi All,

I am trying to ingest csv data into Elasticsearch by using logstash, when i run the conf file manually the data is getting ingested, but when i add the conf file into pipelines.yml and then start logstash as a service the data never gets ingested.

Logs says logstash is started successfully with the pipelines :frowning:

Is there something i'm doing wrong?
Here is my config file:

input {
  file {
    path => "/opt/bulk_data_csv/debug_data.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
 csv {
separator => ","
columns => ["DID","device_name","device_ip","organization","time_stamp(UTC)","index_label","CPU Utilization"]
 }
 }

output {
   elasticsearch {
   hosts => ["https://demo.es.aws.found.io:9243"]
   user => "****"
   password => "*****"
   index => "ingest_data"
   }
#   stdout { codec => rubydebug }
}

Any advice please.

Thanks
Gautham

2

What do you have in the logs?

How are you running logstash manually?

When you run logstash as a service it runs with the user logstash, you need to make sure that this user has reading permissions to the file you want to ingest.

3

Hi @leandrojmp ,

Below is the data i have in the file

|DID|device_name|device_ip|organization|time_stamp(UTC)|index_label|Available Memory|Memory Usage|Swap Usage|
|---|---|---|---|---|---|---|---|---|
|45|demo1|10.10.1.12|org1|2021-09-29 16:35|None|27667|15.57|13.81|
|76|demo2|10.10.2.14|org2|2021-09-29 16:35|None|5147|37.17|24.79|
|23|demo3|10.10.1.16|org3|2021-09-29 16:35|None|27813|15.12|15.6|
|66|demo4|10.10.1.17|org4|2021-09-29 16:35|None|10174|37.9|49.86|
|88|demo5|10.11.1.18|org5|2021-09-29 16:35|None|5676|30.71|19.94|
|65|demo6|10.11.2.20|org6|2021-09-29 16:35|None|22664|30.83|36.07|
|99|demo7|10.11.2.21|org7|2021-09-29 16:35|None|51483|21.44|28.37|
|87|demo8|10.11.2.22|org8|2021-09-29 16:35|None|2369|42.15|19.62|

How are you running logstash manually?

  • /usr/share/logstash/bin/logstash -f demodata.conf

I'm running logstash as root user only, so it has all the permissions i guess.

Thanks
Gautham

4

If you installed logstash using a package manager like apt or yum, and you are running it as a service, you are not running it as the root user, but as the logstash user, unless you changed the service file.

Can you check which user is running logstash when you start it as a service?

If you run it manually as the root user it will have permission to the folder, but it may not have permissions when you run as a service.

5

If you installed logstash using a package manager like apt or yum , and you are running it as a service, you are not running it as the root user, but as the logstash user, unless you changed the service file

I have downloaded the logstash package and installed it as " rpm -ivh logstash"

Can you check which user is running logstash when you start it as a service?

I grep for logstash service, i think its running as logstash user

If you run it manually as the root user it will have permission to the folder, but it may not have permissions when you run as a service.

If i change the ownership to logstash user, will it work?

Thanks
Gautham

6

As I said, you need to check the logs, if logstash can't read the file in that directory, it will log.

You just need to make sure that the logstash user has permission to read files in the directory you want to use in the file input.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.