Stopping logstash

Elastic Stack Logstash
1

Hello, i'm currently working on logstash and i have a question. To launch my logtash script, i use this command:

sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

When I run it in my terminal, to stop logstash I have to press ctrl +C, which is quite annoying because I have to cronize my script so that it runs every day at a given time.

How can I do this?

2

What is your input?

The easiest way is to run Logstash as a service, so it will always be running or you can start and stop it using systemd.

3

How can I do it?

4

When I run this command :

sudo systemctl status logstash

I have this output :

image
image1270×145 6.83 KB

So it seems that logstash is already running as a service.

But when data are ingest in my repository logstash doesn't send them automatically into elasticsearch

5

You have installed LS from the DEB or RPM installation which by default creates the service.

  1. Show us test.conf
  2. Add log.level: debug in logstash.yml
  3. Show us the log: logstash-plain.log
6

You need to share your test.conf file to show what logstash is doing.

7

Here it is

input {
  file {
    path => "/testELK/*.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  csv {
    separator => ","
    skip_header => "true"
    skip_empty_rows => true
    columns => [
      "Plugin_ID",
      "CVE",
      "CVSS_v2.0_Base_Score",
      "Risk",
      "Host",
      "Protocol",
      "Port",
      "Name",
      "CVSS_v3.0_Base_Score",
      "CVSS_v2.0_Temporal_Score",
      "CVSS_v3.0_Temporal_Score",
      "Risk_Factor",
      "Metasploit"
    ]
  }
}

output {
  elasticsearch {
    hosts =>  "*******:9200"
    index => "index"
    template_name => "template"
  }

  stdout {}
}
8

Does the logstash user have permissions to read the files on this path?

When Logstash runs as a service it runs as the logstash user.

9

I have one more question:

Every day I receive csvs in a directory. logstash ingests them into elasticsearch. How do I tell it that after integrating a csv into elasticsearch, it moves to another directory or simply that it doesn't ingest the same file that it already ingested the day before?

10

yes because I applied chmod 777 to my file

11

Logstash cannot move files, only delete, but it needs to use the file input with the mode set as read.

Per default it will use the mode as tail, which will constantly look for changes in the files, if the files in the directory are not being constantly updated, then you could change the mode to read and configure logstash to delete the files.

Check the documentiaton for more information about it.

This is done by the sincedb_path configuration, since you set it to /dev/null you are telling logstash to reprocess everything. If you want it to not process files that it already read you need to point the sincedb_path to a custom file or just remove this setting and logstash will create a sincedb file per default.

Yeah, but this doesn't matter if the logstash user cannot access the path, the logstash user needs to have permissions on the path as well, does it?

12

So, if I understand your point of view correctly, I should remove this parameter. : sincedb_path => "/dev/null"

13

sincedb_path => "/dev/null" is used for in-memory tracking of file processing.

As Leandro said, you can remove/comment than will LS used default settings and keep processed records in a file or set by your own path/file.

In short, if you want to keep the file tracking, just do not use null, remove the line.

15

Hello, this was alread answered in this topic, please check previous answers.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.