Need title/name of file for logstash ingestion

Elastic Stack Logstash
1

Hello,
I am trying to get part of the file names for ingestion for each record, however, I am ingesting multiple files using "cat /filespath/ | /logstashpath/ -f /logstashconfig/" command. The reason i am running it like this is so that the logstash config file only runs "once" on each of the files in the filespath. I am not trying to have it run continuously without end. Doing it this way, however, does not give the path and filename for which to use the grok option and store part of the file name. How can I go about this?

2

When run this way, the Logstash process is receiving a single continuous stream on stdin, which is the result of the cat /filespath/ command.

You may be able to use the File Input Plugin's mode => read (which reads each file to end) and sincedb_path => "/dev/null" (which prevents it from recording that it has already read the files), since it records a [@metadata][path] to each generated event. If you would like to specify the path at the command line still, you may be able to do so with an environment variable:

input {
  file {
    mode => read
    sincedb_path => "/dev/null"
    path => ["${SOURCE_FILE_GLOB}"]
    # ...
  }
}

usage:

SOURCE_FILE_GLOB="/filespath/*.log" logstash -f pipeline.conf
3

Where is the pipeline.conf located or is that a specific file I was using?

4

I assume this is regarding the pipeline.yml. Is there any sort of configuration that must be set up here?

5

I'm sorry, I was using Logstash's -f to specify a path to a pipeline configuration file, as you had used it to point to a directory of files (see: https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html#command-line-flags)

6

Ah understood. I figured that was the only that it could be used, however, would that mean that the file would call itself. since the variable holds it like this.

And then this is to be used with in the input of the config file?

7

Ah. In bash and other POSIX-compliant shells, you can set a variable in-line (which avoids persisting it to other commands).

It's roughly the same as:

export SOURCE_FILE_GLOB="/filespath/*.log"
logstash -f pipeline.conf
unset SOURCE_FILE_GLOB
8

Ok understood. I've tried to do this whole setup, however, it doesn't stop running. The config file that is. It just keeps going and doesn't even in read mode.

9

Is there anyway to have the logstash ingest the files once and then stop? Service is not being run the logstash and config file are being run manually.

10

Not with a file input. The stdin input works like that, and a few others can be made to do so, but not a file input.

11

Using the stdin for multiple files while keeping the path field?

I found that when i try to give it multiple files the only way i know of is to do

  • cat /directory of files | /logstash-execution -f /logstash-config file

However, this method doesnt keep add the path field since everything is being ingested as one full stream of data.

12

I cannot think of a way to get what you want.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.