Logstash Tab / Space Delimiter

Elastic Stack Logstash
1

Hi All. I am trying to ingest some IIS logs into Elastic via logtstash. They are CSVs but the separator is NOT a ,. How can I account for a space or a tab as the separator.

I have multiple different kinds of files where the delimiter is a space or a tab and not a comma so I need to get this working. Any help would be appreciated.

I tried a few different formats:

  1. in between the " " in separator is an actual TAB. I tried the tabs also in the columns row as well. The documentation says to not use \t use the actual character. When I run it like this logstash starts up, but never actually ingests anything.

If I run this as a straight .log file and not a .csv file the data ingests, but I do not get the column names, it shows up as Column 1, column 2.. etc.

csv {
  source => "[message]"
  target => "[csv]"
  separator => "    "
  columns => [ "date","time","s-ip","cs-method","cs-uri-stem","cs-uri-query","s-port","cs-username","c-ip","cs(User-Agent)","cs(Referer)","sc-status","sc-substatus","sc-win32-status","time-taken" ]
  skip_empty_columns => true
  skip_empty_rows => true
}
2

The separator is the space " " not the tab for IIS logs.

W3C Extended format is a customizable ASCII format with a variety of different properties. You can log properties important to you, while limiting log size by omitting unwanted property fields. Properties are separated by spaces. Time is recorded as UTC.

3

Hi,

I had tried the space. Should it be 1 space or 2? I tried both single and double spaces.

4

Can you copy here a sample from the message as text, not a log line?

5

Sure. Here is with the tab:

csv {
source => "[message]"
target => "[csv]"
separator => " "
columns => [ "date","time","s-ip","cs-method","cs-uri-stem","cs-uri-query","s-port","cs-username","c-ip","cs(User-Agent)","cs(Referer)","sc-status","sc-substatus","sc-win32-status","time-taken" ]
skip_empty_columns => true
skip_empty_rows => true

and here with a space.

csv {
source => "[message]"
target => "[csv]"
separator => " "
columns => [ "date","time","s-ip","cs-method","cs-uri-stem","cs-uri-query","s-port","cs-username","c-ip","cs(User-Agent)","cs(Referer)","sc-status","sc-substatus","sc-win32-status","time-taken" ]
skip_empty_columns => true
skip_empty_rows => true
}

When I run logstash with this config I get no errors.. logstash starts up initially but never actually runs the ingest.

This message repeats in the console:

[2024-02-01T13:12:29,268][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Young Generation"}
[2024-02-01T13:12:29,268][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Old Generation"}
[2024-02-01T13:12:32,389][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2024-02-01T13:12:34,070][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu

6

I was able to get this working. There was a few changes I needed to make.

  1. Remove ( ) from the list of Columns in the Conf file

The only issue now is the files are getting deleted after processing.

I have this line in the conf file.
file_completed_action => "log"

any idea why logstash is ignoring it?

7

I mean the message sample, the message field not .conf. Anyway you solved. :upside_down_face:

When in read mode, what action should be carried out when a file is done with.
The default value for mode is tail.

Have you set the mode? Default is tail
And now copy your the input part.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.