What Would be the filetype for our custom patterns in ./pattern directory

Elastic Stack Logstash
1

Hi guys,

i read document of logstash where document said you can define your custom pattern in ./pattern directory. but when i download latest logstash then

  1. There is not any ./pattern directory available in logstash installation directory ?
  2. There is no info regarding what would be the filetype for my custom grok patterns ?
  3. where i can define critical errors log list to compare logs inside incoming logs ?

I require these things to made my configuration generics for another user to understand.

2

There is not any ./pattern directory available in logstash installation directory ?

No, you have to create it yourself. /etc/logstash/patterns would be a good choice.

There is no info regarding what would be the filetype for my custom grok patterns ?

The Custom Patterns section of the grok filter documentation should explain this. If not please be more specific about what is unclear.

where i can define critical errors log list to compare logs inside incoming logs ?

Not sure exactly what you're asking, but it sounds like something you could use custom grok patterns for.

3

Hi @magnusbaeck

No, you have to create it yourself. /etc/logstash/patterns would be a good choice.

what if i installed logstash using tar file not rpm or deb file and created ./pattern directory in extracted logstash directory then how will you set the path for ./pattern directory for logstash configurations ?

The Custom Patterns section of the grok filter documentation should explain this. If not please be more specific about what is unclear.

Suppose, I have logstash configuration file with "logstash.yml" where we define configuration regarding logstash using ".yml" extension similarly, what would be the extension for custom grok pattern files ?

Not sure exactly what you're asking, but it sounds like something you could use custom grok patterns for.

yeah, you're right but here scenario is different.

Suppose, We have our own list of critical error code with error description in a csv format. but which log files we are parsing it does'nt have any error code. so in that scenario i had to match it's error description to my csv file and assign that error to a specific error code how can we do that ?

Is there any option to load custom csv or json file to load statically and used it for comparison of event logs statement ?

4

what if i installed logstash using tar file not rpm or deb file and created ./pattern directory in extracted logstash directory then how will you set the path for ./pattern directory for logstash configurations ?

I'm not sure I understand the question. Put your pattern files in any directory you like and tell Logstash to pick up patterns from there.

what would be the extension for custom grok pattern files ?

Logstash doesn't care about the file extension. The format of the file is described in the grok filter documentation.

We have our own list of critical error code with error description in a csv format. but which log files we are parsing it does'nt have any error code. so in that scenario i had to match it's error description to my csv file and assign that error to a specific error code how can we do that ?

It sounds like the translate filter would be useful to you.

5

Hi @magnusbaeck

I have the problem to use translate plugin for logstash as you recommended. when i run my configuration it's throwing an error where it showing this error message.

17:48:15.793 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Couldn't find any filter plugin named 'translate'. Are you sure this is correct? Trying to load the translate filter plugin resulted in this error: Problems loading the requested plugin named translate of type filter. Error: NameError NameError"}

can you help me on same.
and if this is external plugin then

how can i install this in logstash manually ?
because i don't have internet connection on remote machine.

It seems like translate plugin is not available in logstash default installation

6

The plugin's documentation explains how to install it. Plugin installation on machines without Internet connectivity is explained at https://www.elastic.co/guide/en/logstash/current/offline-plugins.html.

7

Hi @magnusbaeck,

but when i run this command

bin\logstash-plugin install logstash-filter-translate

it is not installing. it is throw validation error.

Validating logstash-filter-translate
Unable to download data from https://rubygems.org - Received fatal alert: access_denied (https://api.rubygems.org/latest_specs.4.8.gz)
ERROR: Installation aborted, verification failed for logstash-filter-translate
8

But that's not the command the documentation in the second link tells you to run.

9

Hi @magnusbaeck

Actually I got answer.

This is community developed plugins so i had to download this plugin executable rubygems file from community in my local system.
Now, I will zip this file using any compression tool. and then i will run above command using for this path location like this.

bin\logstash-plugin install file:///C:/ES-1/logstash-5.4.0/logstash-filter-translate-3.0.1.zip

above was validation error because logstash-plugin batch script expecting a .zip or .tar format for there executable rubygems file. so you must have to compressed this file.

and bingo! your plugin installed successfully. :slight_smile:

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.