Usage of Logstash - Nullifying the logs when not required to be published

[pavan_tiriveedhi]

Hi, We are using Logstash to push logs from our kubernetes environments to Azure for storing and computing, we have installed the services via help chart - helm-charts/logstash at main · elastic/helm-charts · GitHub.

Most of the times we are ending in pushing logs to azure even not required, since our stack comprises large number of microservices its hard for us to stop/resume sending logs into logstash. so we would like to know if there is any possibility to enable/ disable output fields in a way it would send logs only when enabled, else logs get nullified inside pipeline/ storage itself. ?

Can explain / share some more detail information if required.

Thanks

[Priscilla_Parodi]

Hello @pavan_tiriveedhi, welcome to the community!

To achieve your goal, you can use conditionals in the Logstash pipeline to selectively include or exclude certain logs based on specific conditions.

if EXPRESSION {
  ...
} else if EXPRESSION {
  ...
} else {
  ...
}

Hope it helps!

[Badger]

You can use a drop {} filter to discard events.

[pavan_tiriveedhi]

Hi @Badger Thanks for responding back, while using the drop {} am getting below error. IT states the drop plugin is not recongized by logstash helm chart i am using.

ERROR logstash.plugins.registry - Unable to load plugin. {:type=>"output", :name=>"drop"}

ERROR logstash.agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main2, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'drop'. Are you sure this is correct? Trying to load the drop output plugin resulted in this error: Unable to load the requested plugin named drop of type output. The plugin is not installed."

I am using the logstash helm chart - 7.17.3

[pavan_tiriveedhi]

Yes tried using the expressions but am not sure where to point inside the expression to drop/ delete / eliminate the logs. can you please advise if there is a plugin/ command/ config can be used for discarding the logs

[leandrojmp (Leandro Pereira)]

@pavan_tiriveedhi

Thanks for responding back, while using the drop {} am getting below error. IT states the drop plugin is not recongized by logstash helm chart i am using.

You need to drop the document inside the filter block, not in the output block.

So you need to define the conditionals to drop a document inside the filter block.