# Capture Log for Logstash For every successfull pipeline execution

**URL:** <https://discuss.elastic.co/t/capture-log-for-logstash-for-every-successfull-pipeline-execution/326298>\
**Category:** Logstash\
**Created:** [February 23, 2023, 10:27am UTC](https://discuss.elastic.co/t/capture-log-for-logstash-for-every-successfull-pipeline-execution/326298 "2023-02-23T10:27:29Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rakhshunda\_Noorein\_J](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rakhshunda_noorein_j/32/99407_2.png) [@Rakhshunda\_Noorein\_J](https://discuss.elastic.co/u/Rakhshunda_Noorein_J)\
**Post date:** [February 23, 2023, 10:27am UTC](https://discuss.elastic.co/t/capture-log-for-logstash-for-every-successfull-pipeline-execution/326298/1 "2023-02-23T10:27:29Z")

</div>

Hello,

I want to capture log for logstash sucessfully fetching the api data from http\_poller plugin and entering it to my elasticDB.

My configuration is:

```auto
input {
    http_poller {
	id => "test-plugin"
    urls => {
	test_api => {
        method => "POST"
        url => "api url"
		headers => {
                    "Content-Type" => "application/json"
                }
      }
	}	
	request_timeout => 120
	schedule => {cron => "8 5 * * * UTC"}
	codec => "json"
	tags => ["test-api"]
  }
}
filter { 
	mutate {
		remove_field => ["[message]" ]
		remove_field => ["[@version]" ]
		remove_field => ["[event]" ]
	}
}
output {
	if "test-api" in [tags]
	{
		elasticsearch {
			id => "test-output"
			hosts => ["host1"]
			user => "user"
			password => "password"
			index => "my-index"
			document_id => "%{ID}"
			doc_as_upsert => true
			action => "update"
		 }
	}
}

```

After every successfull run, a log will be inserted in csv file...Is there a way to do that....

---

<div class="post-metadata">

**Author:** ![eMitch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emitch/32/93607_2.png) [@eMitch](https://discuss.elastic.co/u/eMitch)\
**Post date:** [February 23, 2023, 7:02pm UTC](https://discuss.elastic.co/t/capture-log-for-logstash-for-every-successfull-pipeline-execution/326298/2 "2023-02-23T19:02:55Z")

</div>

Hi @Rakhshunda_Noorein_J

You should be able to specify a second output and use the [CSV output plugin](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-csv.html).

---

<div class="post-metadata">

**Author:** ![Rakhshunda\_Noorein\_J](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rakhshunda_noorein_j/32/99407_2.png) [@Rakhshunda\_Noorein\_J](https://discuss.elastic.co/u/Rakhshunda_Noorein_J)\
**Post date:** [February 27, 2023, 10:26am UTC](https://discuss.elastic.co/t/capture-log-for-logstash-for-every-successfull-pipeline-execution/326298/3 "2023-02-27T10:26:16Z")

</div>

It is logging all the event in the inputs..

I want to log the events only all the events from my input is complete.

---

<div class="post-metadata">

**Author:** ![eMitch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emitch/32/93607_2.png) [@eMitch](https://discuss.elastic.co/u/eMitch)\
**Post date:** [February 27, 2023, 12:15pm UTC](https://discuss.elastic.co/t/capture-log-for-logstash-for-every-successfull-pipeline-execution/326298/4 "2023-02-27T12:15:39Z")

</div>

You can [read the response code](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html#plugins-inputs-http_poller-ecs_metadata) from the message of the http\_poller and then only allow codes = 200 to hit the CSV output.

Something like:

```auto
output {
	if "test-api" in [tags]
	{
		elasticsearch {
			id => "test-output"
			hosts => ["host1"]
			user => "user"
			password => "password"
			index => "my-index"
			document_id => "%{ID}"
			doc_as_upsert => true
			action => "update"
		 }
	}
    if [@metadata][code] == 200
	{
		csv {
			path => /mypath/output.csv
            fields => ["field1", "[nested][field]"]
            ...
		 }
	}
}

```

Keep in mind, Logstash will send both messages simultaneously to Elasticsearch and CSV outputs if the conditions are met properly.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [February 27, 2023, 1:04pm UTC](https://discuss.elastic.co/t/capture-log-for-logstash-for-every-successfull-pipeline-execution/326298/5 "2023-02-27T13:04:13Z")

</div>

> [@Rakhshunda\_Noorein\_J](#):
>
> It is logging all the event in the inputs..
> 
> I want to log the events only all the events from my input is complete.

You need to provide more context about what is being logged and what you want to log.

Your input is a `http_poller` input filter, if the return of this filter has multiple lines, every single line will be logged.

---

<div class="post-metadata">

**Author:** ![Rakhshunda\_Noorein\_J](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rakhshunda_noorein_j/32/99407_2.png) [@Rakhshunda\_Noorein\_J](https://discuss.elastic.co/u/Rakhshunda_Noorein_J)\
**Post date:** [February 28, 2023, 7:35am UTC](https://discuss.elastic.co/t/capture-log-for-logstash-for-every-successfull-pipeline-execution/326298/6 "2023-02-28T07:35:21Z")

</div>

> [@leandrojmp](#):
>
> put filter, if the return of this filter has multiple lines, every single line will be logged.

yes... But my requirement is for all the event of input plugins successfully ran, then insert an event to the csv file.

ex. soppose I am calling a web api from http\_poller input plugin and sending the input to my elasticsearch index. When this is successfully done then insert only 1 line in csv with date, tag and message for logging pupose so that I can know when the input ran.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [February 28, 2023, 11:23am UTC](https://discuss.elastic.co/t/capture-log-for-logstash-for-every-successfull-pipeline-execution/326298/7 "2023-02-28T11:23:00Z")

</div>

Unless you have some message that identify that the request is finished, Logstash cannot do that.

If you have some message where you can apply a conditional filter, you would be able to only write something on a csv after this message.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 28, 2023, 11:23am UTC](https://discuss.elastic.co/t/capture-log-for-logstash-for-every-successfull-pipeline-execution/326298/8 "2023-03-28T11:23:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.