# Implementing Update by Query in Logstash

**URL:** <https://discuss.elastic.co/t/implementing-update-by-query-in-logstash/251813>\
**Category:** Logstash\
**Created:** [October 12, 2020, 8:47pm UTC](https://discuss.elastic.co/t/implementing-update-by-query-in-logstash/251813 "2020-10-12T20:47:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![NomadicCodeGuy](https://avatars.discourse-cdn.com/v4/letter/n/7cd45c/32.png) [@NomadicCodeGuy](https://discuss.elastic.co/u/NomadicCodeGuy)\
**Post date:** [October 12, 2020, 8:47pm UTC](https://discuss.elastic.co/t/implementing-update-by-query-in-logstash/251813/1 "2020-10-12T20:47:32Z")

</div>

I am attempting to use Update by Query with logstash to copy a field [scenario] from entries that share the same field [file]. I am implementing the solution described by badger in this thread: [https://discuss.elastic.co/t/how-to-share-field-data-between-documents-of-the-same-file-path/250838](https://discuss.elastic.co/t/how-to-share-field-data-between-documents-of-the-same-file-path/250838)  
Badger's solution works like this:

1. Configure the index with a boolean field called something like scenarioAdded that [defaults](https://www.elastic.co/guide/en/elasticsearch/reference/current/null-value.html) to false.
2. Run logstash with an elasticsearch input that fetches all records that have a [scenario] field and [scenarioAdded] set to false
3. then feed those to an http output that makes an [update-by-query](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html) call to elasticsearch to add [scenario] and set scenarioAdded to true for all documents with the same [file]

I am using [this example](https://stackoverflow.com/questions/41976143/logstash-elasticsearch-update-denormalized-data/53406427#53406427) and [this example](https://stackoverflow.com/questions/53330232/does-logstash-support-elasticsearchs-update-by-query/53331640#53331640) as templates but have several questions about how my logstash conf file and how to copy data from files with the same "file" in the query itself. My current .conf file looks like this:

```auto
input {
   file 
   {
	path => "C:/TestInputFolders/*/reports/logs/*.log"
	start_position => "beginning"
	sincedb_path => "NUL"
    }
}
filter {
		grok 
		{
		match => 
			{
			"message" => ["%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}:%{SPACE}%{INT:threadNumber}%{SPACE}%{DATA:class}:%{INT:classLine}%{SPACE}-%{SPACE}%{DATA}scenario:%{SPACE}%{GREEDYDATA:scenario}",
			              "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}:%{INT:threadNumber}%{SPACE}%{GREEDYDATA:class}:%{INT:classLine}%{SPACE}-%{SPACE}%{GREEDYDATA:errorType}%{SPACE}%{GREEDYDATA:errorInfo}"]
			}						
		}
		grok
		{
		match =>
			{
			"path" => "C:/TestInputFolders/%{DATA:folder}/reports/logs/%{GREEDYDATA:file}"
			}
		}
if ![scenario]{
     mutate => {
        add_field => {"hasScenario" => "false" }
        add_field => {"scenario" => ""}
        }
}
output {
    http {
	hosts => ["http://localhost:9200/index/doc/_update_by_query"]
	http_method => "post"
	format => "json"
    }
}

```

and so far my query looks like this although I know it is currently incorrect(not sure how to get the [scenario] field from other entries with the same [file] field:

```auto
POST testLogs/_update_by_query
{
  "script": {
    "source": "ctx._source.scenario += params.scenario",
    "lang": "painless",
    "params": {
      "scenario": ""
    }
  },
  "query": {
    "term": {
      "hasScenario": "false"
    }
  }

```

There is a lot I don't understand about this method and would appreciate help implementing it. For one, where does I put my query? I see that there is a console to test it in Kibana but I imagine it must be saved somewhere. Second, how do I correctly point my inputs and outputs in the config so that I can output to http to do the update by query and then to my elasticsearch index once that is done? And third, how do I structure my query to copy the [scenario] field so that it is shared among all entries with the same [file] field? Any help is greatly appreciated!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 9, 2020, 8:47pm UTC](https://discuss.elastic.co/t/implementing-update-by-query-in-logstash/251813/2 "2020-11-09T20:47:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.