# How to use Logstash filter condition to copy docs **URL:** \ **Category:** Logstash\ **Created:** [February 25, 2016, 6:17am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter-condition-to-copy-docs/42670 "2016-02-25T06:17:34Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [February 25, 2016, 6:17am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter-condition-to-copy-docs/42670/1 "2016-02-25T06:17:34Z")
Hi All, We are trying to copy docs from one index to another index using logstash. Our index (es\_item) doc count is nearly 194,556,883 with primary size of 567.7GB, the docs which are indexed to the es\_item has different item types such as standard\_item & non\_standard\_item. 1. Our need is to copy only standard\_item docs to the new index (es\_item1). Is it possible to add ITEM\_TYPE in filter condition in order to copy only std\_items to the new index. 2. We have a field "ITEM\_TYPE": STANDARD\_ITEM in the es\_item index. Please let us know your suggestions. Below is the logstash configuration setup file, input { # We read from the "old" index elasticsearch { hosts =\> ["10.8.41.121:9200"] user =\> "esadmin" password =\> "XXXX" index =\> "es\_item" size =\> 500 scroll =\> "5m" docinfo =\> true } } filter { mutate { remove\_field =\> ["@timestamp", "@version"] } } output { # We write to the "new" index elasticsearch { host =\> "10.8.41.121:9200" protocol =\> "http" user =\> "esadmin" password =\> "XXXX" index =\> "es\_item1" index\_type =\> "%{[@metadata][\_type]}" document\_id =\> "%{[@metadata][\_id]}" } # We print dots to see it in action stdout { codec =\> "dots" } } Thanks, Ganeshbabu R ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [February 25, 2016, 7:06am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter-condition-to-copy-docs/42670/2 "2016-02-25T07:06:55Z")
Hi All, Please provide some feedback regarding matching item\_type condition in logstash filter by using grok.. filter { grok { patterns\_dir =\> "./patterns" match =\> { "ITEM\_TYPE" =\> "%{SGI}" } } } I tried by using grok inside the logstash conf filter to match ITEM\_TYPE = SGI but I am getting the following response while I start logstash run, [esadmin@tparhebfmi005 logstash-1.5.5]$ bin/logstash -f logstash.conf The error reported is: pattern %{SGI} not defined Please let us know any suggestions. Thanks, Ganeshbabu R ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 25, 2016, 1:28pm UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter-condition-to-copy-docs/42670/3 "2016-02-25T13:28:48Z")
> Our need is to copy only standard\_item docs to the new index (es\_item1). Is it possible to add ITEM\_TYPE in filter condition in order to copy only std\_items to the new index. To only send events whose `item_type` field matches "std\_item" to the elasticsearch output: ```auto output { if [item_type] == "std_item" { elasticsearch { .. } } } ``` ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [February 26, 2016, 6:41am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter-condition-to-copy-docs/42670/5 "2016-02-26T06:41:58Z")
Hi @magnusbaeck I tried by giving the condition in elasticsearch output as like below, output { if [ITEM\_TYPE] == "SGI" { elasticsearch { host =\> "10.8.44.124:9201" protocol =\> "http" user =\> "esadmin" password =\> "XXXX" index =\> "es\_item2" index\_type =\> "%{[@metadata][\_type]}" document\_id =\> "%{[ITEM\_ID]}" } # We print dots to see it in action stdout { codec =\> "dots" } }} I can able to see logstash were running but copying docs to the new index were not happening. Please let us know your suggestions Thanks, Ganeshbabu R ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 26, 2016, 7:10am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter-condition-to-copy-docs/42670/6 "2016-02-26T07:10:17Z")
First, make sure that Logstash has events to process. Disable the ES outputs and verify that events flow nicely. If they do, look in the logs of both Logstash and ES. ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [February 26, 2016, 7:26am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter-condition-to-copy-docs/42670/7 "2016-02-26T07:26:50Z")
Hi @magnusbaeck I am very new to the logstash can you tell us how to find the event process in logstash? Incase If I disable ES output in logstash conf file how & where the copy docs will be indexed. Please let us know your suggestions. Thanks, Ganeshbabu R ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 26, 2016, 7:47am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter-condition-to-copy-docs/42670/8 "2016-02-26T07:47:54Z")
If you don't see any events in ES this can basically be caused by three things: - Logstash isn't processing any events. - Logstash isn't able to send the documents. - You're looking in the wrong place. For example, the events could have a `@timestamp` value that doesn't match your expectations so the time filter in Kibana is wrong. Before we jump to conclusions we need to understand what the problem is. We do that by isolating things. So, disable the elasticsearch output so that the stdout output is the only output and run Logstash. Are you getting dots printed to stdout? ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [February 26, 2016, 8:08am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter-condition-to-copy-docs/42670/9 "2016-02-26T08:08:42Z")
Thanks for your response @magnusbaeck As you said disable elasticsearch output so that the stdout is the only output in logstash conf file, Below is the response of logstash run after disabling elasticsearch output input { # We read from the "old" index elasticsearch { hosts =\> ["10.8.44.124:9201"] user =\> "esadmin" password =\> "dev01" index =\> "es\_item" size =\> 500 scroll =\> "5m" docinfo =\> true } } filter { } output { # We print dots to see it in action stdout { codec =\> "dots" } } Yes, I am getting dots printed in stdout & below is the output [esadmin@tparhebfmi003 logstash-1.5.5]$ bin/logstash -f logstash.conf Logstash startup completed ................................................................................................................................................................................................................................................................................................................................................. Regards, Ganeshbabu R ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 26, 2016, 8:33am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter-condition-to-copy-docs/42670/10 "2016-02-26T08:33:48Z")
Okay, so that seems to work then. So what's in the Logstash and ES logs after you reenable the ES output? ---
**Author:** ![r.ganeshbabu](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@r.ganeshbabu](https://discuss.elastic.co/u/r.ganeshbabu)\ **Post date:** [February 26, 2016, 9:08am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter-condition-to-copy-docs/42670/11 "2016-02-26T09:08:33Z")
After I reenable the ES output in logstash conf file input { # We read from the "old" index elasticsearch { hosts =\> ["10.8.44.122:9200"] user =\> "esadmin" password =\> "dev01" index =\> "es\_item2" size =\> 500 scroll =\> "5m" docinfo =\> true } } filter { } output { if [ITEM\_TYPE] == "SGI" { # We write to the "new" index elasticsearch { host =\> "10.8.44.122:9200" protocol =\> "http" user =\> "esadmin" password =\> "dev01" index =\> "es\_item1" document\_type =\> "%{[@metadata][\_type]}" document\_id =\> "%{[@metadata][\_id]}" } # We print dots to see it in action stdout { codec =\> "dots" } } } below is the response of logstash run and it stucked in somewhere.. [esadmin@tparhebfmi003 logstash-1.5.5]$ bin/logstash -f logstash.conf Logstash startup completed ............................. Below is the ES client logs, [2016-02-26 03:59:18,631][DEBUG][action.search.type] [tparhebfmi003\_PERF\_CLIENT] All shards failed for phase: [query] org.elasticsearch.transport.RemoteTransportException: [tparhebfmi005\_PERF\_DATA][inet[/10.8.44.124:9260]][indices:data/read/search[phase/query]] Caused by: org.elasticsearch.search.SearchParseException: [ogrds\_item2][11]: from[-1],size[-1],sort[,\!]: Parse Failure [Failed to parse source [{"sort":[{"\_score":{"order":"desc"}},{"CRT\_DTTM":{"order":"desc"}}],"query":{"filtered":{"query":{"bool":{"should":[]}},"filter":{"bool":{"must":[{"bool":{"should":[{"nested":{"query":{"match":{"EXTRN\_CODE\_CONTAINS":"4023900543008"}},"path":"XCD"}}]}},{"bool":{"should":[{"nested":{"query":{"filtered":{"filter":{"terms":{"XCD.PROC\_GRP\_ID":["REWE - PETZ REPEZ (DE)"]}}}},"path":"XCD"}}]}},{"bool":{"should":[{"query":{"filtered":{"filter":{"terms":{"ITEM\_SPECIFICITY\_REF\_ID":[184]}}}}}]}}]}}}},"from":0,"fields":["MOD\_DSCR","ITEM\_CODE","ITEM\_DSCR","IS\_SHARED\_IND","HAS\_IMAGE\_IND","DIST.RGN\_NM","HAS\_HIST\_IND","ITEM\_TYPE","CRT\_DTTM","ITEM\_MISUSED\_GTIN\_FLG"],"size":2000}]] at org.elasticsearch.search.SearchService.parseSource(SearchService.java:747) at org.elasticsearch.search.SearchService.createContext(SearchService.java:572) at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544) at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:306) at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776) at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767) at org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:176) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NumberFormatException: For input string: "REWE - PETZ REPEZ (DE)" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Long.parseLong(Long.java:589) at java.lang.Long.parseLong(Long.java:631) at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseLongValue(NumberFieldMapper.java:354) at org.elasticsearch.index.mapper.core.LongFieldMapper.indexedValueForSearch(LongFieldMapper.java:171) at org.elasticsearch.index.mapper.core.AbstractFieldMapper.termFilter(AbstractFieldMapper.java:469) Regards, Ganeshbabu R ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:09am UTC](https://discuss.elastic.co/t/how-to-use-logstash-filter-condition-to-copy-docs/42670/12 "2017-07-06T05:09:38Z")