# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=678 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 679 --- ## [Append tag in Elasticsearch using Logstash](https://discuss.elastic.co/t/append-tag-in-elasticsearch-using-logstash/150161)
**Author:** [@georgeRD](https://discuss.elastic.co/u/georgeRD)\ **Replies:** 6\ **Last updated:** [September 28, 2018, 2:31pm UTC](https://discuss.elastic.co/t/append-tag-in-elasticsearch-using-logstash/150161 "2018-09-28T14:31:03Z")
Hi all, Quick question about tagging with Logstash. I am using the following filter: filter { mutate { add\_tag =\> \[ "test" \] } } It appears that the add\_tag replaces any existing in tags field. Is there any way to… --- ## [Parsing Log4j2 logs from filebeat based on defined Pattern Layout](https://discuss.elastic.co/t/parsing-log4j2-logs-from-filebeat-based-on-defined-pattern-layout/149941)
**Author:** [@sbienert](https://discuss.elastic.co/u/sbienert)\ **Replies:** 7\ **Last updated:** [September 28, 2018, 1:33pm UTC](https://discuss.elastic.co/t/parsing-log4j2-logs-from-filebeat-based-on-defined-pattern-layout/149941 "2018-09-28T13:33:57Z")
Hello, I want to parse my logs from Filebeat into Elasticsearch as if there would be an Elasticsearch appender in Log4j2.xml meaning correct fields with their values and not one field with message where every field valu… --- ## [How do I grok this Cisco log?](https://discuss.elastic.co/t/how-do-i-grok-this-cisco-log/149049)
**Author:** [@ramindia](https://discuss.elastic.co/u/ramindia)\ **Replies:** 8\ **Last updated:** [September 28, 2018, 1:09pm UTC](https://discuss.elastic.co/t/how-do-i-grok-this-cisco-log/149049 "2018-09-28T13:09:44Z")
Hi Thank you for the earlier help, iam progressing in good way. Iam going here is more granular logging and get more information for statistic purpose, at the same time i would like retain some message as it is for use… --- ## [XMl webservice using http\_poller](https://discuss.elastic.co/t/xml-webservice-using-http-poller/150343)
**Author:** [@Android\_Games](https://discuss.elastic.co/u/Android_Games)\ **Replies:** 0\ **Last updated:** [September 28, 2018, 1:04pm UTC](https://discuss.elastic.co/t/xml-webservice-using-http-poller/150343 "2018-09-28T13:04:53Z")
I'm using http\_poller to get data from Webservice. I'm not sure if its able to connect the webservice Config file input { http\_poller { urls =\> { myurl =\> "https://abc.com/SWS/incidents.asmx/GetSev" } co… --- ## [Deprecated config setting "document\_type" set in elasticsearch](https://discuss.elastic.co/t/deprecated-config-setting-document-type-set-in-elasticsearch/150216)
**Author:** [@Erik\_Gast1](https://discuss.elastic.co/u/Erik_Gast1)\ **Replies:** 2\ **Last updated:** [September 28, 2018, 11:47am UTC](https://discuss.elastic.co/t/deprecated-config-setting-document-type-set-in-elasticsearch/150216 "2018-09-28T11:47:44Z")
I am getting this entry in the log when starting Logstash. \[logstash.outputs.elasticsearch\] You are using a deprecated config setting "document\_type" set in elasticsearch. Deprecated settings will continue to work, but … --- ## [Logstash Snort Log Parse Error](https://discuss.elastic.co/t/logstash-snort-log-parse-error/149309)
**Author:** [@kolten](https://discuss.elastic.co/u/kolten)\ **Replies:** 2\ **Last updated:** [September 28, 2018, 11:15am UTC](https://discuss.elastic.co/t/logstash-snort-log-parse-error/149309 "2018-09-28T11:15:43Z")
hi, i want to parse my snort log file with grok but pattern couldnt parse it. Here is my log file example; { "timestamp" : "09/20-07:12:23.851184", "msg" : "Nmap XMAS Tree Scan", "priority" : 0, "proto" : "TCP", "src\_… --- ## [XML Webservice-\>http\_poller-\>elastic](https://discuss.elastic.co/t/xml-webservice-http-poller-elastic/150196)
**Author:** [@Android\_Games](https://discuss.elastic.co/u/Android_Games)\ **Replies:** 2\ **Last updated:** [September 28, 2018, 10:54am UTC](https://discuss.elastic.co/t/xml-webservice-http-poller-elastic/150196 "2018-09-28T10:54:45Z")
Hello All, I'm trying to pull data from xml webservice and load into elastic to show in kibana. Using Client Certificate to authenticate webservice . My conf file looks as below . input { http\_poller { urls =\> {… --- ## [Not getting logs from filebeat to logstash](https://discuss.elastic.co/t/not-getting-logs-from-filebeat-to-logstash/150318)
**Author:** [@narasimha](https://discuss.elastic.co/u/narasimha)\ **Replies:** 0\ **Last updated:** [September 28, 2018, 10:12am UTC](https://discuss.elastic.co/t/not-getting-logs-from-filebeat-to-logstash/150318 "2018-09-28T10:12:38Z")
I am new to ELK stack, I am trying to get the logs from filebeat to logstash and display on stand output(terminal). Filebeat configuration: filebeat.inputs: type: log Change to true to enable this input configurati… --- ## [Data import to Elastic search](https://discuss.elastic.co/t/data-import-to-elastic-search/150298)
**Author:** [@MaralErdene\_Tumursuh](https://discuss.elastic.co/u/MaralErdene_Tumursuh)\ **Replies:** 2\ **Last updated:** [September 28, 2018, 9:27am UTC](https://discuss.elastic.co/t/data-import-to-elastic-search/150298 "2018-09-28T09:27:58Z")
Hello Guys, I have to import Oracle table to Elastic search using Logstash. So mission is my query has string value that separated comma / like a '\[1,2,3\]'/. How insert this string to field with array type of elastic… --- ## [Flushing of pipelines](https://discuss.elastic.co/t/flushing-of-pipelines/150311)
**Author:** [@Carlos\_Magalhaes](https://discuss.elastic.co/u/Carlos_Magalhaes)\ **Replies:** 0\ **Last updated:** [September 28, 2018, 8:57am UTC](https://discuss.elastic.co/t/flushing-of-pipelines/150311 "2018-09-28T08:57:17Z")
Hi there Would really appreciate the help, trying to understand what "pushing flush into pipeline" means. Running some sflow with elastiflow on a now newly installed OS and ELK stack these inputs into logstash in the … --- ## [Winlogbeat and filebeat config can't wokr together](https://discuss.elastic.co/t/winlogbeat-and-filebeat-config-cant-wokr-together/150144)
**Author:** [@kursad](https://discuss.elastic.co/u/kursad)\ **Replies:** 8\ **Last updated:** [September 28, 2018, 8:43am UTC](https://discuss.elastic.co/t/winlogbeat-and-filebeat-config-cant-wokr-together/150144 "2018-09-28T08:43:34Z")
Dear Friends, I already send my application and IIS logs to elasticsearch without any problem. Currently, i'm trying to send windows eventviwer logs to elasticsearch. But interestingly when I start to logstash, logs can… --- ## [Plugin Mayhem](https://discuss.elastic.co/t/plugin-mayhem/150075)
**Author:** [@Carlos\_Magalhaes](https://discuss.elastic.co/u/Carlos_Magalhaes)\ **Replies:** 1\ **Last updated:** [September 28, 2018, 7:52am UTC](https://discuss.elastic.co/t/plugin-mayhem/150075 "2018-09-28T07:52:02Z")
Hi all When I install logstash-codec-sflow with bin/logstash-plugin install logstash-codec-sflow I get: Validating Installing Install Successful I then do a bin/logstash-plugin list --installed and all the plugins … --- ## [50 pipelines and two instances](https://discuss.elastic.co/t/50-pipelines-and-two-instances/149072)
**Author:** [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Replies:** 3\ **Last updated:** [September 28, 2018, 6:11am UTC](https://discuss.elastic.co/t/50-pipelines-and-two-instances/149072 "2018-09-28T06:11:06Z")
Hello. I have two nodes on which the Logstash and Elastic runs. At the moment I am using only one node in production for Logstash, 50 pipelines, processing approximately 600,000 records per 1h (running scheduler with h… --- ## [LOCATION as template with an array of two values](https://discuss.elastic.co/t/location-as-template-with-an-array-of-two-values/149806)
**Author:** [@Olha1](https://discuss.elastic.co/u/Olha1)\ **Replies:** 1\ **Last updated:** [September 28, 2018, 6:04am UTC](https://discuss.elastic.co/t/location-as-template-with-an-array-of-two-values/149806 "2018-09-28T06:04:42Z")
Hi! I have config file with this in filter: mutate { add\_field =\> { "\[location\]" =\> "%{latitude},%{longitude}" } remove\_field =\> \["latitude", "longitude"\] } But, in 5 version in logstash, location was array, and I… --- ## [Kafka Output Plugin -\> Issues with Bootstrap\_Server](https://discuss.elastic.co/t/kafka-output-plugin-issues-with-bootstrap-server/149929)
**Author:** [@dyongbcst](https://discuss.elastic.co/u/dyongbcst)\ **Replies:** 1\ **Last updated:** [September 28, 2018, 6:01am UTC](https://discuss.elastic.co/t/kafka-output-plugin-issues-with-bootstrap-server/149929 "2018-09-28T06:01:20Z")
Hi, I am trying to use the logstash-output-kafka ////My codes input { stdin {} } output { kafka { topic\_id =\> "mytopic" bootstrap\_server =\> ":9092" } } However, when executing this configuration, I am getting … --- ## [Throttle filter seems not be working if I chose json as codec for Kafka input plugin](https://discuss.elastic.co/t/throttle-filter-seems-not-be-working-if-i-chose-json-as-codec-for-kafka-input-plugin/150286)
**Author:** [@weizhu](https://discuss.elastic.co/u/weizhu)\ **Replies:** 0\ **Last updated:** [September 28, 2018, 4:30am UTC](https://discuss.elastic.co/t/throttle-filter-seems-not-be-working-if-i-chose-json-as-codec-for-kafka-input-plugin/150286 "2018-09-28T04:30:21Z")
I tried throttle plugin with Kafka input plugin. If the codec is Json, the throttle doesn't take affect. If I choose plain, the throttle works as expected. Is this a known issue? logstash 6.4.1 logstash-input-kafka (8… --- ## [S3 plugin relative path error](https://discuss.elastic.co/t/s3-plugin-relative-path-error/149586)
**Author:** [@Harshet\_Jain](https://discuss.elastic.co/u/Harshet_Jain)\ **Replies:** 2\ **Last updated:** [September 27, 2018, 11:13pm UTC](https://discuss.elastic.co/t/s3-plugin-relative-path-error/149586 "2018-09-27T23:13:50Z")
I keep getting an error that relative path has been specified Any ideas? input { s3 { id =\> 'main' access\_key\_id =\> '' secret\_access\_key =\> '/O' bucket =\> 'sales-data' region =\> 'us-east-1' prefix =\> '/' typ… --- ## [Flushing into pipeline and nothing else happens](https://discuss.elastic.co/t/flushing-into-pipeline-and-nothing-else-happens/149619)
**Author:** [@Harshet\_Jain](https://discuss.elastic.co/u/Harshet_Jain)\ **Replies:** 2\ **Last updated:** [September 27, 2018, 11:13pm UTC](https://discuss.elastic.co/t/flushing-into-pipeline-and-nothing-else-happens/149619 "2018-09-27T23:13:37Z")
I have seen many other people who have posted similar issues but none of them have posted how they resolved it. Some even say that they waited and it started working. Here is my conf file input { file { path =\> … --- ## [Clearpass logs \[SOLVED\]](https://discuss.elastic.co/t/clearpass-logs-solved/149906)
**Author:** [@Dr\_Natas](https://discuss.elastic.co/u/Dr_Natas)\ **Replies:** 6\ **Last updated:** [September 27, 2018, 10:39pm UTC](https://discuss.elastic.co/t/clearpass-logs-solved/149906 "2018-09-27T22:39:10Z")
This is my first grok filter, if any one has any input I'm more than welcome to receive any. I'm able to parse a good portion of Aruba Clearpass logs but this field I'm unable split it by columns e.g. I would like to … --- ## [Performances ruby/redis vs translate filter](https://discuss.elastic.co/t/performances-ruby-redis-vs-translate-filter/150094)
**Author:** [@drosys](https://discuss.elastic.co/u/drosys)\ **Replies:** 2\ **Last updated:** [September 27, 2018, 10:37pm UTC](https://discuss.elastic.co/t/performances-ruby-redis-vs-translate-filter/150094 "2018-09-27T22:37:28Z")
Dear All, I have a use case where i need to be able to compare high volume of data (fields like ip addr) to large IP databases. I have think about the approach of using ruby filter with redis socket connection (with in… --- ## [Logstash elasticsearch input snapshot](https://discuss.elastic.co/t/logstash-elasticsearch-input-snapshot/150238)
**Author:** [@frankfoti](https://discuss.elastic.co/u/frankfoti)\ **Replies:** 4\ **Last updated:** [September 27, 2018, 7:09pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-snapshot/150238 "2018-09-27T19:09:17Z")
We are trying to input elasticsearch snapshots to logstash to support a dashboard on kibana to display elasticsearch cluster snapshots. Below is not working. No error just no index gets created. The curl command does … --- ## [How to escape SSL certificate verification in Elastic search input for logstash](https://discuss.elastic.co/t/how-to-escape-ssl-certificate-verification-in-elastic-search-input-for-logstash/150235)
**Author:** [@Dan](https://discuss.elastic.co/u/Dan)\ **Replies:** 0\ **Last updated:** [September 27, 2018, 5:56pm UTC](https://discuss.elastic.co/t/how-to-escape-ssl-certificate-verification-in-elastic-search-input-for-logstash/150235 "2018-09-27T17:56:07Z")
I Installed read-only rest plugin for Elastic search 5.6.1 And I changed elasticsearch.yml file for security and SSL. They are working fine but SSL I generated locally. When I am using elastic search as output in logs… --- ## [How logstash is working?](https://discuss.elastic.co/t/how-logstash-is-working/149143)
**Author:** [@Gosforth](https://discuss.elastic.co/u/Gosforth)\ **Replies:** 2\ **Last updated:** [September 27, 2018, 5:35pm UTC](https://discuss.elastic.co/t/how-logstash-is-working/149143 "2018-09-27T17:35:28Z")
I try lo load 500 thousands MySQL rows into Elasticsearch. It takes 10h to do that. 16GB memory, i7 processor; enough resources to fly. All components on one server (no LAN network between). It takes huuuuge amount of … --- ## [Persistent queue not creating any file inside the configured directory](https://discuss.elastic.co/t/persistent-queue-not-creating-any-file-inside-the-configured-directory/150212)
**Author:** [@Nikhil\_Jaiswal](https://discuss.elastic.co/u/Nikhil_Jaiswal)\ **Replies:** 1\ **Last updated:** [September 27, 2018, 5:23pm UTC](https://discuss.elastic.co/t/persistent-queue-not-creating-any-file-inside-the-configured-directory/150212 "2018-09-27T17:23:23Z")
Hi All, I have configured persistent queue in logstash.yml but after restarting logstash i am not able to see any file which logstash usually create. please find my configuration below: queue.type: persisted path.que… --- ## [How to get Number type by using Grok](https://discuss.elastic.co/t/how-to-get-number-type-by-using-grok/150084)
**Author:** [@saad](https://discuss.elastic.co/u/saad)\ **Replies:** 3\ **Last updated:** [September 27, 2018, 3:36pm UTC](https://discuss.elastic.co/t/how-to-get-number-type-by-using-grok/150084 "2018-09-27T15:36:32Z")
Hi, I have this field below and I want to use grok to get the number 427 at the end and add it to new field as type number I did the following grok { match =\> { "name" =\> "%{GREEDYDATA}device: %{NUMBER:concurrent… --- ## [How bring the browser information using logstash](https://discuss.elastic.co/t/how-bring-the-browser-information-using-logstash/149113)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 12\ **Last updated:** [September 27, 2018, 3:15pm UTC](https://discuss.elastic.co/t/how-bring-the-browser-information-using-logstash/149113 "2018-09-27T15:15:37Z")
HI Team, I have browser information in one field and how could i fetch browser version and name everything using logstash, userAgent = Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Ge… --- ## [How to avoid data loss in logstash?](https://discuss.elastic.co/t/how-to-avoid-data-loss-in-logstash/149582)
**Author:** [@Nikhil\_Jaiswal](https://discuss.elastic.co/u/Nikhil_Jaiswal)\ **Replies:** 5\ **Last updated:** [September 27, 2018, 2:35pm UTC](https://discuss.elastic.co/t/how-to-avoid-data-loss-in-logstash/149582 "2018-09-27T14:35:24Z")
Hi All, I observed there are data loss in my elasticsearch cluster, I am comparing the log count from two log monitoring tools,one is ELK and another a third party vendor. I have verified syslog configuration in all ne… --- ## [Partially flatten json structure in logstash filter](https://discuss.elastic.co/t/partially-flatten-json-structure-in-logstash-filter/150173)
**Author:** [@johnjingle](https://discuss.elastic.co/u/johnjingle)\ **Replies:** 3\ **Last updated:** [September 27, 2018, 2:16pm UTC](https://discuss.elastic.co/t/partially-flatten-json-structure-in-logstash-filter/150173 "2018-09-27T14:16:48Z")
Hi, We have { "recordA": { "recordB": { "someField1" : "john" "somefield2": "smith" } } } and wish to move the fields up one level so recordB disappears and the fields are directly under recordA. { "recordA"… --- ## [Logstash trying to connect to 127.0.0.1:9200 which is not there](https://discuss.elastic.co/t/logstash-trying-to-connect-to-127-0-0-1-9200-which-is-not-there/150089)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 3\ **Last updated:** [September 27, 2018, 2:09pm UTC](https://discuss.elastic.co/t/logstash-trying-to-connect-to-127-0-0-1-9200-which-is-not-there/150089 "2018-09-27T14:09:13Z")
After setting up everything. suddenly logstash just keep trying to connect to 127.0.0.1 my ES is running on IP address \[2018-09-26T16:09:48,981\]\[INFO \]\[logstash.pipeline \] Starting pipeline {:pipeline\_id=\>"main"… --- ## [MySql Query is taking more to execute](https://discuss.elastic.co/t/mysql-query-is-taking-more-to-execute/149939)
**Author:** [@JoseRohit](https://discuss.elastic.co/u/JoseRohit)\ **Replies:** 14\ **Last updated:** [September 27, 2018, 12:58pm UTC](https://discuss.elastic.co/t/mysql-query-is-taking-more-to-execute/149939 "2018-09-27T12:58:37Z")
I'm migrating MYSQL data to ElasticSearch using logstash. The table has more than 9 crore records. I have tested the query in mysql workbeanch which is taking 0.032 sec to execute. When i run it from logstash, its taki… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=677) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=679)