# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=536 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 537 --- ## [Get \_id document on query](https://discuss.elastic.co/t/get-id-document-on-query/183732)
**Author:** [@pr0b3l](https://discuss.elastic.co/u/pr0b3l)\ **Replies:** 2\ **Last updated:** [June 3, 2019, 1:51pm UTC](https://discuss.elastic.co/t/get-id-document-on-query/183732 "2019-06-03T13:51:24Z")
Hello! I need to get id in previous document. My query: query.json { "size" : 1, "sort" : \[ {"timestamp": "desc"} \], "query": { "query\_string": { "query": "AS:%{\[name\]}" } }, "\_id": "%{\[@metadata\]\[\_id\]}" } … --- ## [Logstash kibana](https://discuss.elastic.co/t/logstash-kibana/183319)
**Author:** [@krishnakekan](https://discuss.elastic.co/u/krishnakekan)\ **Replies:** 14\ **Last updated:** [June 3, 2019, 12:32pm UTC](https://discuss.elastic.co/t/logstash-kibana/183319 "2019-06-03T12:32:11Z")
not able to load data from mysql to elasticsearch using logstash? --- ## [Extract specifc field from greedydata](https://discuss.elastic.co/t/extract-specifc-field-from-greedydata/183122)
**Author:** [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Replies:** 4\ **Last updated:** [June 3, 2019, 12:04pm UTC](https://discuss.elastic.co/t/extract-specifc-field-from-greedydata/183122 "2019-06-03T12:04:11Z")
Hi Team, I am working on specific set of logs and it follows simple grok structure as shown below. %{TIMESTAMP} %{LOGLEVEL} %{METHODNAME} %{GREEDYDATA:LOGMESSAGE} the "LOGMESSAGE" field does contain some user related … --- ## [Merge two urls json at one doc](https://discuss.elastic.co/t/merge-two-urls-json-at-one-doc/183707)
**Author:** [@igor.alvarez.lanit](https://discuss.elastic.co/u/igor.alvarez.lanit)\ **Replies:** 6\ **Last updated:** [June 3, 2019, 10:37am UTC](https://discuss.elastic.co/t/merge-two-urls-json-at-one-doc/183707 "2019-06-03T10:37:20Z")
Hi, I use http\_poller imput logstash but i have two URLs and this create two docs.... I want to create only one doc with all fields of this urls, its is possible? input { http\_poller { urls =\> { station\_information… --- ## [Logstash infinity reboot when cant reach Elasticsearch](https://discuss.elastic.co/t/logstash-infinity-reboot-when-cant-reach-elasticsearch/183943)
**Author:** [@ep4sh](https://discuss.elastic.co/u/ep4sh)\ **Replies:** 0\ **Last updated:** [June 3, 2019, 11:27am UTC](https://discuss.elastic.co/t/logstash-infinity-reboot-when-cant-reach-elasticsearch/183943 "2019-06-03T11:27:04Z")
Good day! I have an issue with my Logstash. I have many JMS inputs and one Elasticsearch output. Output config is simple: elasticsearch { hosts =\> "${LS\_ES\_HOSTS}" ssl =\> true manage\_template =\> false document\_t… --- ## [Remote data store with ElasticSearch 7.1 and log4j2.11.1](https://discuss.elastic.co/t/remote-data-store-with-elasticsearch-7-1-and-log4j2-11-1/183935)
**Author:** [@Joseph\_Schooling](https://discuss.elastic.co/u/Joseph_Schooling)\ **Replies:** 0\ **Last updated:** [June 3, 2019, 10:29am UTC](https://discuss.elastic.co/t/remote-data-store-with-elasticsearch-7-1-and-log4j2-11-1/183935 "2019-06-03T10:29:07Z")
Hi, I am using ElasticSearch 7.1. It comes with log4j2.11.1.jar. The problem comes when I am trying to setup a remote data store with log4j2 running as a TcpSocketServer. I would then use log4j logging API in different … --- ## [Logstash 7.1 configuration file throws error](https://discuss.elastic.co/t/logstash-7-1-configuration-file-throws-error/183694)
**Author:** [@vinu89](https://discuss.elastic.co/u/vinu89)\ **Replies:** 9\ **Last updated:** [June 3, 2019, 10:16am UTC](https://discuss.elastic.co/t/logstash-7-1-configuration-file-throws-error/183694 "2019-06-03T10:16:45Z")
Hi, I have successfully configured elastic and kibana using SSL/TLS but when it comes to logstash it is throwing me below errrhosts =\> \["https://\*.\*.\*.\*:9200", "https://\*.\*.\*.\*:9200"\] index =\> "logstash-bindlive\_logs" s… --- ## [Elapsed filter ends in "elapsed\_end\_without\_start"](https://discuss.elastic.co/t/elapsed-filter-ends-in-elapsed-end-without-start/183874)
**Author:** [@ajayraghuraj](https://discuss.elastic.co/u/ajayraghuraj)\ **Replies:** 2\ **Last updated:** [June 3, 2019, 9:45am UTC](https://discuss.elastic.co/t/elapsed-filter-ends-in-elapsed-end-without-start/183874 "2019-06-03T09:45:22Z")
elapsed filter ends in "elapsed\_end\_without\_start". I am running on a VM with 1 vCPU . How do I get over this problem ? cpu cores : 1 /usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.config … --- ## [Use of metadata in ILM configuration](https://discuss.elastic.co/t/use-of-metadata-in-ilm-configuration/182918)
**Author:** [@Harm](https://discuss.elastic.co/u/Harm)\ **Replies:** 1\ **Last updated:** [June 3, 2019, 9:29am UTC](https://discuss.elastic.co/t/use-of-metadata-in-ilm-configuration/182918 "2019-06-03T09:29:19Z")
Hi! We have a logstash server that should be able to handle different types of beats. Previously we used an index pattern like this: index =\> "%{\[@metadata\]\[beat\]}-%{\[@metadata\]\[version\]}-%{+YYYY.MM.dd}" Now, I'm tryi… --- ## [Is it possible to make the logstash config file read values from another config file](https://discuss.elastic.co/t/is-it-possible-to-make-the-logstash-config-file-read-values-from-another-config-file/183855)
**Author:** [@mahmoud\_darwish](https://discuss.elastic.co/u/mahmoud_darwish)\ **Replies:** 2\ **Last updated:** [June 3, 2019, 8:49am UTC](https://discuss.elastic.co/t/is-it-possible-to-make-the-logstash-config-file-read-values-from-another-config-file/183855 "2019-06-03T08:49:12Z")
hello i have a big logstash config file that read near 110 fields in each event. then i have to do many tests and changes on each field, some of my changes uses a static variable. i ask if there a way to set all my var… --- ## [Logstash Http Configuration](https://discuss.elastic.co/t/logstash-http-configuration/183868)
**Author:** [@Vigneshwar](https://discuss.elastic.co/u/Vigneshwar)\ **Replies:** 2\ **Last updated:** [June 3, 2019, 8:26am UTC](https://discuss.elastic.co/t/logstash-http-configuration/183868 "2019-06-03T08:26:18Z")
Hi All, I am trying to create multiple indices based on the input received to logstash via http. Here is my logstash configuration, input { http { host =\> "0.0.0.0" port =\> "2020" type =\> "" } } output{ elastic… --- ## [Config Error : configuration is invalid. Reason: Expected one of #, =\>](https://discuss.elastic.co/t/config-error-configuration-is-invalid-reason-expected-one-of/183866)
**Author:** [@Markandayan\_P](https://discuss.elastic.co/u/Markandayan_P)\ **Replies:** 3\ **Last updated:** [June 3, 2019, 7:12am UTC](https://discuss.elastic.co/t/config-error-configuration-is-invalid-reason-expected-one-of/183866 "2019-06-03T07:12:24Z")
Here is my logstash configuration filter { grok { match =\> { "message" =\> { '%{DATA:req\_id} %{IPV4:host} %{DATA:machine\_name} %{DATA:location} %{DATA:zuid} %{DATA:zoid} %{DATA:thread\_id} %{DATA:class\_name} %{DATA:met… --- ## [Logstash RabbitMQ input plugin - How to define multiple routing\_key](https://discuss.elastic.co/t/logstash-rabbitmq-input-plugin-how-to-define-multiple-routing-key/183817)
**Author:** [@Ahmad\_Ahmadi](https://discuss.elastic.co/u/Ahmad_Ahmadi)\ **Replies:** 1\ **Last updated:** [June 3, 2019, 6:38am UTC](https://discuss.elastic.co/t/logstash-rabbitmq-input-plugin-how-to-define-multiple-routing-key/183817 "2019-06-03T06:38:23Z")
I'm using RabbitMQ plugin of Logstash to pull from RabbitMQ and push to Elasticsearch using this pipeline: input { rabbitmq { queue =\> "Elasticsearch\_Queue" host =\> "rabbitmq" exchange =\> "my… --- ## [Automate deletion of indices in aws elasticcluster](https://discuss.elastic.co/t/automate-deletion-of-indices-in-aws-elasticcluster/181682)
**Author:** [@Shubham\_Yadav1](https://discuss.elastic.co/u/Shubham_Yadav1)\ **Replies:** 7\ **Last updated:** [June 2, 2019, 1:00pm UTC](https://discuss.elastic.co/t/automate-deletion-of-indices-in-aws-elasticcluster/181682 "2019-06-02T13:00:55Z")
I have 3 questions: how to automate deletion of indices stored in aws\_es: I am using an aws\_elasticsearch to store data where each index is created everyday based on the time stamp they have. I am using -%{+YYYY.MM.d… --- ## [Logstash sincedb file not getting updated](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated/183860)
**Author:** [@Rocky\_RK](https://discuss.elastic.co/u/Rocky_RK)\ **Replies:** 0\ **Last updated:** [June 2, 2019, 12:57pm UTC](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated/183860 "2019-06-02T12:57:26Z")
I currently deployed ELK version 6.8.0 and i see my file in the sincedb not getting updated /data/registry-1 while the same config works fine in version 6.7.x. My input file section below.. input { file { path =\> … --- ## [How to monitor logstash performance?](https://discuss.elastic.co/t/how-to-monitor-logstash-performance/183859)
**Author:** [@Shubham\_Yadav1](https://discuss.elastic.co/u/Shubham_Yadav1)\ **Replies:** 1\ **Last updated:** [June 2, 2019, 12:04pm UTC](https://discuss.elastic.co/t/how-to-monitor-logstash-performance/183859 "2019-06-02T12:04:47Z")
I tried to do a load test on logstash which generated 10gb of data in 1 hour. In my system the logs are created every hour, a new index is created in my es everyday. But after watching the index size in elasticsearch i o… --- ## [Grok and match windows path](https://discuss.elastic.co/t/grok-and-match-windows-path/183560)
**Author:** [@Sam\_Kisada](https://discuss.elastic.co/u/Sam_Kisada)\ **Replies:** 3\ **Last updated:** [June 2, 2019, 5:04am UTC](https://discuss.elastic.co/t/grok-and-match-windows-path/183560 "2019-06-02T05:04:36Z")
Hi. I have some troubles with matching path to field here is path in table: log.file.path C:\\Program Files\\1cv8\\srvinfo\\reg\_1541\\d3367601-19f9-4ad2-afa0-078939886fc5\\1Cv8Log\\20190217000000.lgp in json it looks like: … --- ## [Accessing non document fields ElasticSearch Filter](https://discuss.elastic.co/t/accessing-non-document-fields-elasticsearch-filter/183844)
**Author:** [@James\_Summers](https://discuss.elastic.co/u/James_Summers)\ **Replies:** 0\ **Last updated:** [June 2, 2019, 2:26am UTC](https://discuss.elastic.co/t/accessing-non-document-fields-elasticsearch-filter/183844 "2019-06-02T02:26:49Z")
In converting some custom ruby code to run as a logstash pipeline, I added an elasticsearch filter pipeline, I'm able to do the query but am unable to access the sort field from the elasticsearch result. The Filter: e… --- ## [How does last\_run work for S3?](https://discuss.elastic.co/t/how-does-last-run-work-for-s3/183617)
**Author:** [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Replies:** 3\ **Last updated:** [June 1, 2019, 8:38pm UTC](https://discuss.elastic.co/t/how-does-last-run-work-for-s3/183617 "2019-06-01T20:38:17Z")
From what I've seen, the S3 input last\_run is a date. I didn't think you could use an S3 API to process objects stored from a certain time onward. I thought it needed the key of the last object processed as a marker. Is … --- ## [Uploading TEXT file using logstatsh](https://discuss.elastic.co/t/uploading-text-file-using-logstatsh/183829)
**Author:** [@Yoav\_Ben\_Moha](https://discuss.elastic.co/u/Yoav_Ben_Moha)\ **Replies:** 1\ **Last updated:** [June 1, 2019, 10:10pm UTC](https://discuss.elastic.co/t/uploading-text-file-using-logstatsh/183829 "2019-06-01T22:10:22Z")
Hello A 3rd party application generate a TEXT file contain 100K rows every 5 minutes. The columns in this TEXT file is seprated by ; How can i upload this TEXT file using logstatsh ? Thanks --- ## [How to input arguments via command line to conf?](https://discuss.elastic.co/t/how-to-input-arguments-via-command-line-to-conf/183798)
**Author:** [@Prapat\_Polchan](https://discuss.elastic.co/u/Prapat_Polchan)\ **Replies:** 1\ **Last updated:** [June 1, 2019, 11:49am UTC](https://discuss.elastic.co/t/how-to-input-arguments-via-command-line-to-conf/183798 "2019-06-01T11:49:04Z")
example i have one conf file input{ 'stdin'=\>$sss } i want to input $sss via command line example logstash -f a.conf "value replace to $sss" can possible in logstash conf ? --- ## [Logstash on K8S spammed with Connection Reset By Peer](https://discuss.elastic.co/t/logstash-on-k8s-spammed-with-connection-reset-by-peer/183815)
**Author:** [@triplewy](https://discuss.elastic.co/u/triplewy)\ **Replies:** 0\ **Last updated:** [June 1, 2019, 7:52am UTC](https://discuss.elastic.co/t/logstash-on-k8s-spammed-with-connection-reset-by-peer/183815 "2019-06-01T07:52:18Z")
I have deployed logstash on a remote k8s cluster and created a service for it that opened up port 5044 and 9600. I then use filebeat on my local machine to send logs to the service ip. It seems that filebeat is working … --- ## [Ignoring the 'pipelines.yml' file because modules or command line options are specified - RPM](https://discuss.elastic.co/t/ignoring-the-pipelines-yml-file-because-modules-or-command-line-options-are-specified-rpm/183795)
**Author:** [@Bhargav\_Ram\_Reddy](https://discuss.elastic.co/u/Bhargav_Ram_Reddy)\ **Replies:** 5\ **Last updated:** [May 31, 2019, 11:02pm UTC](https://discuss.elastic.co/t/ignoring-the-pipelines-yml-file-because-modules-or-command-line-options-are-specified-rpm/183795 "2019-05-31T23:02:37Z")
i have installed logstash through RPM on my ec2 instance and started \< ./logstash --path.settings /etc/logstash/ --path.config /etc/logstash/ /\> but i see below error. Can someone help out here? \[root@ip-10-1-1-133 bi… --- ## [Tagging for public IP address](https://discuss.elastic.co/t/tagging-for-public-ip-address/183433)
**Author:** [@psmaan1](https://discuss.elastic.co/u/psmaan1)\ **Replies:** 8\ **Last updated:** [May 31, 2019, 4:54pm UTC](https://discuss.elastic.co/t/tagging-for-public-ip-address/183433 "2019-05-31T16:54:23Z")
Hi, I am ingesting windows event logs in ELK setup (7.0.1). I need to add a tag to events e.g. "src\_public" or "dest\_public" whenever source IP or Destination IP or both in the logs are public IP address. For windows L… --- ## [Where does logstash store the json file it sends to elasticsearch or does it?](https://discuss.elastic.co/t/where-does-logstash-store-the-json-file-it-sends-to-elasticsearch-or-does-it/183757)
**Author:** [@decodeit](https://discuss.elastic.co/u/decodeit)\ **Replies:** 3\ **Last updated:** [May 31, 2019, 4:16pm UTC](https://discuss.elastic.co/t/where-does-logstash-store-the-json-file-it-sends-to-elasticsearch-or-does-it/183757 "2019-05-31T16:16:44Z")
I am sending a json from the REST API to logstash using http\_poller and it stores in elasticsearch. Does logstash stores this json? I am using docker-compose to run the containers, Thanks --- ## [Logstash::ConfigurationError-message](https://discuss.elastic.co/t/logstash-configurationerror-message/183364)
**Author:** [@soufiane1](https://discuss.elastic.co/u/soufiane1)\ **Replies:** 8\ **Last updated:** [May 31, 2019, 3:51pm UTC](https://discuss.elastic.co/t/logstash-configurationerror-message/183364 "2019-05-31T15:51:31Z")
Hello everyone, i am uploading a csv file with the following code : ############################################################################################################### input { file { path =\> "C:/Users/… --- ## [Codec Multiline Patterns doesn\`t work](https://discuss.elastic.co/t/codec-multiline-patterns-doesn-t-work/183758)
**Author:** [@Carlos.limaElaw](https://discuss.elastic.co/u/Carlos.limaElaw)\ **Replies:** 2\ **Last updated:** [May 31, 2019, 3:42pm UTC](https://discuss.elastic.co/t/codec-multiline-patterns-doesn-t-work/183758 "2019-05-31T15:42:25Z")
Hi, there I'm trying to use the codec multiline in my input, because some lines of my file csv come like this: 2019052238000853;RES: DADOS PARA O CADASTRO DO ELAW;2019-05-22 09:48:08;2019-05-22 10:06:05;;Consultores;n… --- ## [How to properly read and ingest XML file into Elastic](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582)
**Author:** [@edster](https://discuss.elastic.co/u/edster)\ **Replies:** 12\ **Last updated:** [May 31, 2019, 3:41pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582 "2019-05-31T15:41:41Z")
Hello, I have been trying to ingest an XML file into Elastic using Logstash. I have been using Xpath, target, source in the XML filter options and gsub for mutate filter, however, I keep running into an Error parsing xm… --- ## [Logstash multithreading?](https://discuss.elastic.co/t/logstash-multithreading/183036)
**Author:** [@dkuiate1](https://discuss.elastic.co/u/dkuiate1)\ **Replies:** 1\ **Last updated:** [May 31, 2019, 3:31pm UTC](https://discuss.elastic.co/t/logstash-multithreading/183036 "2019-05-31T15:31:03Z")
Hello everyone !! I have a logstash config file that has 15 inputs. When running logstash in debug mode i can realize that every input are read one by one. if one input is blocked, it tries several time to load it befor… --- ## [ElasticSearch output plugin create index with analyzers](https://discuss.elastic.co/t/elasticsearch-output-plugin-create-index-with-analyzers/183748)
**Author:** [@Samantha\_Molina\_Sier](https://discuss.elastic.co/u/Samantha_Molina_Sier)\ **Replies:** 0\ **Last updated:** [May 31, 2019, 2:42pm UTC](https://discuss.elastic.co/t/elasticsearch-output-plugin-create-index-with-analyzers/183748 "2019-05-31T14:42:05Z")
Hi all, I am working with ElasticSearch and Logstash 6.5, and creating an ES index. I would like to know if it possible to create the index with specific analyzer configuration. Thank you! This is the output configura… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=535) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=537)