# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=449 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 450 --- ## [Stashing Event Store logs](https://discuss.elastic.co/t/stashing-event-store-logs/207149)
**Author:** [@dj0b](https://discuss.elastic.co/u/dj0b)\ **Replies:** 2\ **Last updated:** [November 8, 2019, 5:05pm UTC](https://discuss.elastic.co/t/stashing-event-store-logs/207149 "2019-11-08T17:05:30Z")
Good Afternoon, I am wondering if anyone has any information on the best approach for stashing Event Store logs? Particularly, I am interested in the construction of the Message within the Event Store log, and how best … --- ## [Multiple filebeat to one logstash. How to optimize the configuration](https://discuss.elastic.co/t/multiple-filebeat-to-one-logstash-how-to-optimize-the-configuration/207045)
**Author:** [@katara](https://discuss.elastic.co/u/katara)\ **Replies:** 3\ **Last updated:** [November 8, 2019, 3:47pm UTC](https://discuss.elastic.co/t/multiple-filebeat-to-one-logstash-how-to-optimize-the-configuration/207045 "2019-11-08T15:47:56Z")
I have 10 servers that i have Filebeat installed in. Each server monitors 2 applications, a total of 20 applications. I have one Logstash server which collects all the above logs and passes it to Elasticsearch after fil… --- ## [What are best practices for matching incoming log sources?](https://discuss.elastic.co/t/what-are-best-practices-for-matching-incoming-log-sources/206496)
**Author:** [@meatwad](https://discuss.elastic.co/u/meatwad)\ **Replies:** 3\ **Last updated:** [November 8, 2019, 3:42pm UTC](https://discuss.elastic.co/t/what-are-best-practices-for-matching-incoming-log-sources/206496 "2019-11-08T15:42:50Z")
Hi, I'm getting started on sorting incoming log sources and was wondering what best practice is for doing so. For example, I'm collecting logs from a variety of sources including Palo Alto firewalls, Aruba NAC, PulseSe… --- ## [Docker Logstash internal logging to file?](https://discuss.elastic.co/t/docker-logstash-internal-logging-to-file/206951)
**Author:** [@sam.brooks](https://discuss.elastic.co/u/sam.brooks)\ **Replies:** 2\ **Last updated:** [November 8, 2019, 3:20pm UTC](https://discuss.elastic.co/t/docker-logstash-internal-logging-to-file/206951 "2019-11-08T15:20:25Z")
I'm having a similar problem as two posters before me, whose posts did not get resolved: I am hoping that this time, we can find a solution. I am using the Logstash docker container (docker.elastic.co/logstash/logsta… --- ## [Issue with date](https://discuss.elastic.co/t/issue-with-date/207028)
**Author:** [@mbrown007](https://discuss.elastic.co/u/mbrown007)\ **Replies:** 1\ **Last updated:** [November 8, 2019, 3:07pm UTC](https://discuss.elastic.co/t/issue-with-date/207028 "2019-11-08T15:07:47Z")
I have found when I add an entry there is no issue but when I attempt to modify the record I see the following and at no time do I have anything in my log file that says "tco". The only location for this is where It says… --- ## [Combine request and response based on field in log - ELK](https://discuss.elastic.co/t/combine-request-and-response-based-on-field-in-log-elk/207073)
**Author:** [@pdkgamage](https://discuss.elastic.co/u/pdkgamage)\ **Replies:** 1\ **Last updated:** [November 8, 2019, 3:02pm UTC](https://discuss.elastic.co/t/combine-request-and-response-based-on-field-in-log-elk/207073 "2019-11-08T15:02:51Z")
We have ELK setup with filebeat, logstash and elasticsearch and kibana. I need aggregate request and response at the logstash. I have configured pipeline configuration as below. now log aggregation working without any i… --- ## [Kibana index is not showing](https://discuss.elastic.co/t/kibana-index-is-not-showing/207112)
**Author:** [@saravana\_hariharan](https://discuss.elastic.co/u/saravana_hariharan)\ **Replies:** 1\ **Last updated:** [November 8, 2019, 3:00pm UTC](https://discuss.elastic.co/t/kibana-index-is-not-showing/207112 "2019-11-08T15:00:34Z")
In kibana its not showing index. in elasticsearch box but file successfully configured in logstash indexpattern below rectangular box cant show the elasticsearch indices how to i get that regards saravanan --- ## [Replace old documents](https://discuss.elastic.co/t/replace-old-documents/207126)
**Author:** [@shankaran](https://discuss.elastic.co/u/shankaran)\ **Replies:** 0\ **Last updated:** [November 8, 2019, 2:10pm UTC](https://discuss.elastic.co/t/replace-old-documents/207126 "2019-11-08T14:10:59Z")
in My application Every day 6 times (4hrs per run) index will be updated by latest exchange rates documents. Any one could guide me on best approach how to delete my old documents (having SourceCurrency=USD) from logsta… --- ## [Merging two documents from existing ES indices into a new document](https://discuss.elastic.co/t/merging-two-documents-from-existing-es-indices-into-a-new-document/206794)
**Author:** [@domino](https://discuss.elastic.co/u/domino)\ **Replies:** 3\ **Last updated:** [November 8, 2019, 7:55am UTC](https://discuss.elastic.co/t/merging-two-documents-from-existing-es-indices-into-a-new-document/206794 "2019-11-08T07:55:44Z")
Hi all I'd like to have logstash running through all documents in the index "timeseries" and merge them with documents in another index called "contract". The resulting new merged document should be written into a desti… --- ## [Filebeat to logstash using csv filter not working](https://discuss.elastic.co/t/filebeat-to-logstash-using-csv-filter-not-working/206955)
**Author:** [@nareshreddyp](https://discuss.elastic.co/u/nareshreddyp)\ **Replies:** 2\ **Last updated:** [November 8, 2019, 7:48am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-using-csv-filter-not-working/206955 "2019-11-08T07:48:45Z")
Hello, I have a csv file with following entries (Each column is already with double quotes) "2019/11/07 13:19:18.538";"138.106.61.153";"sxisl";"sxist0";"mxobjecttype:(""BUSINESSOBJECT"")";"xx";"0";"2";"1";"1";"118820"; … --- ## [Data getting double in target table](https://discuss.elastic.co/t/data-getting-double-in-target-table/207041)
**Author:** [@Sumati\_Jain](https://discuss.elastic.co/u/Sumati_Jain)\ **Replies:** 0\ **Last updated:** [November 8, 2019, 4:29am UTC](https://discuss.elastic.co/t/data-getting-double-in-target-table/207041 "2019-11-08T04:29:34Z")
Hi everyone, I'm fetching data from Elasticsearch and putting into Bigquery through Logstash pipeline. When I'm fetching 15 days of data then I'm getting same number of records as source. No issues with 15 days of data… --- ## [Logstash with TCP input, TCP connection leak](https://discuss.elastic.co/t/logstash-with-tcp-input-tcp-connection-leak/207032)
**Author:** [@Fawlk](https://discuss.elastic.co/u/Fawlk)\ **Replies:** 0\ **Last updated:** [November 8, 2019, 2:43am UTC](https://discuss.elastic.co/t/logstash-with-tcp-input-tcp-connection-leak/207032 "2019-11-08T02:43:14Z")
I have setup Logstash to receive log via TCP with only one client sending logs. After run for a while, I found that TCP Established connections and never timeout. After googled, there's a parameter data\_timeout that I co… --- ## [How to fix that errors?](https://discuss.elastic.co/t/how-to-fix-that-errors/207027)
**Author:** [@pbona](https://discuss.elastic.co/u/pbona)\ **Replies:** 0\ **Last updated:** [November 8, 2019, 12:51am UTC](https://discuss.elastic.co/t/how-to-fix-that-errors/207027 "2019-11-08T00:51:03Z")
Followed this document: https://www.elastic.co/blog/how-to-setup-tls-for-elasticsearch-kibana-logstash-filebeat-with-offline-install-in-linux\](https://www.elastic.co/blog/how-to-setup-tls-for-elasticsearch-kibana-logsta… --- ## [How to transform JDBC input into a specific JSON format](https://discuss.elastic.co/t/how-to-transform-jdbc-input-into-a-specific-json-format/207012)
**Author:** [@seePhor](https://discuss.elastic.co/u/seePhor)\ **Replies:** 1\ **Last updated:** [November 7, 2019, 11:40pm UTC](https://discuss.elastic.co/t/how-to-transform-jdbc-input-into-a-specific-json-format/207012 "2019-11-07T23:40:56Z")
Hello, New to Logstash and trying to backfill a new Elasticsearch instance from a MySQL table using the JDBC input plugin. I have the input plugin working, but right now the output is just going to stdout. I want to tr… --- ## [Manticore issue with Azure plugin](https://discuss.elastic.co/t/manticore-issue-with-azure-plugin/207005)
**Author:** [@ablanco722](https://discuss.elastic.co/u/ablanco722)\ **Replies:** 2\ **Last updated:** [November 7, 2019, 9:37pm UTC](https://discuss.elastic.co/t/manticore-issue-with-azure-plugin/207005 "2019-11-07T21:37:40Z")
So working on setting up the Azure plugin and keep on getting this strange stack trace error. Wondering if anyone has seen this \[ERROR\]\[logstash.config.sourceloader\] Could not fetch all the sources {:exception=\>LogStash… --- ## [How to validate incoming JSON body using Http-Input plugin](https://discuss.elastic.co/t/how-to-validate-incoming-json-body-using-http-input-plugin/206956)
**Author:** [@iFriT](https://discuss.elastic.co/u/iFriT)\ **Replies:** 2\ **Last updated:** [November 7, 2019, 8:10pm UTC](https://discuss.elastic.co/t/how-to-validate-incoming-json-body-using-http-input-plugin/206956 "2019-11-07T20:10:24Z")
Dears Elastic, Using Http-Input plugin, I would like to know if there is a way to validate incoming Message (JSON) against a pre-defined JSON schema and return a specific response for example (Invalid Message) before pa… --- ## [Sending new fields to exisiting Elastic document, losing old data](https://discuss.elastic.co/t/sending-new-fields-to-exisiting-elastic-document-losing-old-data/206992)
**Author:** [@EdgeSync](https://discuss.elastic.co/u/EdgeSync)\ **Replies:** 1\ **Last updated:** [November 7, 2019, 5:30pm UTC](https://discuss.elastic.co/t/sending-new-fields-to-exisiting-elastic-document-losing-old-data/206992 "2019-11-07T17:30:07Z")
I have an index in elastic, which is fed by two pipelines. pipeline 1 provides the base data for the document fields: id, name, number pipeline 2 needs to add new fields to the existing document based on the ID. fiel… --- ## [Geoip.location is not displaying in kibana when set up in logstash conf](https://discuss.elastic.co/t/geoip-location-is-not-displaying-in-kibana-when-set-up-in-logstash-conf/206980)
**Author:** [@Apollyon](https://discuss.elastic.co/u/Apollyon)\ **Replies:** 2\ **Last updated:** [November 7, 2019, 4:57pm UTC](https://discuss.elastic.co/t/geoip-location-is-not-displaying-in-kibana-when-set-up-in-logstash-conf/206980 "2019-11-07T16:57:16Z")
hello, I am on ELK 7.4. When using geoip in a Logstash conf the geoip.location doesn't appear to be displayed in the index fields in Kibana. Below is the Logstash conf - showing the fields filter input under the geoip … --- ## [How to get data from other index (indexb) and add it to current index (temp) based on common field](https://discuss.elastic.co/t/how-to-get-data-from-other-index-indexb-and-add-it-to-current-index-temp-based-on-common-field/206965)
**Author:** [@aasoft](https://discuss.elastic.co/u/aasoft)\ **Replies:** 4\ **Last updated:** [November 7, 2019, 4:35pm UTC](https://discuss.elastic.co/t/how-to-get-data-from-other-index-indexb-and-add-it-to-current-index-temp-based-on-common-field/206965 "2019-11-07T16:35:22Z")
I have two indexes: current one: temp PUT temp/doc/1 {"Field1" : "data1", "Field2" : "data2", "cnt\_no":"1"} and other index: indexb PUT indexb/doc/1 {"Field3" : "data3", "Field4" : "data4", "cnt\_no":"1"} I would li… --- ## [Syslog Input - Grok Error](https://discuss.elastic.co/t/syslog-input-grok-error/206752)
**Author:** [@gerard.ramos](https://discuss.elastic.co/u/gerard.ramos)\ **Replies:** 2\ **Last updated:** [November 7, 2019, 4:28pm UTC](https://discuss.elastic.co/t/syslog-input-grok-error/206752 "2019-11-07T16:28:09Z")
Hello! I have this input: \<189\>date=2019-11-06 time=09:18:24 devname="FW" devid="FG200D4Q645" logid="0000000011" type="traffic" subtype="forward" level="notice" vd="josep" eventtime=1573045874 srcip=192.168.1.1 srcpor… --- ## [Error after v5.6 upgrade from v5.5](https://discuss.elastic.co/t/error-after-v5-6-upgrade-from-v5-5/205704)
**Author:** [@ronaldom](https://discuss.elastic.co/u/ronaldom)\ **Replies:** 3\ **Last updated:** [November 7, 2019, 3:48pm UTC](https://discuss.elastic.co/t/error-after-v5-6-upgrade-from-v5-5/205704 "2019-11-07T15:48:24Z")
Hi guys, We just recently upgrade our Elastic Stack from v5.5 to v5.6. We encountered the error below when we tried running logstash to populate the new v5.6 indices. Any idea what's going on here? We already re-install… --- ## [Replace empty strings with Ruby code](https://discuss.elastic.co/t/replace-empty-strings-with-ruby-code/206590)
**Author:** [@vinay.rao](https://discuss.elastic.co/u/vinay.rao)\ **Replies:** 5\ **Last updated:** [November 7, 2019, 3:47pm UTC](https://discuss.elastic.co/t/replace-empty-strings-with-ruby-code/206590 "2019-11-07T15:47:39Z")
Dear All, I am here seeking your help on the following. I have empty strings in fields and i need to replace it with NULL or NILL . Since am very new to Ruby, Can you help me with the complete query to achieve the abov… --- ## [How to parse XML?](https://discuss.elastic.co/t/how-to-parse-xml/206238)
**Author:** [@ja123](https://discuss.elastic.co/u/ja123)\ **Replies:** 7\ **Last updated:** [November 7, 2019, 2:58pm UTC](https://discuss.elastic.co/t/how-to-parse-xml/206238 "2019-11-07T14:58:56Z")
I'm trying to parse the following xml from an api through xml plugin. \ **Author:** [@RafaelPeggau](https://discuss.elastic.co/u/RafaelPeggau)\ **Replies:** 1\ **Last updated:** [November 7, 2019, 2:52pm UTC](https://discuss.elastic.co/t/preparar-documento-para-indexar-no-elasticseach/206973 "2019-11-07T14:52:39Z") Montei uma estrutura, ELK. vou configurar o logstash com jdbc, para fazer consultas cron em uma base sql. E inserir no elastic, para depois montar alguns dashboards com o Kibana. Perfeito isso sei que funciona. Agora a… --- ## [Simplify "host" field output from logstash](https://discuss.elastic.co/t/simplify-host-field-output-from-logstash/206850)
**Author:** [@thadc](https://discuss.elastic.co/u/thadc)\ **Replies:** 0\ **Last updated:** [November 6, 2019, 6:37pm UTC](https://discuss.elastic.co/t/simplify-host-field-output-from-logstash/206850 "2019-11-06T18:37:56Z")
Hello, The current output of the default host field from my logstash is: "host": { "name": " name\_of\_host " }, I have a need for the host to appear simply as: "host": " name\_of\_host " The reason for this needed ch… --- ## [Getting error while installing ELK](https://discuss.elastic.co/t/getting-error-while-installing-elk/206327)
**Author:** [@Shradha\_Bharti](https://discuss.elastic.co/u/Shradha_Bharti)\ **Replies:** 2\ **Last updated:** [November 7, 2019, 10:37am UTC](https://discuss.elastic.co/t/getting-error-while-installing-elk/206327 "2019-11-07T10:37:23Z")
HI I am upgrading the elk version of my server ..previosuly it was 2.4.6 now ,we are doing 7.3.0 while installing all the rpms ,m getting below error in logstash \[2019-11-04T00:00:16,098\]\[ERROR\]\[logstash.plugins.regi… --- ## [Getting PacketBeat Raw TCP Payloads](https://discuss.elastic.co/t/getting-packetbeat-raw-tcp-payloads/206926)
**Author:** [@alpayk](https://discuss.elastic.co/u/alpayk)\ **Replies:** 0\ **Last updated:** [November 7, 2019, 9:01am UTC](https://discuss.elastic.co/t/getting-packetbeat-raw-tcp-payloads/206926 "2019-11-07T09:01:42Z")
Is it possible to extract and send the network traffic to logstash as a byte array using PacketBeat? I want to post-process the raw network traffic bytes after sending them to logstash because of the heavy network load. … --- ## [Logstash/beats keeps track files shipped](https://discuss.elastic.co/t/logstash-beats-keeps-track-files-shipped/206707)
**Author:** [@taos](https://discuss.elastic.co/u/taos)\ **Replies:** 1\ **Last updated:** [November 7, 2019, 7:47am UTC](https://discuss.elastic.co/t/logstash-beats-keeps-track-files-shipped/206707 "2019-11-07T07:47:45Z")
Hi all, I have data/text files in directory generated by date -- path date, e.g 20191011. Q. Doese any way or plugins in that Logstash, beats keeps track files shipped to ELK in order to ignore them? Thanks --- ## [I want to get sequence information！](https://discuss.elastic.co/t/i-want-to-get-sequence-information/206910)
**Author:** [@smash](https://discuss.elastic.co/u/smash)\ **Replies:** 0\ **Last updated:** [November 7, 2019, 7:12am UTC](https://discuss.elastic.co/t/i-want-to-get-sequence-information/206910 "2019-11-07T07:12:48Z")
Hello After getting the number of arrays and storing them in a custom field To obtain array information using a custom field as an array parameter I don't know how to expand custom fields. Can anyone tell me? sample… --- ## [Sync mongodb data to elasticsearch(update if exists or insert) using logstash](https://discuss.elastic.co/t/sync-mongodb-data-to-elasticsearch-update-if-exists-or-insert-using-logstash/206897)
**Author:** [@hannu98](https://discuss.elastic.co/u/hannu98)\ **Replies:** 0\ **Last updated:** [November 7, 2019, 6:03am UTC](https://discuss.elastic.co/t/sync-mongodb-data-to-elasticsearch-update-if-exists-or-insert-using-logstash/206897 "2019-11-07T06:03:42Z")
I just want to sync the data from mongodb to elastic using logstash. Its working good, when any new record comes in mongodb, logstash pushes into elastic. But when I update any record in mongodb then it does not change i… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=448) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=450)