# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=880 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 881 --- ## [How to use mutate to flatten dataset](https://discuss.elastic.co/t/how-to-use-mutate-to-flatten-dataset/106529)
**Author:** [@charmingmel](https://discuss.elastic.co/u/charmingmel)\ **Replies:** 0\ **Last updated:** [November 6, 2017, 1:17pm UTC](https://discuss.elastic.co/t/how-to-use-mutate-to-flatten-dataset/106529 "2017-11-06T13:17:13Z")
Hi there, I am new to logstash. Please I need a little help transforming my dataset to a file so that it can be easy to query. Data set: { "parent\_id": "e4a085a9-0ec0-d4f4-0047-eab10eca4f79", "year": "2017", "month": … --- ## [Can't grok multiline logs](https://discuss.elastic.co/t/cant-grok-multiline-logs/106435)
**Author:** [@p0f1g1st](https://discuss.elastic.co/u/p0f1g1st)\ **Replies:** 1\ **Last updated:** [November 6, 2017, 11:51am UTC](https://discuss.elastic.co/t/cant-grok-multiline-logs/106435 "2017-11-06T11:51:51Z")
I have logs where each event is: ExitNode FF33F91CC06B6CC5C3EE804E7D8DBE42CB5707F9 Published 2017-11-05 02:55:09 LastStatus 2017-11-05 04:02:27 ExitAddress 66.42.224.235 2017-11-05 04:06:26 I tried to use multiline: i… --- ## [Logstash and drop filter for Windows event](https://discuss.elastic.co/t/logstash-and-drop-filter-for-windows-event/106322)
**Author:** [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Replies:** 2\ **Last updated:** [November 6, 2017, 11:49am UTC](https://discuss.elastic.co/t/logstash-and-drop-filter-for-windows-event/106322 "2017-11-06T11:49:42Z")
Hello, I'm trying to drop events using the following filter but its not working: if \[event\_id\] == 4658 and \[event\_data.ProcessName\] == "C:\\Windows\\System32\\CpqMgmt\\cqmghost/cqmghost.exe" { drop {} } I've tried usin… --- ## [Ruby code not getting executed](https://discuss.elastic.co/t/ruby-code-not-getting-executed/106479)
**Author:** [@Ganesh\_Venkataraman](https://discuss.elastic.co/u/Ganesh_Venkataraman)\ **Replies:** 5\ **Last updated:** [November 6, 2017, 9:19am UTC](https://discuss.elastic.co/t/ruby-code-not-getting-executed/106479 "2017-11-06T09:19:52Z")
Hi, The ruby code in BOLD is not getting executed. Any idea? The first ruby code is getting executed and tags are getting added but not the second one. filter { csv { separator =\> "," columns =\> \["tds\_audittimestamp… --- ## [Add or update field in filter or elsewhere](https://discuss.elastic.co/t/add-or-update-field-in-filter-or-elsewhere/106297)
**Author:** [@Ram\_Sharan\_Mittal](https://discuss.elastic.co/u/Ram_Sharan_Mittal)\ **Replies:** 3\ **Last updated:** [November 6, 2017, 8:22am UTC](https://discuss.elastic.co/t/add-or-update-field-in-filter-or-elsewhere/106297 "2017-11-06T08:22:35Z")
based on \[@metadata\]\[category\] field , log need to uploaded to different file location. if message have some pattern then need to update \[@metadata\]\[category\] , I am doing in filter filter { if \[message\] =~ ".regextp… --- ## [Handling related logs](https://discuss.elastic.co/t/handling-related-logs/106420)
**Author:** [@novoselrok](https://discuss.elastic.co/u/novoselrok)\ **Replies:** 3\ **Last updated:** [November 6, 2017, 8:16am UTC](https://discuss.elastic.co/t/handling-related-logs/106420 "2017-11-06T08:16:13Z")
I have three types of logs, e.g. A, B and C, where A is the parent of both B and C. The A log comes first and is saved into elastic search, but B and C can come minutes or hours later. B and C contain the reference to A… --- ## [Remove a line block according to a certain string value](https://discuss.elastic.co/t/remove-a-line-block-according-to-a-certain-string-value/106210)
**Author:** [@cy8el1us](https://discuss.elastic.co/u/cy8el1us)\ **Replies:** 2\ **Last updated:** [November 6, 2017, 8:02am UTC](https://discuss.elastic.co/t/remove-a-line-block-according-to-a-certain-string-value/106210 "2017-11-06T08:02:24Z")
I am new with ELK and I am trying to skip/remove a line block according to a certain string value from a grok tag. 2015-05-21 18:53:20,560 -3- \[Class\] A certain string 2015-05-21 18:53:20,560 -3- \[Class\] Col1 Col2… --- ## [How to calculate timediff of two fields in the same event](https://discuss.elastic.co/t/how-to-calculate-timediff-of-two-fields-in-the-same-event/106417)
**Author:** [@Ganesh\_Venkataraman](https://discuss.elastic.co/u/Ganesh_Venkataraman)\ **Replies:** 2\ **Last updated:** [November 6, 2017, 7:58am UTC](https://discuss.elastic.co/t/how-to-calculate-timediff-of-two-fields-in-the-same-event/106417 "2017-11-06T07:58:14Z")
LOGSTASH - I would like to calculate time taken in seconds for two timestamps in the same event and store it in a different field say "e2e\_latency\_in\_seconds". The timestamps are in different format. I have successfully… --- ## [How load template with logstash using elasticsearch output plugin](https://discuss.elastic.co/t/how-load-template-with-logstash-using-elasticsearch-output-plugin/106325)
**Author:** [@citanionarrei](https://discuss.elastic.co/u/citanionarrei)\ **Replies:** 2\ **Last updated:** [November 6, 2017, 7:49am UTC](https://discuss.elastic.co/t/how-load-template-with-logstash-using-elasticsearch-output-plugin/106325 "2017-11-06T07:49:11Z")
hi friends i want to load my template on the output of logstash to map correctly my json i'm trying this but template is not loaded output { elasticsearch { index =\> "ciao-%{+YYYY.MM.dd}" document\_type =\> "ciao\_… --- ## [Combining multiple documents based on ID](https://discuss.elastic.co/t/combining-multiple-documents-based-on-id/105534)
**Author:** [@qube](https://discuss.elastic.co/u/qube)\ **Replies:** 21\ **Last updated:** [November 6, 2017, 7:22am UTC](https://discuss.elastic.co/t/combining-multiple-documents-based-on-id/105534 "2017-11-06T07:22:34Z")
Hello logstash! I'm new to Elastic stack. We are transferring data from Sql Server to ElasticSearch using LogStash's jdbc plugin. So far, it's fine. Normally jdbc plugin creates a new document for each Sql Server row, … --- ## [Fluent issue with logstash](https://discuss.elastic.co/t/fluent-issue-with-logstash/106471)
**Author:** [@rehan](https://discuss.elastic.co/u/rehan)\ **Replies:** 0\ **Last updated:** [November 6, 2017, 6:32am UTC](https://discuss.elastic.co/t/fluent-issue-with-logstash/106471 "2017-11-06T06:32:32Z")
Hi, I am trying to push logs from fluentd to logstash and to achieve this I followed the link https://www.elastic.co/guide/en/logstash/current/plugins-codecs-fluent.html#\_installation\_146 Nothing is clear in this docu… --- ## [Logstash HTTP input plugin hangs with large amount connections in WAIT\_CLOSE status](https://discuss.elastic.co/t/logstash-http-input-plugin-hangs-with-large-amount-connections-in-wait-close-status/106463)
**Author:** [@tishion](https://discuss.elastic.co/u/tishion)\ **Replies:** 0\ **Last updated:** [November 6, 2017, 5:23am UTC](https://discuss.elastic.co/t/logstash-http-input-plugin-hangs-with-large-amount-connections-in-wait-close-status/106463 "2017-11-06T05:23:57Z")
we have 3 Logsasth instances with HTTP input plugin running on port 9600. All of the three instances will hang with large amount connections in WAIT\_CLOSE status after several days running. Any suggestions to locate the … --- ## [Working with logstash with agentless](https://discuss.elastic.co/t/working-with-logstash-with-agentless/106418)
**Author:** [@Offer\_Sasson](https://discuss.elastic.co/u/Offer_Sasson)\ **Replies:** 2\ **Last updated:** [November 5, 2017, 9:20pm UTC](https://discuss.elastic.co/t/working-with-logstash-with-agentless/106418 "2017-11-05T21:20:35Z")
Hi, i am new to this.... I managed to install all beats agents and imported dashboards which were working great. Now i am asked to get the same data but with out installing any agent. this means that i have currently… --- ## [Logstash to elasticsearch question](https://discuss.elastic.co/t/logstash-to-elasticsearch-question/106415)
**Author:** [@feranwq](https://discuss.elastic.co/u/feranwq)\ **Replies:** 1\ **Last updated:** [November 5, 2017, 8:47pm UTC](https://discuss.elastic.co/t/logstash-to-elasticsearch-question/106415 "2017-11-05T20:47:38Z")
when the network broked,Which way to save the data and continue the transfer when the network is recovered? --- ## [How does input tcp plugin handle queue being full](https://discuss.elastic.co/t/how-does-input-tcp-plugin-handle-queue-being-full/106222)
**Author:** [@eprothro](https://discuss.elastic.co/u/eprothro)\ **Replies:** 3\ **Last updated:** [November 5, 2017, 7:37pm UTC](https://discuss.elastic.co/t/how-does-input-tcp-plugin-handle-queue-being-full/106222 "2017-11-05T19:37:04Z")
Input plugins handle back pressure from logstash independently, however I cannot find how, specifically, the tcp input plugin handles this. When the logstash queue is full and no longer accepting events from the tcp inp… --- ## [Aggregation doesn't work](https://discuss.elastic.co/t/aggregation-doesnt-work/106438)
**Author:** [@lkofman](https://discuss.elastic.co/u/lkofman)\ **Replies:** 0\ **Last updated:** [November 5, 2017, 5:26pm UTC](https://discuss.elastic.co/t/aggregation-doesnt-work/106438 "2017-11-05T17:26:21Z")
I try aggregate data by componentname a-and number how many times it appear in log input { beats{ type =\> beats port =\> 5044 } } filter{ if \[type\] == "beats" { grok { break\_on\_match =\> true keep\_empty\_captures… --- ## [Logstash jdbc and microsoft ad user](https://discuss.elastic.co/t/logstash-jdbc-and-microsoft-ad-user/106384)
**Author:** [@rolandbp](https://discuss.elastic.co/u/rolandbp)\ **Replies:** 1\ **Last updated:** [November 4, 2017, 4:51pm UTC](https://discuss.elastic.co/t/logstash-jdbc-and-microsoft-ad-user/106384 "2017-11-04T16:51:47Z")
Continuing the discussion from Mssql jdbc: --- ## [Logstash can not connect to Elasticsearch host](https://discuss.elastic.co/t/logstash-can-not-connect-to-elasticsearch-host/106340)
**Author:** [@Scoyle](https://discuss.elastic.co/u/Scoyle)\ **Replies:** 1\ **Last updated:** [November 4, 2017, 7:10am UTC](https://discuss.elastic.co/t/logstash-can-not-connect-to-elasticsearch-host/106340 "2017-11-04T07:10:00Z")
Hello! so i recently made a new ES cluster and that is up and running fine. I start sending info to it via logstash and logstash can not connect to the ES host. here is the dump from the logstash log on the remote serve… --- ## [Limit the number of config files logstash reads from a directory](https://discuss.elastic.co/t/limit-the-number-of-config-files-logstash-reads-from-a-directory/106249)
**Author:** [@Dulanjan\_Katuwawela](https://discuss.elastic.co/u/Dulanjan_Katuwawela)\ **Replies:** 4\ **Last updated:** [November 3, 2017, 11:10pm UTC](https://discuss.elastic.co/t/limit-the-number-of-config-files-logstash-reads-from-a-directory/106249 "2017-11-03T23:10:36Z")
I'll try and outline the problem we're trying to solve. We have a process that automatically writes out multiple files (the content, layouts, filenames are all unique and different) into a specific directory. The files c… --- ## [Beginner question add new field based on field in array of json objects](https://discuss.elastic.co/t/beginner-question-add-new-field-based-on-field-in-array-of-json-objects/106256)
**Author:** [@shrvnkmr](https://discuss.elastic.co/u/shrvnkmr)\ **Replies:** 2\ **Last updated:** [November 3, 2017, 5:49pm UTC](https://discuss.elastic.co/t/beginner-question-add-new-field-based-on-field-in-array-of-json-objects/106256 "2017-11-03T17:49:51Z")
We have input json objects looks like below { "ids": \[ { "id": 1, "name": "name1" }, { "id": 2, "name": "name2" } \], "codes": \[ { "code": "4" }, { "c… --- ## [How to provide a simple input text message to logstash](https://discuss.elastic.co/t/how-to-provide-a-simple-input-text-message-to-logstash/106347)
**Author:** [@ayushi.agrahari](https://discuss.elastic.co/u/ayushi.agrahari)\ **Replies:** 0\ **Last updated:** [November 3, 2017, 5:07pm UTC](https://discuss.elastic.co/t/how-to-provide-a-simple-input-text-message-to-logstash/106347 "2017-11-03T17:07:05Z")
I am following https://github.com/miguel-rodriguez/Docker-ELK-Alfresco-Monitoring and trying to monitor the alfresco logs on kibana.However the kibana is running on port 5601 and the status of kibana is green showing ela… --- ## [Split Nested Object](https://discuss.elastic.co/t/split-nested-object/104911)
**Author:** [@filipe.ferreira](https://discuss.elastic.co/u/filipe.ferreira)\ **Replies:** 2\ **Last updated:** [November 3, 2017, 4:54pm UTC](https://discuss.elastic.co/t/split-nested-object/104911 "2017-11-03T16:54:03Z")
I have a json log that looks like this: { "a" : "info", "data" : { "x1" : { Object }, "x2" : { Object }, "x3" : { Object } } } I would like to convert it to look like this: { "a" : "info", "data" : { "x1" : … --- ## [How to parse .gz (compressed file) using logstash](https://discuss.elastic.co/t/how-to-parse-gz-compressed-file-using-logstash/105950)
**Author:** [@jotsnadasari](https://discuss.elastic.co/u/jotsnadasari)\ **Replies:** 2\ **Last updated:** [November 3, 2017, 2:35pm UTC](https://discuss.elastic.co/t/how-to-parse-gz-compressed-file-using-logstash/105950 "2017-11-03T14:35:56Z")
Hi All, Does latest logstash support parsing .gz files using logstash? We have a requirement to get .gz files dynamically every hour. We do not know the file names and content. All we know is the file is compressed in … --- ## [Cannot package development logstash plugins for offline installation](https://discuss.elastic.co/t/cannot-package-development-logstash-plugins-for-offline-installation/106319)
**Author:** [@NetAppBlueDevil](https://discuss.elastic.co/u/NetAppBlueDevil)\ **Replies:** 0\ **Last updated:** [November 3, 2017, 1:49pm UTC](https://discuss.elastic.co/t/cannot-package-development-logstash-plugins-for-offline-installation/106319 "2017-11-03T13:49:43Z")
I'm following the procedures here https://www.elastic.co/guide/en/logstash/current/offline-plugins.html#building-offline-packs I need to install the development plugins to an environment that does not have internet con… --- ## [Basic logstash grok pattern - need help](https://discuss.elastic.co/t/basic-logstash-grok-pattern-need-help/106140)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 11\ **Last updated:** [November 3, 2017, 1:22pm UTC](https://discuss.elastic.co/t/basic-logstash-grok-pattern-need-help/106140 "2017-11-03T13:22:17Z")
Hi Guys, I have file this and I am trying to build a pattern for the same. Somehow this is not working. Can some one please help /var/log/badhost.txt = 125.118.243.164 216.245.208.133 199.59.242.150 104.145.232.3 … --- ## [Elasticsearch not recognizing my new indexes?](https://discuss.elastic.co/t/elasticsearch-not-recognizing-my-new-indexes/106294)
**Author:** [@bobwilly](https://discuss.elastic.co/u/bobwilly)\ **Replies:** 1\ **Last updated:** [November 3, 2017, 10:30am UTC](https://discuss.elastic.co/t/elasticsearch-not-recognizing-my-new-indexes/106294 "2017-11-03T10:30:44Z")
Hello! Since today I\`m not enable to create a new index. I've deleted my old index so I could start over. But ever since then I'm not enable to get it working again. I already got 3 indexes working (They are from some … --- ## [Logstash if condition issue](https://discuss.elastic.co/t/logstash-if-condition-issue/105843)
**Author:** [@ruranga](https://discuss.elastic.co/u/ruranga)\ **Replies:** 8\ **Last updated:** [November 3, 2017, 9:00am UTC](https://discuss.elastic.co/t/logstash-if-condition-issue/105843 "2017-11-03T09:00:07Z")
I have configured logstash filter as follows. But following filed "NFC\_APP\_TYPE" (newly added filed by using a if condition) doesn't appear in Kibana dash board filter{ ......... if \[AppCode\] == "505050" { mutate {ad… --- ## [Mapping json with logstash](https://discuss.elastic.co/t/mapping-json-with-logstash/106228)
**Author:** [@citanionarrei](https://discuss.elastic.co/u/citanionarrei)\ **Replies:** 4\ **Last updated:** [November 3, 2017, 7:55am UTC](https://discuss.elastic.co/t/mapping-json-with-logstash/106228 "2017-11-03T07:55:07Z")
hi friends I'm starting now to use elk and for this reason I'm not very experienced. I do not understand where and how to create a template on logstash to map my json. someone can help me with a practical example. I ha… --- ## [Trying to send postgresql query from Logstash to ElasticSearch](https://discuss.elastic.co/t/trying-to-send-postgresql-query-from-logstash-to-elasticsearch/106113)
**Author:** [@a\_dom](https://discuss.elastic.co/u/a_dom)\ **Replies:** 3\ **Last updated:** [November 3, 2017, 6:30am UTC](https://discuss.elastic.co/t/trying-to-send-postgresql-query-from-logstash-to-elasticsearch/106113 "2017-11-03T06:30:37Z")
Hi I'm following this tutorial (https://www.elastic.co/blog/logstash-jdbc-input-plugin) and I managed to recreate example 1 using my own table. But example 2 give me the following error and stops. ./bin/logstash -f logs… --- ## [Begginers help grok and check if data is making it into elastic search](https://discuss.elastic.co/t/begginers-help-grok-and-check-if-data-is-making-it-into-elastic-search/106099)
**Author:** [@burntfaceman](https://discuss.elastic.co/u/burntfaceman)\ **Replies:** 3\ **Last updated:** [November 3, 2017, 6:28am UTC](https://discuss.elastic.co/t/begginers-help-grok-and-check-if-data-is-making-it-into-elastic-search/106099 "2017-11-03T06:28:03Z")
Hi, I have just started to play with elasticstack for the first time. My objective to create a simple to use system for firewall rule log aggregation/searching. I have followed the basic guide to setup logstash, Elastic… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=879) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=881)