# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=210 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 211 --- ## [Not able to get Floating value in Kibana](https://discuss.elastic.co/t/not-able-to-get-floating-value-in-kibana/279021)
**Author:** [@Saravana37](https://discuss.elastic.co/u/Saravana37)\ **Replies:** 0\ **Last updated:** [July 19, 2021, 6:57am UTC](https://discuss.elastic.co/t/not-able-to-get-floating-value-in-kibana/279021 "2021-07-19T06:57:20Z")
Hello All , I am trying to display one SQL column value TIME\_DIFF in Kibana. which is shown below. But it is displaying in Kibana as shown below . I am trying to convert this using mutate , \> filter { \> mutate { … --- ## [Logstash and Json, How to ingest this fragment:](https://discuss.elastic.co/t/logstash-and-json-how-to-ingest-this-fragment/278966)
**Author:** [@cibernicola](https://discuss.elastic.co/u/cibernicola)\ **Replies:** 2\ **Last updated:** [July 18, 2021, 11:02am UTC](https://discuss.elastic.co/t/logstash-and-json-how-to-ingest-this-fragment/278966 "2021-07-18T11:02:59Z")
I'm trying to ingest json data with this format: {"info":{"page":1,"pages":53431,"results":53431},"plugins":\[{"name":"Contact Form 7","slug":"contact-form-7","version":"5.4.2","author":"\ **Author:** [@Sunflower](https://discuss.elastic.co/u/Sunflower)\ **Replies:** 5\ **Last updated:** [July 18, 2021, 6:52am UTC](https://discuss.elastic.co/t/filter-with-grok-and-kv/278697 "2021-07-18T06:52:22Z") Hi, I'm trying to understand if I'm using the correct configuration for the following logs, or if there is a more efficient option for it. 2021-07-13T10:05:06.061Z 10.20.30.40 \<110\>1 2021-07-13T08:46:58Z 44.236.133.39… --- ## [Lost data in kibana but show in syslog logstash](https://discuss.elastic.co/t/lost-data-in-kibana-but-show-in-syslog-logstash/278894)
**Author:** [@quyennguyen](https://discuss.elastic.co/u/quyennguyen)\ **Replies:** 11\ **Last updated:** [July 17, 2021, 7:49pm UTC](https://discuss.elastic.co/t/lost-data-in-kibana-but-show-in-syslog-logstash/278894 "2021-07-17T19:49:07Z")
I have conf file in logstash like this input { beats { host =\> "0.0.0.0" port =\> 5044 ssl =\> false } } filter { if \[fileset\]\[name\] == "auth" { grok { match =\> { "message" =\> "%{SYSLOGT… --- ## [HELP! Logstash restarting after ip2location plugin update](https://discuss.elastic.co/t/help-logstash-restarting-after-ip2location-plugin-update/278957)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 4\ **Last updated:** [July 17, 2021, 6:33pm UTC](https://discuss.elastic.co/t/help-logstash-restarting-after-ip2location-plugin-update/278957 "2021-07-17T18:33:59Z")
I had an issue with logstash restarting so I updated ip2location plugin and now it keeps restarting every 15 seconds. The error before was ConcurrentModificationException: null. I use logstash v 7.4.2 and the plugin ver… --- ## [I am trying to pick a field from one event to another using aggregation filter](https://discuss.elastic.co/t/i-am-trying-to-pick-a-field-from-one-event-to-another-using-aggregation-filter/278955)
**Author:** [@Kamikaze\_K](https://discuss.elastic.co/u/Kamikaze_K)\ **Replies:** 1\ **Last updated:** [July 17, 2021, 6:10pm UTC](https://discuss.elastic.co/t/i-am-trying-to-pick-a-field-from-one-event-to-another-using-aggregation-filter/278955 "2021-07-17T18:10:01Z")
have log files that I am able to get fields based on two different if/grok statements and patterns. The output from the two are like below; { timestamp" =\> 2021-06-09T03:08:30.943Z, "Loc" =\> "91340", … --- ## [Ailed to parse field \[host\] of type \[text\] in document with id](https://discuss.elastic.co/t/ailed-to-parse-field-host-of-type-text-in-document-with-id/278963)
**Author:** [@Bhuwaneshwar](https://discuss.elastic.co/u/Bhuwaneshwar)\ **Replies:** 1\ **Last updated:** [July 17, 2021, 6:05pm UTC](https://discuss.elastic.co/t/ailed-to-parse-field-host-of-type-text-in-document-with-id/278963 "2021-07-17T18:05:00Z")
Dear Team I am getting below error on Logstash. "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field \[host\] of type \[text\] in document with id '4qrUs3oBqKFEAtQHASue'. Preview of field's value:… --- ## [Covert logs to JSON in output section of Logstash before it leaves output](https://discuss.elastic.co/t/covert-logs-to-json-in-output-section-of-logstash-before-it-leaves-output/278975)
**Author:** [@shadu88](https://discuss.elastic.co/u/shadu88)\ **Replies:** 1\ **Last updated:** [July 17, 2021, 5:59pm UTC](https://discuss.elastic.co/t/covert-logs-to-json-in-output-section-of-logstash-before-it-leaves-output/278975 "2021-07-17T17:59:29Z")
Hello ELkan's Hope all are safe!!! I'm using kustos output plugin to forward logs to azure data explore which only supports json format, I want to convert my syslog input logs on TCP port to JSON only in kustos output … --- ## [How to parse names like "This+is+name%2+field" in links?](https://discuss.elastic.co/t/how-to-parse-names-like-this-is-name-2-field-in-links/278858)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 8\ **Last updated:** [July 17, 2021, 5:23pm UTC](https://discuss.elastic.co/t/how-to-parse-names-like-this-is-name-2-field-in-links/278858 "2021-07-17T17:23:32Z")
Hi! I have a name field like: "This+is+name%2+field" How can I parse it in logstash grok filters so that I can have: "This is name 2 field" in Kibana? --- ## [Parsing Syslog with Logstash Grock Filter isn't working with Kibana](https://discuss.elastic.co/t/parsing-syslog-with-logstash-grock-filter-isnt-working-with-kibana/278844)
**Author:** [@olg32](https://discuss.elastic.co/u/olg32)\ **Replies:** 2\ **Last updated:** [July 17, 2021, 12:48am UTC](https://discuss.elastic.co/t/parsing-syslog-with-logstash-grock-filter-isnt-working-with-kibana/278844 "2021-07-17T00:48:45Z")
Hi, I have crated a very basic grok filter to parse Cisco Syslogs: input { udp { port =\> 5140 type =\> syslog } } filter { grok { match =\> { "message"=\> "%{TIMESTAMP\_ISO8601:Timestamp\_Local… --- ## [How do I conditionally parse a CSV into different columns?](https://discuss.elastic.co/t/how-do-i-conditionally-parse-a-csv-into-different-columns/278884)
**Author:** [@nsouth](https://discuss.elastic.co/u/nsouth)\ **Replies:** 1\ **Last updated:** [July 16, 2021, 7:57pm UTC](https://discuss.elastic.co/t/how-do-i-conditionally-parse-a-csv-into-different-columns/278884 "2021-07-16T19:57:58Z")
I have CSV log files where the columns change from line to line. The first field is an eventCode, which I can use to know which subsequent columns to expect. Here's a representative example. 1,sameTaskId,username,size 2… --- ## [Logstash mapping logfile time to @timestamp](https://discuss.elastic.co/t/logstash-mapping-logfile-time-to-timestamp/278772)
**Author:** [@Vajb12](https://discuss.elastic.co/u/Vajb12)\ **Replies:** 5\ **Last updated:** [July 16, 2021, 5:11pm UTC](https://discuss.elastic.co/t/logstash-mapping-logfile-time-to-timestamp/278772 "2021-07-16T17:11:18Z")
Hi Team, Im trying to map the logfile time with @timestamp in logstash. The logfile time is in IST. I would require some help in converting the logfile time to UTC and map it to @timestamp. I used the below ruby code… --- ## [Basic understating regarding index creation](https://discuss.elastic.co/t/basic-understating-regarding-index-creation/278189)
**Author:** [@cool999](https://discuss.elastic.co/u/cool999)\ **Replies:** 9\ **Last updated:** [July 16, 2021, 3:34pm UTC](https://discuss.elastic.co/t/basic-understating-regarding-index-creation/278189 "2021-07-16T15:34:58Z")
Hi Team, I am facing issue where I can see index has not got created after some changes in logstash pipeline file. Can you please help me with below questions to understand. I am finding difficulty in getting it. I ha… --- ## [Logstash log4j2.properties configuration for logs rotation](https://discuss.elastic.co/t/logstash-log4j2-properties-configuration-for-logs-rotation/278616)
**Author:** [@Kshema](https://discuss.elastic.co/u/Kshema)\ **Replies:** 3\ **Last updated:** [July 16, 2021, 2:37pm UTC](https://discuss.elastic.co/t/logstash-log4j2-properties-configuration-for-logs-rotation/278616 "2021-07-16T14:37:54Z")
In our application, Logstash plain log file - logstas-plain.log is getting converted to a zip file logstash-plain-\.log.gz each day. The zip files are getting piled up and causing a memory issue. Any help on config… --- ## [Aggregate filter](https://discuss.elastic.co/t/aggregate-filter/278817)
**Author:** [@sergio\_junior](https://discuss.elastic.co/u/sergio_junior)\ **Replies:** 7\ **Last updated:** [July 16, 2021, 1:27pm UTC](https://discuss.elastic.co/t/aggregate-filter/278817 "2021-07-16T13:27:18Z")
hello guys, i need help with the aggregate filter. I'm trying to merge these logs' lines into a single event. 2021-07-08 00:00:00,181 INFO \[stdout\] (default task-1876) ###HawbPesquisaInteligenteConverter | Iniciando p… --- ## [How to insert only newly added rows in csv in logstash in old index](https://discuss.elastic.co/t/how-to-insert-only-newly-added-rows-in-csv-in-logstash-in-old-index/278866)
**Author:** [@Sarthak\_Mishra](https://discuss.elastic.co/u/Sarthak_Mishra)\ **Replies:** 4\ **Last updated:** [July 16, 2021, 1:23pm UTC](https://discuss.elastic.co/t/how-to-insert-only-newly-added-rows-in-csv-in-logstash-in-old-index/278866 "2021-07-16T13:23:04Z")
Hi community, If I have a csv with 5 rows and I have shipped it in logstash. In future, the csv gets updated with 5(old) + 2(new) =7 rows. I want to ship the updated csv in the same index. Can I ship only the updated 2(… --- ## [Logstash works but put a lot of 403 errors in log](https://discuss.elastic.co/t/logstash-works-but-put-a-lot-of-403-errors-in-log/278892)
**Author:** [@SzymonZy](https://discuss.elastic.co/u/SzymonZy)\ **Replies:** 1\ **Last updated:** [July 16, 2021, 1:16pm UTC](https://discuss.elastic.co/t/logstash-works-but-put-a-lot-of-403-errors-in-log/278892 "2021-07-16T13:16:01Z")
Hello I have setup els and it seems to work beside many errors: "Got response code '403' contacting Elasticsearch at URL 'https://localhost:9200/logstash'", : exception=\>LogStash::Outputs::ElasticSearch::HttpClient::Po… --- ## [Updating role privileges is not effective](https://discuss.elastic.co/t/updating-role-privileges-is-not-effective/278799)
**Author:** [@kpe](https://discuss.elastic.co/u/kpe)\ **Replies:** 2\ **Last updated:** [July 16, 2021, 8:52am UTC](https://discuss.elastic.co/t/updating-role-privileges-is-not-effective/278799 "2021-07-16T08:52:30Z")
Hello, I've set an api\_key to manage an index but I'm getting this error : Error: \[403\] {"error":{"root\_cause":\[{"type":"security\_exception","reason":"action \[indices:data/read/search\] is unauthorized for API key id \[\*… --- ## [Query hints - make some records more important than others](https://discuss.elastic.co/t/query-hints-make-some-records-more-important-than-others/278869)
**Author:** [@prze\_gee](https://discuss.elastic.co/u/prze_gee)\ **Replies:** 0\ **Last updated:** [July 16, 2021, 8:31am UTC](https://discuss.elastic.co/t/query-hints-make-some-records-more-important-than-others/278869 "2021-07-16T08:31:39Z")
Let say I have an index of cities: cities\_ind. I'm querying this data with search\_at\_you\_type to get the best recommendation while typing. Then, I want to make a few of them more important than others. The biggest cities… --- ## [How do I configure line codec to output plain events?](https://discuss.elastic.co/t/how-do-i-configure-line-codec-to-output-plain-events/278791)
**Author:** [@albgus](https://discuss.elastic.co/u/albgus)\ **Replies:** 2\ **Last updated:** [July 16, 2021, 7:08am UTC](https://discuss.elastic.co/t/how-do-i-configure-line-codec-to-output-plain-events/278791 "2021-07-16T07:08:45Z")
Hello, I assumed that the plain and line plugins would output the event message unmodified, but this does not appear to be the case, as a bunch of "garbage" data I don't care about is prepended to the event. 2021-07-15… --- ## [How to write while loops inside Logstash Elasticsearch](https://discuss.elastic.co/t/how-to-write-while-loops-inside-logstash-elasticsearch/278805)
**Author:** [@Hieu\_Luong](https://discuss.elastic.co/u/Hieu_Luong)\ **Replies:** 1\ **Last updated:** [July 15, 2021, 3:59pm UTC](https://discuss.elastic.co/t/how-to-write-while-loops-inside-logstash-elasticsearch/278805 "2021-07-15T15:59:18Z")
Is there any way to write loops inside output of Logstash For a certain message "total:100 name:product 1" I can parse total from this message and want to send 100 times to elasticsearch My code is just pseudocode, … --- ## [Lumberjack with Filebeat pipelines not working](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304)
**Author:** [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)\ **Replies:** 9\ **Last updated:** [July 15, 2021, 10:27pm UTC](https://discuss.elastic.co/t/lumberjack-with-filebeat-pipelines-not-working/278304 "2021-07-15T22:27:26Z")
I am testing a setup which is Filebeat --\> Logstash(1) --\> logstash(2) --\> Elasticsearch with custom indices. In logstash-2 i am using below config file (simplified). For logstash to logstash communication i am using Lu… --- ## [Is it possible to detect a specific file that enters inside a folder by type with logstash?](https://discuss.elastic.co/t/is-it-possible-to-detect-a-specific-file-that-enters-inside-a-folder-by-type-with-logstash/278829)
**Author:** [@raylight](https://discuss.elastic.co/u/raylight)\ **Replies:** 1\ **Last updated:** [July 15, 2021, 7:09pm UTC](https://discuss.elastic.co/t/is-it-possible-to-detect-a-specific-file-that-enters-inside-a-folder-by-type-with-logstash/278829 "2021-07-15T19:09:53Z")
I know I can watch files entering in a folder with logstash using the following syntax: input { file { type =\> "mytype" path =\> "/home/user/watchedFolder" start\_position =\> "beginning" … --- ## [JDBC Timestamp doesn't refresh, leads to large queues](https://discuss.elastic.co/t/jdbc-timestamp-doesnt-refresh-leads-to-large-queues/278393)
**Author:** [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\ **Replies:** 5\ **Last updated:** [July 15, 2021, 3:00pm UTC](https://discuss.elastic.co/t/jdbc-timestamp-doesnt-refresh-leads-to-large-queues/278393 "2021-07-15T15:00:11Z")
Hi all, I am using the jdbc plugin with LS 7.13.2 to retrieve records from an Oracle DB. When users make changes to their info, I record the timestamp of the change in the dateChanged column of the DB. Through the jdbc … --- ## [Error on parsing gzip files "Unexpected end of ZLIB input stream"](https://discuss.elastic.co/t/error-on-parsing-gzip-files-unexpected-end-of-zlib-input-stream/278789)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 0\ **Last updated:** [July 15, 2021, 1:09pm UTC](https://discuss.elastic.co/t/error-on-parsing-gzip-files-unexpected-end-of-zlib-input-stream/278789 "2021-07-15T13:09:31Z")
I have a gzip file about ~600MB that seems to be running into a plugin error mentioned below . \[2021-07-15T13:00:29,121\]\[ERROR\]\[logstash.javapipeline \]\[main\]\[4ea2794871220248b7ea22dd607e71d79bddbc1bff243a66da02aa3b… --- ## [Issues in performance](https://discuss.elastic.co/t/issues-in-performance/278788)
**Author:** [@Arthi\_Jaiswal](https://discuss.elastic.co/u/Arthi_Jaiswal)\ **Replies:** 0\ **Last updated:** [July 15, 2021, 1:07pm UTC](https://discuss.elastic.co/t/issues-in-performance/278788 "2021-07-15T13:07:18Z")
I have a case where I am ingesting over 5 MN log lines / minute into logstash hosted on an Azure . The performance is really sluggish Any recommendations on how I can improve the performance --- ## [How to process logs from certain period of time? (I use elasticsearch input plugin)](https://discuss.elastic.co/t/how-to-process-logs-from-certain-period-of-time-i-use-elasticsearch-input-plugin/278702)
**Author:** [@John\_Smith1](https://discuss.elastic.co/u/John_Smith1)\ **Replies:** 4\ **Last updated:** [July 15, 2021, 7:28am UTC](https://discuss.elastic.co/t/how-to-process-logs-from-certain-period-of-time-i-use-elasticsearch-input-plugin/278702 "2021-07-15T07:28:32Z")
In pipeline I receive logs from server with elasticsearch input plugin and I want to process not all logs, but for example only last 2 months, or, preferably from certain date to certain date. When I receive logs there… --- ## [Logstash restarts with "ConcurrentModificationException"](https://discuss.elastic.co/t/logstash-restarts-with-concurrentmodificationexception/278738)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 0\ **Last updated:** [July 15, 2021, 5:03am UTC](https://discuss.elastic.co/t/logstash-restarts-with-concurrentmodificationexception/278738 "2021-07-15T05:03:20Z")
Hi, I have an issue with logstash restarting itself showing “ConcurrentModificationException“ error before stoping and starting again, does anyone have any idea what it might be about? --- ## [Error when restart logstash LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError](https://discuss.elastic.co/t/error-when-restart-logstash-logstash-hostunreachableerror/278687)
**Author:** [@quyennguyen](https://discuss.elastic.co/u/quyennguyen)\ **Replies:** 4\ **Last updated:** [July 15, 2021, 2:36am UTC](https://discuss.elastic.co/t/error-when-restart-logstash-logstash-hostunreachableerror/278687 "2021-07-15T02:36:45Z")
Hello I install Logstash in Ubuntu 18. After changed ip from 192.168.186.147 to 192.168.186.157 I also fixed the .yml and conf.d but it's cant restart with error. /var/log/logstash/logstash-plain.log \[2021-07-14T22:18… --- ## [Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool](https://discuss.elastic.co/t/attempted-to-send-a-bulk-request-to-elasticsearch-but-no-there-are-no-living-connections-in-the-connection-pool/277307)
**Author:** [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Replies:** 2\ **Last updated:** [July 15, 2021, 1:50am UTC](https://discuss.elastic.co/t/attempted-to-send-a-bulk-request-to-elasticsearch-but-no-there-are-no-living-connections-in-the-connection-pool/277307 "2021-07-15T01:50:43Z")
I am getting the following error in the log of logstash. $ tail /var/log/logstash/logstash-plain.log \[2021-06-29T01:17:53,584\]\[ERROR\]\[logstash.outputs.elasticsearch\]\[main\]\[9acc851acb29a066b989c90cdd9ba461ce676a2f7643e20… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=209) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=211)