# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=200 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 201 --- ## [Multiple concatenation in ruby file filter](https://discuss.elastic.co/t/multiple-concatenation-in-ruby-file-filter/282101)
**Author:** [@LuigiDelavega](https://discuss.elastic.co/u/LuigiDelavega)\ **Replies:** 1\ **Last updated:** [August 23, 2021, 3:13pm UTC](https://discuss.elastic.co/t/multiple-concatenation-in-ruby-file-filter/282101 "2021-08-23T15:13:29Z")
Hello, I would like to concatenate 5 fields which can exist or not and i would like to do it in my ruby file filter because we already have others working filters in it. my fields looks like this, they can have a value… --- ## [Cant ship big log from filebeat or logstash to elastic](https://discuss.elastic.co/t/cant-ship-big-log-from-filebeat-or-logstash-to-elastic/282152)
**Author:** [@hagaluly](https://discuss.elastic.co/u/hagaluly)\ **Replies:** 3\ **Last updated:** [August 23, 2021, 1:01pm UTC](https://discuss.elastic.co/t/cant-ship-big-log-from-filebeat-or-logstash-to-elastic/282152 "2021-08-23T13:01:15Z")
i have a 3 nodes cluster with 1 master and 2 data nodes each is set for 1TB i have increased both -Xms24g -Xmx24g to half my ram (48GB total) i than successfully upload 140mb file from Kibana to elk from the GUI after i… --- ## [Logstash aggregate filter sum incorrect](https://discuss.elastic.co/t/logstash-aggregate-filter-sum-incorrect/282199)
**Author:** [@zhyp](https://discuss.elastic.co/u/zhyp)\ **Replies:** 0\ **Last updated:** [August 23, 2021, 8:40am UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-sum-incorrect/282199 "2021-08-23T08:40:29Z")
Hi, :slightly\_smiling\_face: Here is my es data.I can query them correctly. {"count":1,"time":"2021-08-10T00:15:00.000+08:00"} {"count":2,"time":"2021-08-10T00:30:00.000+08:00"} I want to sum the count by day. Here … --- ## [Unstructured Tomcat Logs](https://discuss.elastic.co/t/unstructured-tomcat-logs/282190)
**Author:** [@dipinsugathan](https://discuss.elastic.co/u/dipinsugathan)\ **Replies:** 1\ **Last updated:** [August 23, 2021, 8:28am UTC](https://discuss.elastic.co/t/unstructured-tomcat-logs/282190 "2021-08-23T08:28:18Z")
Hi All, Need help in ingesting tomcat unstructured logs to ELK. While ingesting via filebeat -\>logstash-\>elasticsearch -\> kibana everything is coming in one message field. I was trying to split it with mutate filter pl… --- ## [Logstash GeoIP filter with words](https://discuss.elastic.co/t/logstash-geoip-filter-with-words/281638)
**Author:** [@rodri.gz](https://discuss.elastic.co/u/rodri.gz)\ **Replies:** 3\ **Last updated:** [August 23, 2021, 7:01am UTC](https://discuss.elastic.co/t/logstash-geoip-filter-with-words/281638 "2021-08-23T07:01:21Z")
hello! I have a log with countries but it does not have an IP field and I would like to make a map in kibana giving a latitude and longitude to that field. If I had an IP I would know how to do it with geoip. Can this… --- ## [How to change type of field in logstash for xml plugin](https://discuss.elastic.co/t/how-to-change-type-of-field-in-logstash-for-xml-plugin/282160)
**Author:** [@Ankit-github-26](https://discuss.elastic.co/u/Ankit-github-26)\ **Replies:** 1\ **Last updated:** [August 22, 2021, 1:00pm UTC](https://discuss.elastic.co/t/how-to-change-type-of-field-in-logstash-for-xml-plugin/282160 "2021-08-22T13:00:20Z")
I want to change some field types from text(by default) to float for aggregation purpose from xml parsed data. kindly find my logstash configuration below xml { source =\> "\[data\]\[cdr\]" target =\> "cdr" } mutate { re… --- ## [Questions On Logstash Conf File](https://discuss.elastic.co/t/questions-on-logstash-conf-file/282149)
**Author:** [@Jared9922](https://discuss.elastic.co/u/Jared9922)\ **Replies:** 1\ **Last updated:** [August 22, 2021, 1:05am UTC](https://discuss.elastic.co/t/questions-on-logstash-conf-file/282149 "2021-08-22T01:05:43Z")
Hello I was just wondering if somebody could help clear up some confusion I am having when it comes to logstash. So in my current .conf file I have the output set to elasticsearch. This is where my questions start to po… --- ## [Extract file name and add as a field](https://discuss.elastic.co/t/extract-file-name-and-add-as-a-field/282105)
**Author:** [@Sherabu](https://discuss.elastic.co/u/Sherabu)\ **Replies:** 2\ **Last updated:** [August 21, 2021, 5:30pm UTC](https://discuss.elastic.co/t/extract-file-name-and-add-as-a-field/282105 "2021-08-21T17:30:50Z")
Hi bro and sis I will be sending file from filebeat to logstash. Is there a way I can extract file name on logstash filter? I want to add as a new field My file name Path/sample.log Path/sample.20210821 Your small a… --- ## [\_grokparsefailure is occurring even after the grok pattern is success in grokdebugger](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784)
**Author:** [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)\ **Replies:** 19\ **Last updated:** [August 21, 2021, 6:56am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784 "2021-08-21T06:56:58Z")
Hi Team, I'm using logstash 6.8.3, and I'm trying to parse ES slow logs and my sample field is a ES source\_query which looks like "source\_query":{"from":0,"size":0,"post\_filter":{"bool":{"must":\[{"term":{" \*\*someId\*\* "… --- ## [Api downloading Excel file,need to process it using logstash and send data to Elasticsearch](https://discuss.elastic.co/t/api-downloading-excel-file-need-to-process-it-using-logstash-and-send-data-to-elasticsearch/282090)
**Author:** [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)\ **Replies:** 0\ **Last updated:** [August 20, 2021, 3:32pm UTC](https://discuss.elastic.co/t/api-downloading-excel-file-need-to-process-it-using-logstash-and-send-data-to-elasticsearch/282090 "2021-08-20T15:32:37Z")
Hi, I am using esign genie API which can download all folder reports as an Excel file. I need to fetch data in an excel file to elasticsearch. I checked the option for using logstash but excel needs to export to CSV. I… --- ## [Aggregate filter - Parsing Jenkins log](https://discuss.elastic.co/t/aggregate-filter-parsing-jenkins-log/281994)
**Author:** [@Uplift](https://discuss.elastic.co/u/Uplift)\ **Replies:** 4\ **Last updated:** [August 20, 2021, 12:57pm UTC](https://discuss.elastic.co/t/aggregate-filter-parsing-jenkins-log/281994 "2021-08-20T12:57:38Z")
Dear ELK community, I'm quite new to ELK stack and I'm trying to figure out what can I really achieve with Logstash. I'm trying to extract some information from Jenkins logs, and I found the aggregate filter plugin whi… --- ## [Using recursion for paging?](https://discuss.elastic.co/t/using-recursion-for-paging/282079)
**Author:** [@jasenj1](https://discuss.elastic.co/u/jasenj1)\ **Replies:** 0\ **Last updated:** [August 20, 2021, 12:55pm UTC](https://discuss.elastic.co/t/using-recursion-for-paging/282079 "2021-08-20T12:55:10Z")
Given a web service that supports a typical paging construct of specifying the record to start from and a page size, is using the "pipeline" input and output to recursively call a config file a viable option? Or will it … --- ## [Config Logstash with https to Elasticsearch cluster](https://discuss.elastic.co/t/config-logstash-with-https-to-elasticsearch-cluster/282053)
**Author:** [@kiran80511](https://discuss.elastic.co/u/kiran80511)\ **Replies:** 0\ **Last updated:** [August 20, 2021, 9:19am UTC](https://discuss.elastic.co/t/config-logstash-with-https-to-elasticsearch-cluster/282053 "2021-08-20T09:19:24Z")
Error Logs in Logstash Aug 20 14:28:50 elastic.sys logstash\[5101\]: \[2021-08-20T14:28:50,172\]\[WARN \]\[logstash.outputs.elasticsearch\]\[main\] Attempted to resurrect connection to dead ES instance, but got an error {:url=\>"h… --- ## [Logstash not creating index in kibana](https://discuss.elastic.co/t/logstash-not-creating-index-in-kibana/280780)
**Author:** [@kiran80511](https://discuss.elastic.co/u/kiran80511)\ **Replies:** 25\ **Last updated:** [August 20, 2021, 7:27am UTC](https://discuss.elastic.co/t/logstash-not-creating-index-in-kibana/280780 "2021-08-20T07:27:08Z")
This is my logstash plain log file \[2021-06-18T08:21:45,675\]\[INFO \]\[logstash.runner \] Log4j configuration path used is: /etc/logstash/log4j2.properties \[2021-06-18T08:21:45,686\]\[INFO \]\[logstash.runner … --- ## [Logstash http\_poller body calls the SQL API of elasticsearch error](https://discuss.elastic.co/t/logstash-http-poller-body-calls-the-sql-api-of-elasticsearch-error/282042)
**Author:** [@zhyp](https://discuss.elastic.co/u/zhyp)\ **Replies:** 0\ **Last updated:** [August 20, 2021, 7:14am UTC](https://discuss.elastic.co/t/logstash-http-poller-body-calls-the-sql-api-of-elasticsearch-error/282042 "2021-08-20T07:14:48Z")
Hi I used logstash http\_ poller calls the SQL API of elasticsearch. SQL API \_sql?format=txt error. The following is some configuration information. input { http\_poller { urls =\> { item =\> { method =\> post url =\> … --- ## [Convert elasticsearch query output of float fields( latitude & longtitude) to String](https://discuss.elastic.co/t/convert-elasticsearch-query-output-of-float-fields-latitude-longtitude-to-string/282001)
**Author:** [@Sagar\_kadu](https://discuss.elastic.co/u/Sagar_kadu)\ **Replies:** 3\ **Last updated:** [August 19, 2021, 8:59pm UTC](https://discuss.elastic.co/t/convert-elasticsearch-query-output-of-float-fields-latitude-longtitude-to-string/282001 "2021-08-19T20:59:18Z")
I'm using elasticsearch input plugin to fetch data from elasticsearch using scroll api and it returns JSON document which consists of \[customerAddresses\] array containing latitude & logitude fields of float type...I woul… --- ## [Remove hashtags "#" using gsub](https://discuss.elastic.co/t/remove-hashtags-using-gsub/282028)
**Author:** [@mario\_kazela](https://discuss.elastic.co/u/mario_kazela)\ **Replies:** 2\ **Last updated:** [August 20, 2021, 3:53am UTC](https://discuss.elastic.co/t/remove-hashtags-using-gsub/282028 "2021-08-20T03:53:02Z")
Hi, I have some trouble using gsub on logstash because I'm still new on logstash configuration. Just want to asking. Example I have data like "#ynwa". How to remove "#" and leave the word "ynwa" only using gsub ? mutat… --- ## [Timezone logstash date filter](https://discuss.elastic.co/t/timezone-logstash-date-filter/281791)
**Author:** [@Bigbad](https://discuss.elastic.co/u/Bigbad)\ **Replies:** 4\ **Last updated:** [August 19, 2021, 6:25pm UTC](https://discuss.elastic.co/t/timezone-logstash-date-filter/281791 "2021-08-19T18:25:50Z")
Hello, I want to have another date field in Europe/Paris timezone with logstash date filter. filter { mutate { add\_field =\>{ "my\_date" =\> "%{@timestamp}" } } date { match =\> \["my\_date", "yyyy-… --- ## [Bool From JDBC](https://discuss.elastic.co/t/bool-from-jdbc/280850)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 2\ **Last updated:** [August 19, 2021, 6:22pm UTC](https://discuss.elastic.co/t/bool-from-jdbc/280850 "2021-08-19T18:22:02Z")
Is it possible to query a SQL database and, if the results have a column value matching a given string, return a boolean? I am wanting to query a table whose results will return multiple rows. I want to check the colum… --- ## [Is there a way to deploy a custom logstash input when using Elastic Cloud?](https://discuss.elastic.co/t/is-there-a-way-to-deploy-a-custom-logstash-input-when-using-elastic-cloud/281874)
**Author:** [@Mrc0113](https://discuss.elastic.co/u/Mrc0113)\ **Replies:** 2\ **Last updated:** [August 19, 2021, 5:17pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-deploy-a-custom-logstash-input-when-using-elastic-cloud/281874 "2021-08-19T17:17:52Z")
Is there a way to deploy a custom logstash input when using Elastic Cloud? I would like to be able to ingest messages into Elasticsearch from a Solace Event Broker. Options would be via MQTT, JMS or a custom Solace (SMF)… --- ## [Grok Pattern](https://discuss.elastic.co/t/grok-pattern/281878)
**Author:** [@leemase004](https://discuss.elastic.co/u/leemase004)\ **Replies:** 4\ **Last updated:** [August 19, 2021, 2:54pm UTC](https://discuss.elastic.co/t/grok-pattern/281878 "2021-08-19T14:54:53Z")
What is the correct syntax to incorporate raw regex expressions in between grok patterns? Ex (Cisco device): LOG: Aug 1 22:15:10 abc-hostname tcp GROK pattern: %{CISCOTIMESTAMP:timestamp} \\b\\w+-\\w+\\b %{WORD:protocol} … --- ## [Logstash elapsed filter plugin: can't connect start and end events](https://discuss.elastic.co/t/logstash-elapsed-filter-plugin-cant-connect-start-and-end-events/281927)
**Author:** [@alexandrpaliy](https://discuss.elastic.co/u/alexandrpaliy)\ **Replies:** 2\ **Last updated:** [August 19, 2021, 1:26pm UTC](https://discuss.elastic.co/t/logstash-elapsed-filter-plugin-cant-connect-start-and-end-events/281927 "2021-08-19T13:26:56Z")
Hi. Currently, my logstash filter looks something like this: filter { if ('text1' in \[message\]) { grok { ... add\_tag =\> \[ "connection\_established" \] } aggregate { ... } elapsed… --- ## [Remove fields with specific prefix at the end of LogStash pipeline](https://discuss.elastic.co/t/remove-fields-with-specific-prefix-at-the-end-of-logstash-pipeline/281911)
**Author:** [@itokai](https://discuss.elastic.co/u/itokai)\ **Replies:** 1\ **Last updated:** [August 19, 2021, 1:23pm UTC](https://discuss.elastic.co/t/remove-fields-with-specific-prefix-at-the-end-of-logstash-pipeline/281911 "2021-08-19T13:23:49Z")
Hello, how to remove children fields with specific prefix at the end of LogStash pipeline? parent.level2.level3.removewiththisprefix\_1, parent.level2.level3.removewiththisprefix\_2,... parent.level2.level3.removewiththi… --- ## [Logstash systemd service restarts itself](https://discuss.elastic.co/t/logstash-systemd-service-restarts-itself/279941)
**Author:** [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\ **Replies:** 2\ **Last updated:** [August 19, 2021, 8:53am UTC](https://discuss.elastic.co/t/logstash-systemd-service-restarts-itself/279941 "2021-08-19T08:53:15Z")
Hello community, I'm running a logstash instance with the following config via a systemd service for minute-wise updates of an ES index: input { jdbc { jdbc\_driver\_class =\> "Java::oracle.jdbc.dr… --- ## [Starting separate logstash pipeline incorrectly starts Logstash Service](https://discuss.elastic.co/t/starting-separate-logstash-pipeline-incorrectly-starts-logstash-service/281933)
**Author:** [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\ **Replies:** 0\ **Last updated:** [August 19, 2021, 9:19am UTC](https://discuss.elastic.co/t/starting-separate-logstash-pipeline-incorrectly-starts-logstash-service/281933 "2021-08-19T09:19:18Z")
Hi all, I've noticed recently that my Logstash service sometimes gets started for an unknown reason when I start a separate Logstash pipeline. The expected behavior is that this single Logstash instance is started and … --- ## [Webhdfs output writes not all fields](https://discuss.elastic.co/t/webhdfs-output-writes-not-all-fields/281919)
**Author:** [@ThreatInter](https://discuss.elastic.co/u/ThreatInter)\ **Replies:** 1\ **Last updated:** [August 19, 2021, 7:54am UTC](https://discuss.elastic.co/t/webhdfs-output-writes-not-all-fields/281919 "2021-08-19T07:54:50Z")
Hi, community, I'm using webhdfs to write my logstash events to hdfs. But in hdfs I see only part of initial messages. In logstash: {"@timestamp":"2021-08-19T07:45:33.373Z","eventtype":"166","program":"Hostd","@version… --- ## [Logstash OData endpoint](https://discuss.elastic.co/t/logstash-odata-endpoint/281910)
**Author:** [@Jakub\_Kaczmarek](https://discuss.elastic.co/u/Jakub_Kaczmarek)\ **Replies:** 1\ **Last updated:** [August 19, 2021, 6:57am UTC](https://discuss.elastic.co/t/logstash-odata-endpoint/281910 "2021-08-19T06:57:08Z")
Hi, I'm using http\_pooler in order to fetch data from OData endpoint. url =\> "https://odata-XXXXX.com/v2.0/Inputs?apikey=apikey" -\> works correct url =\> "https://odata-XXXXX.com/v2.0/Inputs?$orderby=timestamp desc&api… --- ## [Logstash Crashback](https://discuss.elastic.co/t/logstash-crashback/281909)
**Author:** [@Vincent\_Ngai](https://discuss.elastic.co/u/Vincent_Ngai)\ **Replies:** 0\ **Last updated:** [August 19, 2021, 6:42am UTC](https://discuss.elastic.co/t/logstash-crashback/281909 "2021-08-19T06:42:27Z")
Here is background My logstash version : logstash\_oss 681 My logstash is a dockerversion (install into k8s) and my logstash docker file like this FROM docker.elastic.co/logstash/logstash-oss:6.8.1 RUN logstash-plugin … --- ## [Dynamic index names](https://discuss.elastic.co/t/dynamic-index-names/281816)
**Author:** [@rudo1](https://discuss.elastic.co/u/rudo1)\ **Replies:** 5\ **Last updated:** [August 19, 2021, 4:32am UTC](https://discuss.elastic.co/t/dynamic-index-names/281816 "2021-08-19T04:32:20Z")
Hi. I'm facing a problem with creating dynamic index names. I have few if statemnts like shown below to add new fields, but I can't see anything in kibana when I try to name index dynamically based on its value. filte… --- ## [HTTP Input/Output](https://discuss.elastic.co/t/http-input-output/281593)
**Author:** [@tofubeats](https://discuss.elastic.co/u/tofubeats)\ **Replies:** 6\ **Last updated:** [August 18, 2021, 10:20pm UTC](https://discuss.elastic.co/t/http-input-output/281593 "2021-08-18T22:20:26Z")
Hello, I am working on connecting two instances of logstash by an HTTP plugin. On node 1, the config is set to ingest from a CSV and outputs those docs to a local HTTP url. As in, the URL is set to a port on node 1. O… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=199) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=201)