# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=315 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 316 --- ## [How does JDBC input plugin keeps timestamp last value?](https://discuss.elastic.co/t/how-does-jdbc-input-plugin-keeps-timestamp-last-value/245246)
**Author:** [@ffknob](https://discuss.elastic.co/u/ffknob)\ **Replies:** 2\ **Last updated:** [August 17, 2020, 2:52pm UTC](https://discuss.elastic.co/t/how-does-jdbc-input-plugin-keeps-timestamp-last-value/245246 "2020-08-17T14:52:04Z")
Hello I just want to figure it out how does JDBC input plugin writes the state value to the last\_run\_metadata\_path file. Let me explain: We have a SQL Server database and we are ingesting a table that now has a "last… --- ## [SEQUENTIAL FIELD WHEN MATCH ID DOC](https://discuss.elastic.co/t/sequential-field-when-match-id-doc/245196)
**Author:** [@Daniel\_Lopez](https://discuss.elastic.co/u/Daniel_Lopez)\ **Replies:** 1\ **Last updated:** [August 17, 2020, 2:27pm UTC](https://discuss.elastic.co/t/sequential-field-when-match-id-doc/245196 "2020-08-17T14:27:36Z")
Hi I will try to explain what I'm in I have these lines in a log ID:1,PID:123, Path:\\123\\123 ID:2,PID:234, Path:\\234\\234 ID:1,PID:345, Path:\\345\\345 The result of indexed doc I need to get \_Id: 1 PID0:123 Path0:… --- ## [Using aggregate filter to count number of events and sum field's values](https://discuss.elastic.co/t/using-aggregate-filter-to-count-number-of-events-and-sum-fields-values/244448)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 28\ **Last updated:** [August 17, 2020, 2:24pm UTC](https://discuss.elastic.co/t/using-aggregate-filter-to-count-number-of-events-and-sum-fields-values/244448 "2020-08-17T14:24:06Z")
Hi all, I have an index in elasticsearch name as "myindex" so that have fields like"date\_eng","company\_name","service\_name", "amount". I want to use aggregate filter so that counts the times which an services has been u… --- ## [Best approach to 50+ input sources](https://discuss.elastic.co/t/best-approach-to-50-input-sources/245249)
**Author:** [@ffknob](https://discuss.elastic.co/u/ffknob)\ **Replies:** 0\ **Last updated:** [August 17, 2020, 2:20pm UTC](https://discuss.elastic.co/t/best-approach-to-50-input-sources/245249 "2020-08-17T14:20:42Z")
Hello I'd appreciate your thoughts on how to approcah the following use case: I have to configure a Logstash pipeline to index a table in SQL Server The thing is, although the table structure is the same for every cli… --- ## [Connecting to the SAP Hana Database](https://discuss.elastic.co/t/connecting-to-the-sap-hana-database/245210)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 1\ **Last updated:** [August 17, 2020, 2:07pm UTC](https://discuss.elastic.co/t/connecting-to-the-sap-hana-database/245210 "2020-08-17T14:07:06Z")
Is it possible to connect to the SAP Hana Database using the logstash jdbc input plugin? Whether it is officially supported database just like MySQL and Oracle? --- ## [How to control Logstash multiple output rate](https://discuss.elastic.co/t/how-to-control-logstash-multiple-output-rate/245224)
**Author:** [@vaishno.avi](https://discuss.elastic.co/u/vaishno.avi)\ **Replies:** 0\ **Last updated:** [August 17, 2020, 11:15am UTC](https://discuss.elastic.co/t/how-to-control-logstash-multiple-output-rate/245224 "2020-08-17T11:15:10Z")
Hello, I am using logstash with multiple output, with the below config if \[channel\] == "test" { elasticsearch { hosts =\> \["elk\_url"\] index =\> "logs-%{+YYYY.MM.dd}" } … --- ## [DLQ - dead\_letter\_queue input plugin features and ability](https://discuss.elastic.co/t/dlq-dead-letter-queue-input-plugin-features-and-ability/245219)
**Author:** [@Apollyon](https://discuss.elastic.co/u/Apollyon)\ **Replies:** 0\ **Last updated:** [August 17, 2020, 10:27am UTC](https://discuss.elastic.co/t/dlq-dead-letter-queue-input-plugin-features-and-ability/245219 "2020-08-17T10:27:12Z")
Hello all Running Logstash v7.8.1 and dead\_letter\_queue input plugin v1.1.5. Just a question about the functionalilty of the DLQ input plugin for Logstash. Seeing a couple of issues myself and then after looking aroun… --- ## [How to split field value into an array?](https://discuss.elastic.co/t/how-to-split-field-value-into-an-array/245197)
**Author:** [@Robo](https://discuss.elastic.co/u/Robo)\ **Replies:** 1\ **Last updated:** [August 17, 2020, 9:57am UTC](https://discuss.elastic.co/t/how-to-split-field-value-into-an-array/245197 "2020-08-17T09:57:15Z")
Hello, I would like to split a field value into an array of values. Field type is keyword and I would like to separate these values, so we can filter each value separately (e.g. "app1"). example: applname: app1, app2,… --- ## [Error parsing xml with XmlSimple: Bad encoding name](https://discuss.elastic.co/t/error-parsing-xml-with-xmlsimple-bad-encoding-name/245147)
**Author:** [@insaneboiz](https://discuss.elastic.co/u/insaneboiz)\ **Replies:** 2\ **Last updated:** [August 17, 2020, 3:58am UTC](https://discuss.elastic.co/t/error-parsing-xml-with-xmlsimple-bad-encoding-name/245147 "2020-08-17T03:58:56Z")
Hi guys, I've been having this trouble with Logstash v7.8.1 and v6.5.4 with EPO Syslog. It's a straight simple XML data, but the XML plugin could not parse it somehow. Sending Logstash logs to /home/XXX/Desktop/logstash… --- ## [Logstash](https://discuss.elastic.co/t/logstash/245152)
**Author:** [@shashanksingh](https://discuss.elastic.co/u/shashanksingh)\ **Replies:** 1\ **Last updated:** [August 16, 2020, 6:54pm UTC](https://discuss.elastic.co/t/logstash/245152 "2020-08-16T18:54:18Z")
hi , i am new to elastic stack . i am configuring filebeat logstash elasticsearch kibana. and i am getting some error. please help me as soon as possible. thank you. root@ip-10-0-1-187:~# sudo /usr/share/logstash/bin/lo… --- ## [Logstash S3 Input Plugin Error](https://discuss.elastic.co/t/logstash-s3-input-plugin-error/245104)
**Author:** [@Kevin\_f](https://discuss.elastic.co/u/Kevin_f)\ **Replies:** 14\ **Last updated:** [August 16, 2020, 3:19pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-error/245104 "2020-08-16T15:19:56Z")
Hi All, I am using S3 plugin to ingest logs from Oracle Cloud Infrastructure bucket. They have advised that the S3 plugin is what can be used. So I have now configured the input config as: input { s3 { bucket =\> … --- ## [How to transfer log files from log collector(client side) to gateway collector(server side) using logstash](https://discuss.elastic.co/t/how-to-transfer-log-files-from-log-collector-client-side-to-gateway-collector-server-side-using-logstash/245131)
**Author:** [@Gunda\_Someswararao](https://discuss.elastic.co/u/Gunda_Someswararao)\ **Replies:** 0\ **Last updated:** [August 16, 2020, 10:22am UTC](https://discuss.elastic.co/t/how-to-transfer-log-files-from-log-collector-client-side-to-gateway-collector-server-side-using-logstash/245131 "2020-08-16T10:22:07Z")
How to transfer log files from log collector(client side) to gateway collector(server side) using logstash. I am new in ES. please help me, and give some reference links for learning. Thank you. --- ## [Why is this logstah filter not working as intended?](https://discuss.elastic.co/t/why-is-this-logstah-filter-not-working-as-intended/244874)
**Author:** [@drjosephliu](https://discuss.elastic.co/u/drjosephliu)\ **Replies:** 1\ **Last updated:** [August 16, 2020, 8:06am UTC](https://discuss.elastic.co/t/why-is-this-logstah-filter-not-working-as-intended/244874 "2020-08-16T08:06:25Z")
Sorry if this is too related to my previous post, but here goes: I have two types of messages that I want to send to logstash. One is a "standard" log type message and the other is a JSON object. I only want the message… --- ## [Grok parse failure](https://discuss.elastic.co/t/grok-parse-failure/245124)
**Author:** [@jclemons7](https://discuss.elastic.co/u/jclemons7)\ **Replies:** 1\ **Last updated:** [August 16, 2020, 8:00am UTC](https://discuss.elastic.co/t/grok-parse-failure/245124 "2020-08-16T08:00:59Z")
Hello, I have the following log from snort (from PFSense) roku \[classification: misc activity\] \[priority: 3\] {tcp} 10.0.2.11:37183 -\> 151.101.65.190:80 I've created the following GROK pattern, which works fine in the … --- ## [Logstash is not starting](https://discuss.elastic.co/t/logstash-is-not-starting/245082)
**Author:** [@H158880](https://discuss.elastic.co/u/H158880)\ **Replies:** 8\ **Last updated:** [August 16, 2020, 4:00am UTC](https://discuss.elastic.co/t/logstash-is-not-starting/245082 "2020-08-16T04:00:07Z")
I am trying to initiate logstash how ever getting below error. Can this forum help? C:\\Users\\h158880\\Documents\\lib\\logstash-7.8.1\\bin\>logstash -f apache\_logsatsh.conf Java HotSpot(TM) 64-Bit Server VM warning: Ignoring… --- ## [Developing a Logstash plugin within Docker](https://discuss.elastic.co/t/developing-a-logstash-plugin-within-docker/245093)
**Author:** [@cknz](https://discuss.elastic.co/u/cknz)\ **Replies:** 0\ **Last updated:** [August 15, 2020, 10:32am UTC](https://discuss.elastic.co/t/developing-a-logstash-plugin-within-docker/245093 "2020-08-15T10:32:44Z")
Hi, I'm more of a Python and C developer than a Ruby a Java developer, but I have a need to make my first real logstash plugin for some work I'm doing with Elastic. My development environment is Windows 10 Enterprise wi… --- ## [Logstash monitoring warning](https://discuss.elastic.co/t/logstash-monitoring-warning/244835)
**Author:** [@gsk](https://discuss.elastic.co/u/gsk)\ **Replies:** 1\ **Last updated:** [August 15, 2020, 9:04am UTC](https://discuss.elastic.co/t/logstash-monitoring-warning/244835 "2020-08-15T09:04:06Z")
Hi all, I have a cluster of three master-eligible nodes, which are also data nodes, and one coordinating node which also runs a kibana instance. There's also a pipeline node. It's about 7.8.0 version. All the cluster c… --- ## [Filebeat to logstash reading all file from beginning](https://discuss.elastic.co/t/filebeat-to-logstash-reading-all-file-from-beginning/244620)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 17\ **Last updated:** [August 14, 2020, 8:16pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-reading-all-file-from-beginning/244620 "2020-08-14T20:16:09Z")
I have a filebeat running and properly configure I have logstash running which has input - filter- output. and all works good when I startup problem is when I restart the logstash it reads all the file located in that… --- ## [Successfully started Logstash API endpoint {:port=\>9600} stuck here i am using window 10](https://discuss.elastic.co/t/successfully-started-logstash-api-endpoint-port-9600-stuck-here-i-am-using-window-10/245039)
**Author:** [@SUSHIL\_KUMAR1](https://discuss.elastic.co/u/SUSHIL_KUMAR1)\ **Replies:** 1\ **Last updated:** [August 14, 2020, 4:15pm UTC](https://discuss.elastic.co/t/successfully-started-logstash-api-endpoint-port-9600-stuck-here-i-am-using-window-10/245039 "2020-08-14T16:15:18Z")
input{ file{ path=\> "C:/Users/sushils/Desktop/metadata.csv" start\_position=\>"beginning" sincedb\_path=\> "NULL" } } filter { csv{ separator =\> "," columns =\> \["cord\_uid", "sha", "source\_x", "title", "doi", "pmci… --- ## [JMX Plugin Attributes (more than metric path)](https://discuss.elastic.co/t/jmx-plugin-attributes-more-than-metric-path/245017)
**Author:** [@Rodrigo\_Maia](https://discuss.elastic.co/u/Rodrigo_Maia)\ **Replies:** 0\ **Last updated:** [August 14, 2020, 1:27pm UTC](https://discuss.elastic.co/t/jmx-plugin-attributes-more-than-metric-path/245017 "2020-08-14T13:27:57Z")
Let's say I have the JMX config for my ActiveMQ: { "object\_name" : "org.apache.activemq:type=Broker,brokerName=\*,destinationName=\*,destinationType=Topic", "attributes" : \[ "AverageBlockedTime","AverageEn… --- ## [Ignore groups that are not in the log](https://discuss.elastic.co/t/ignore-groups-that-are-not-in-the-log/244754)
**Author:** [@idyuzhev](https://discuss.elastic.co/u/idyuzhev)\ **Replies:** 7\ **Last updated:** [August 14, 2020, 2:56pm UTC](https://discuss.elastic.co/t/ignore-groups-that-are-not-in-the-log/244754 "2020-08-14T14:56:32Z")
Hello. I hope for your help. I need to parse the nginx-error-log, but the problem is that this or that group in the log may be absent and then no match occurs. Here's an example. 2020/08/11 14:16:35 \[warn\] 230521#230… --- ## [How drop messages with total fields bigger than 1000](https://discuss.elastic.co/t/how-drop-messages-with-total-fields-bigger-than-1000/245012)
**Author:** [@dorinand](https://discuss.elastic.co/u/dorinand)\ **Replies:** 0\ **Last updated:** [August 14, 2020, 1:08pm UTC](https://discuss.elastic.co/t/how-drop-messages-with-total-fields-bigger-than-1000/245012 "2020-08-14T13:08:40Z")
I am receiving a lot of Limit of total fields \[1000\] errors. I increase the size to 2000 but the errors still occurs and dont want to increase the limit to bigger size . I was not able to find a filter or other way, how … --- ## [Logstash pipelines not working together, but separately yes](https://discuss.elastic.co/t/logstash-pipelines-not-working-together-but-separately-yes/244973)
**Author:** [@Andrea\_Bozzano](https://discuss.elastic.co/u/Andrea_Bozzano)\ **Replies:** 5\ **Last updated:** [August 14, 2020, 12:46pm UTC](https://discuss.elastic.co/t/logstash-pipelines-not-working-together-but-separately-yes/244973 "2020-08-14T12:46:00Z")
Hi everyone, and sorry to bother you but i can't find a proper solution to this. I'm trying to setup a pipelines.yml file to ingest data with Logstash through 3 pipelines. I'm running Logstash on a single node ELK stac… --- ## [Logstash add field metadata not working](https://discuss.elastic.co/t/logstash-add-field-metadata-not-working/244911)
**Author:** [@frankfoti](https://discuss.elastic.co/u/frankfoti)\ **Replies:** 4\ **Last updated:** [August 14, 2020, 12:40pm UTC](https://discuss.elastic.co/t/logstash-add-field-metadata-not-working/244911 "2020-08-14T12:40:42Z")
I have tried a number of different solutions and cannot get this to work. What am I missing? https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html filter { mutate { id =\> "add-pipeline" … --- ## [Help Logstash XML Parsing to Xpath](https://discuss.elastic.co/t/help-logstash-xml-parsing-to-xpath/241051)
**Author:** [@akim](https://discuss.elastic.co/u/akim)\ **Replies:** 22\ **Last updated:** [August 14, 2020, 10:06am UTC](https://discuss.elastic.co/t/help-logstash-xml-parsing-to-xpath/241051 "2020-08-14T10:06:52Z")
Hello! I need help how to parse this kind of XML File: \ \ \ \ **Author:** [@Rebin](https://discuss.elastic.co/u/Rebin)\ **Replies:** 4\ **Last updated:** [August 14, 2020, 9:13am UTC](https://discuss.elastic.co/t/passing-current-time-in-logstash-input-plugin-exec-for-executing-api-in-curl/243933 "2020-08-14T09:13:55Z") I want to execute API in Logstash & schedule it every 30 mins and store the data in elastic search. I'm using exec logstash plugin to do, In the curl command there is start and end query param where i need to specify epo… --- ## [Grok filter pattern for nginx to parse request body](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx-to-parse-request-body/244898)
**Author:** [@Santu](https://discuss.elastic.co/u/Santu)\ **Replies:** 4\ **Last updated:** [August 14, 2020, 7:24am UTC](https://discuss.elastic.co/t/grok-filter-pattern-for-nginx-to-parse-request-body/244898 "2020-08-14T07:24:09Z")
Hi, i'm trying to parse the request body with logstash grok pattern for nginx generated logs. below is my grok pattern- "%{WORD:req\_body}" and i get the parsed field like below- "req\_body" : "req\_body", "@version"… --- ## [Convert all the fields to integer in csv filter](https://discuss.elastic.co/t/convert-all-the-fields-to-integer-in-csv-filter/244949)
**Author:** [@saipadu](https://discuss.elastic.co/u/saipadu)\ **Replies:** 2\ **Last updated:** [August 14, 2020, 7:02am UTC](https://discuss.elastic.co/t/convert-all-the-fields-to-integer-in-csv-filter/244949 "2020-08-14T07:02:01Z")
Hi, I'm parsing the csv output of a command to logstash which has a dynamic number of columns. All the column values are interger except one which is a time field. By default logstash takes them all as strings. How can … --- ## [Why not recognize simple syslog text as SYSLOGLINE in Grok Debugger?](https://discuss.elastic.co/t/why-not-recognize-simple-syslog-text-as-syslogline-in-grok-debugger/244905)
**Author:** [@patsevanton](https://discuss.elastic.co/u/patsevanton)\ **Replies:** 2\ **Last updated:** [August 14, 2020, 3:30am UTC](https://discuss.elastic.co/t/why-not-recognize-simple-syslog-text-as-syslogline-in-grok-debugger/244905 "2020-08-14T03:30:12Z")
Hello! Source log: Mar 16 00:01:25 evita postfix/smtpd\[1713\]: connect from camomile.cloud9.net\[168.100.1.3\] Grok pattern - SYSLOGLINE Output - No Matches Screenshot: --- ## [Where can we apply business logic in ELK? For instance, I don't want logs just the way they are rather some more metadata to each row. (assuming logs are in csv)](https://discuss.elastic.co/t/where-can-we-apply-business-logic-in-elk-for-instance-i-dont-want-logs-just-the-way-they-are-rather-some-more-metadata-to-each-row-assuming-logs-are-in-csv/243765)
**Author:** [@AditiKhalatkar](https://discuss.elastic.co/u/AditiKhalatkar)\ **Replies:** 4\ **Last updated:** [August 14, 2020, 1:26am UTC](https://discuss.elastic.co/t/where-can-we-apply-business-logic-in-elk-for-instance-i-dont-want-logs-just-the-way-they-are-rather-some-more-metadata-to-each-row-assuming-logs-are-in-csv/243765 "2020-08-14T01:26:44Z")
Elastic search is mostly for querying data. However, I am stuck with the part where I want to ingest some logs but probably with business logic. So that the data is transformed by the time it reaches Elastic search. Can … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=314) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=316)