# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=440 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 441 --- ## [How to remove the root element while xml parsing using Logstash?](https://discuss.elastic.co/t/how-to-remove-the-root-element-while-xml-parsing-using-logstash/209412)
**Author:** [@Reshmi](https://discuss.elastic.co/u/Reshmi)\ **Replies:** 2\ **Last updated:** [November 27, 2019, 6:13am UTC](https://discuss.elastic.co/t/how-to-remove-the-root-element-while-xml-parsing-using-logstash/209412 "2019-11-27T06:13:10Z")
My sample XML structure is as below: \ \\ \ \\ \ \ With the below code: … --- ## [Fetching Data from couch db(nosql db) to elasticsearch through logstash?](https://discuss.elastic.co/t/fetching-data-from-couch-db-nosql-db-to-elasticsearch-through-logstash/209600)
**Author:** [@Sagar\_Mandal](https://discuss.elastic.co/u/Sagar_Mandal)\ **Replies:** 1\ **Last updated:** [November 27, 2019, 6:11am UTC](https://discuss.elastic.co/t/fetching-data-from-couch-db-nosql-db-to-elasticsearch-through-logstash/209600 "2019-11-27T06:11:06Z")
Hi, I want to fetch data from couch db which is a nosql db to elasticsearch through logstash. but i don't know what the logstash config file will look like, I mean what Input section will look like in the config file. … --- ## [Matching field to a single digit number](https://discuss.elastic.co/t/matching-field-to-a-single-digit-number/209405)
**Author:** [@sreliable](https://discuss.elastic.co/u/sreliable)\ **Replies:** 2\ **Last updated:** [November 27, 2019, 6:06am UTC](https://discuss.elastic.co/t/matching-field-to-a-single-digit-number/209405 "2019-11-27T06:06:12Z")
I have a field which contains strings like "0","1"..."10","11"..."23". How do I get an if statement where I take only fields which match single digits? ie. "0","1"...."9"? Currently what I have is this: if \[tmHour\] =~… --- ## [Send Only updates to Kafka topic](https://discuss.elastic.co/t/send-only-updates-to-kafka-topic/209524)
**Author:** [@giridhar\_varma](https://discuss.elastic.co/u/giridhar_varma)\ **Replies:** 1\ **Last updated:** [November 27, 2019, 6:04am UTC](https://discuss.elastic.co/t/send-only-updates-to-kafka-topic/209524 "2019-11-27T06:04:48Z")
Hi Team, we need to send the updates received into ElasticSearch Index(example: testcaseindex) to Kafka topic The Solution i am looking for is:- To only pick up UPDATES not the whole of index data everytime we schedul… --- ## [Logstash pipeline stops working](https://discuss.elastic.co/t/logstash-pipeline-stops-working/209604)
**Author:** [@b725](https://discuss.elastic.co/u/b725)\ **Replies:** 3\ **Last updated:** [November 27, 2019, 5:54am UTC](https://discuss.elastic.co/t/logstash-pipeline-stops-working/209604 "2019-11-27T05:54:33Z")
I noticed today that a pipeline stopped working Background events get logged into different Amazon S3 buckets paths a logstash pipeline is created for different bucket paths so that they run in parallel plus different… --- ## [Can I get logstash's instance-id or hostname / node name in processing?](https://discuss.elastic.co/t/can-i-get-logstashs-instance-id-or-hostname-node-name-in-processing/209030)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 1\ **Last updated:** [November 27, 2019, 5:11am UTC](https://discuss.elastic.co/t/can-i-get-logstashs-instance-id-or-hostname-node-name-in-processing/209030 "2019-11-27T05:11:05Z")
Hi, I am runnnig logstash in kubernetes environment with identical configuration with multiple replicas as deployment. I need to debug the lost of some message parts, so I want process the message normally (parse and s… --- ## [Converting string to ip datatype](https://discuss.elastic.co/t/converting-string-to-ip-datatype/207890)
**Author:** [@bibin](https://discuss.elastic.co/u/bibin)\ **Replies:** 6\ **Last updated:** [November 27, 2019, 4:27am UTC](https://discuss.elastic.co/t/converting-string-to-ip-datatype/207890 "2019-11-27T04:27:48Z")
Hi team ES... Can anyone help me how to convert a string to ip datatype ,I have already indexed huge data... As I am new to ES ,i couldn't understand solutions to similar situation sample data // current format str… --- ## [Add Field In Alter Plugin Not Showing values inside field instead showing field name](https://discuss.elastic.co/t/add-field-in-alter-plugin-not-showing-values-inside-field-instead-showing-field-name/209492)
**Author:** [@Ramya\_Tanikanti](https://discuss.elastic.co/u/Ramya_Tanikanti)\ **Replies:** 14\ **Last updated:** [November 27, 2019, 4:20am UTC](https://discuss.elastic.co/t/add-field-in-alter-plugin-not-showing-values-inside-field-instead-showing-field-name/209492 "2019-11-27T04:20:33Z")
Hi have created the following code using Alter plugin because the mutate plugin won't work in my logstash server. filter { date{match =\> \["REQUEST\_TIMESTAMP","yyyy-MM-dd HH:mm:ss"\]} alter { add\_field =\> { "\[loc… --- ## [Need help with adding geoip to syslog and then how to visualize... -](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113)
**Author:** [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\ **Replies:** 12\ **Last updated:** [November 27, 2019, 1:43am UTC](https://discuss.elastic.co/t/need-help-with-adding-geoip-to-syslog-and-then-how-to-visualize/209113 "2019-11-27T01:43:47Z")
Hi, i'm receiving network device syslog via filebeats - works fine, using default syslog filter grok { match =\> { "message" =\> \["%{SYSLOGTIMESTAMP:\[system\]\[syslog\]\[timestamp\]} %{SYSLOGHOST:\[system\]\[syslog\]\[hostname\]} … --- ## [How to handle large message in log](https://discuss.elastic.co/t/how-to-handle-large-message-in-log/209542)
**Author:** [@Kwanchai\_Jaroensiric](https://discuss.elastic.co/u/Kwanchai_Jaroensiric)\ **Replies:** 0\ **Last updated:** [November 26, 2019, 3:38pm UTC](https://discuss.elastic.co/t/how-to-handle-large-message-in-log/209542 "2019-11-26T15:38:57Z")
i found large message in some log and i count word of log result : 2.5k++ and i using this code.. My sample code \ filter { if "service\_daily" in \[tags\] { mutate { gsub =\> \[ "message", "TIMESTAMP|", "", "mes… --- ## [Filter date in logstash](https://discuss.elastic.co/t/filter-date-in-logstash/209582)
**Author:** [@Sirius](https://discuss.elastic.co/u/Sirius)\ **Replies:** 4\ **Last updated:** [November 26, 2019, 10:35pm UTC](https://discuss.elastic.co/t/filter-date-in-logstash/209582 "2019-11-26T22:35:17Z")
I am traying to filter a format date like below message log with grok, but not luck 01-NOV-2019 00:09:01 \* (ADDRESS=(PROTOCOL=tcp)(HOST=10.10.84.62) (PORT=48482)) \* \ \* 12537 From debugger works b… --- ## [How to check if logstash is sending the logs to Elasticsearch](https://discuss.elastic.co/t/how-to-check-if-logstash-is-sending-the-logs-to-elasticsearch/208444)
**Author:** [@Magesh](https://discuss.elastic.co/u/Magesh)\ **Replies:** 1\ **Last updated:** [November 26, 2019, 8:44pm UTC](https://discuss.elastic.co/t/how-to-check-if-logstash-is-sending-the-logs-to-elasticsearch/208444 "2019-11-26T20:44:12Z")
Hi, could you please let us know How to check if logstash is sending the logs to Elasticsearch. logstash,elasticsearch,kibana has been started without any issues.however i'm unable to view the logs/indexes in Kibana. P… --- ## [Logstash monitoring vs management](https://discuss.elastic.co/t/logstash-monitoring-vs-management/209387)
**Author:** [@ibexit](https://discuss.elastic.co/u/ibexit)\ **Replies:** 2\ **Last updated:** [November 26, 2019, 6:41pm UTC](https://discuss.elastic.co/t/logstash-monitoring-vs-management/209387 "2019-11-26T18:41:18Z")
using docker-compose I´m able to start and monitor my logstash instance with this environment settings: - XPACK\_SECURITY\_ENABLED=true - XPACK.MONITORING.ENABLED=true - XPACK.MONITORING.ELASTICSEARCH.HOSTS=\["https://192.… --- ## [How to sum together values in different objects](https://discuss.elastic.co/t/how-to-sum-together-values-in-different-objects/209554)
**Author:** [@twright8](https://discuss.elastic.co/u/twright8)\ **Replies:** 2\ **Last updated:** [November 26, 2019, 7:24pm UTC](https://discuss.elastic.co/t/how-to-sum-together-values-in-different-objects/209554 "2019-11-26T19:24:29Z")
Hi, I have nested objects in an array. The objects are the same structure. I want to add together all \[array\]\[amount\] values (which are all numbers) where x can be any number up to around 40, to create a new field \[amoun… --- ## [Ignore a string from logstash from grok pattern](https://discuss.elastic.co/t/ignore-a-string-from-logstash-from-grok-pattern/209561)
**Author:** [@whoatemyjam](https://discuss.elastic.co/u/whoatemyjam)\ **Replies:** 4\ **Last updated:** [November 26, 2019, 6:55pm UTC](https://discuss.elastic.co/t/ignore-a-string-from-logstash-from-grok-pattern/209561 "2019-11-26T18:55:22Z")
hi from a log file like this 2019-11-22 13:02:01 INFO sometext:999 \[applicationId=xxxxx\] \[version=xxxx\] \[hostname=hostname.domain\] \[logmessage=all went well\] can we remove the "sometext" and just show value of 999… --- ## [Mutate - Split - datatype remains string](https://discuss.elastic.co/t/mutate-split-datatype-remains-string/207995)
**Author:** [@varshesh](https://discuss.elastic.co/u/varshesh)\ **Replies:** 3\ **Last updated:** [November 26, 2019, 4:52pm UTC](https://discuss.elastic.co/t/mutate-split-datatype-remains-string/207995 "2019-11-26T16:52:11Z")
Hi, I have a field with comma separated city names. I have applied simple split as below. filter { mutate { split =\> { "city" =\> ',' } } } But, in index mapping, the data type displayed is string. Can someone guide … --- ## [Java.lang.illegalstateexception](https://discuss.elastic.co/t/java-lang-illegalstateexception/209467)
**Author:** [@Sagar\_Mandal](https://discuss.elastic.co/u/Sagar_Mandal)\ **Replies:** 1\ **Last updated:** [November 26, 2019, 4:15pm UTC](https://discuss.elastic.co/t/java-lang-illegalstateexception/209467 "2019-11-26T16:15:19Z")
Hi i'm new to ELK Stack, I'm currently trying to ingest data from filebeat to logstash and using grok filter in it. i accidentally deleted the whole log folder of logstash /var/log/logstash, so i created the folder by… --- ## [Unable to connect to Postgresql installed in local from logstash docker](https://discuss.elastic.co/t/unable-to-connect-to-postgresql-installed-in-local-from-logstash-docker/209549)
**Author:** [@pilo](https://discuss.elastic.co/u/pilo)\ **Replies:** 0\ **Last updated:** [November 26, 2019, 4:10pm UTC](https://discuss.elastic.co/t/unable-to-connect-to-postgresql-installed-in-local-from-logstash-docker/209549 "2019-11-26T16:10:09Z")
HI everyone, I am a beginner in logstash. I'm running Logstash version 7.4.1 on docker. I want to create a pipeline as simple as possible to load my database in Postgresql installed in local but i find an error with lo… --- ## [Logstash mapping sql xml column to elasticsearch index properties](https://discuss.elastic.co/t/logstash-mapping-sql-xml-column-to-elasticsearch-index-properties/209495)
**Author:** [@karlm](https://discuss.elastic.co/u/karlm)\ **Replies:** 3\ **Last updated:** [November 26, 2019, 3:38pm UTC](https://discuss.elastic.co/t/logstash-mapping-sql-xml-column-to-elasticsearch-index-properties/209495 "2019-11-26T15:38:13Z")
Im creating "my\_events" index in elasticsearch \> PUT http://localhost:9200/my\_events \> { \> "settings" : { \> "number\_of\_shards" : 1 \> }, \> "mappings" : { \> "propert… --- ## [How to use and in logstash filter](https://discuss.elastic.co/t/how-to-use-and-in-logstash-filter/209376)
**Author:** [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Replies:** 7\ **Last updated:** [November 26, 2019, 2:24pm UTC](https://discuss.elastic.co/t/how-to-use-and-in-logstash-filter/209376 "2019-11-26T14:24:28Z")
I want to check if "domain" and "\[100\]" in \[message\] { drop { } } but it is dropping only first "domain" and not "\[100\]". What am I doing wrong? --- ## [Nested objects, to flatten or split?](https://discuss.elastic.co/t/nested-objects-to-flatten-or-split/209281)
**Author:** [@twright8](https://discuss.elastic.co/u/twright8)\ **Replies:** 1\ **Last updated:** [November 26, 2019, 2:12pm UTC](https://discuss.elastic.co/t/nested-objects-to-flatten-or-split/209281 "2019-11-26T14:12:26Z")
Hi there, I'm using logstash and Kibana to visualise data stored in json format (about 2gigs worth). The ELK stack is all 7.4.2 However my understanding is that kibana doesn't support nested arrays of objects, of which… --- ## [How to parse ccda raw xml file using logstash](https://discuss.elastic.co/t/how-to-parse-ccda-raw-xml-file-using-logstash/209266)
**Author:** [@rameshkr1994](https://discuss.elastic.co/u/rameshkr1994)\ **Replies:** 3\ **Last updated:** [November 26, 2019, 1:52pm UTC](https://discuss.elastic.co/t/how-to-parse-ccda-raw-xml-file-using-logstash/209266 "2019-11-26T13:52:56Z")
Dear All. i have some use to parse or directly point Elasticsearch to query DSL on top of raw ccda xml file. below is the demo ccdaraw.xml file link: link raw CCDA.xml file Note : we have huge number of raw CCDA.xml … --- ## [Need help with metricbeat field update](https://discuss.elastic.co/t/need-help-with-metricbeat-field-update/209382)
**Author:** [@Jonny\_Reyes](https://discuss.elastic.co/u/Jonny_Reyes)\ **Replies:** 3\ **Last updated:** [November 26, 2019, 1:30pm UTC](https://discuss.elastic.co/t/need-help-with-metricbeat-field-update/209382 "2019-11-26T13:30:08Z")
Hi, I'm trying to replace a string value (process status that has values like running or stopped) but i'm failing. I only want to update "running" as an 1 and "stopped" as a 0. I've testing with filters like: filter {… --- ## [LOGSTASH system exit error while running logstash conf file](https://discuss.elastic.co/t/logstash-system-exit-error-while-running-logstash-conf-file/209500)
**Author:** [@Viveck](https://discuss.elastic.co/u/Viveck)\ **Replies:** 0\ **Last updated:** [November 26, 2019, 12:04pm UTC](https://discuss.elastic.co/t/logstash-system-exit-error-while-running-logstash-conf-file/209500 "2019-11-26T12:04:01Z")
When running my logstash conf file getting this type of error,plz help \[root@localhost logstash\]# ./bin/logstash -f /etc/logstash/conf.d/logstash-sample.conf Thread.exclusive is deprecated, use Thread::Mutex WARNING: … --- ## [If condition inside aggregate filter](https://discuss.elastic.co/t/if-condition-inside-aggregate-filter/209486)
**Author:** [@Anuwa\_A](https://discuss.elastic.co/u/Anuwa_A)\ **Replies:** 0\ **Last updated:** [November 26, 2019, 11:08am UTC](https://discuss.elastic.co/t/if-condition-inside-aggregate-filter/209486 "2019-11-26T11:08:04Z")
I want to create my document using mysql data, here when creating document i want to use if condition, This is my mysql data, id myid1 myid2 name 1 1 2 abc 2 1 2 xyz i want to… --- ## [Logstash - input filter ruby. Persisted vs memory queue: different way to store a blob column from Oracle](https://discuss.elastic.co/t/logstash-input-filter-ruby-persisted-vs-memory-queue-different-way-to-store-a-blob-column-from-oracle/209481)
**Author:** [@evecitas89](https://discuss.elastic.co/u/evecitas89)\ **Replies:** 0\ **Last updated:** [November 26, 2019, 11:02am UTC](https://discuss.elastic.co/t/logstash-input-filter-ruby-persisted-vs-memory-queue-different-way-to-store-a-blob-column-from-oracle/209481 "2019-11-26T11:02:16Z")
Hello everyone. I have a Logstash pipeline reading from Oracle database a Blob column, named INPUT\_MESSAGE. In the pipepiline.yml I was using this configuration for the pipeline: queue.type: memory In the pipeline.co… --- ## [Problem with JSON](https://discuss.elastic.co/t/problem-with-json/208896)
**Author:** [@Parry](https://discuss.elastic.co/u/Parry)\ **Replies:** 2\ **Last updated:** [November 26, 2019, 10:49am UTC](https://discuss.elastic.co/t/problem-with-json/208896 "2019-11-26T10:49:47Z")
Hello. I have problem with json logs. Some of logs isn't pushed to Elastic+Kibana properly. I get this error: "\[2019-11-21T08:51:24,441\]\[ERROR\]\[logstash.codecs.json \]\[main\] JSON parse error, original data now in me… --- ## [Logstash stops executing jdbc query](https://discuss.elastic.co/t/logstash-stops-executing-jdbc-query/209460)
**Author:** [@Ayden](https://discuss.elastic.co/u/Ayden)\ **Replies:** 0\ **Last updated:** [November 26, 2019, 9:56am UTC](https://discuss.elastic.co/t/logstash-stops-executing-jdbc-query/209460 "2019-11-26T09:56:24Z")
Hello, I have logstash running a jdbc input with a 5 minute scheduler and after few hours logstash stops doing the query. The latest log I have is: \[2019-11-25T18:00:31,645\]\[DEBUG\]\[logstash.inputs.jdbc \] Executing… --- ## [Max\_poll\_records doesn't works on startup](https://discuss.elastic.co/t/max-poll-records-doesnt-works-on-startup/209046)
**Author:** [@Sergey\_Smirnov](https://discuss.elastic.co/u/Sergey_Smirnov)\ **Replies:** 7\ **Last updated:** [November 26, 2019, 9:04am UTC](https://discuss.elastic.co/t/max-poll-records-doesnt-works-on-startup/209046 "2019-11-26T09:04:57Z")
Parameter max\_poll\_records of kafka plagin doesn't works on service startup and message sequence doesn't save. Logstash config: input { kafka { bootstrap\_servers =\> "0.dual.kafka.qa-env.com:9092,1.dual.kafka.qa-env… --- ## [Hostname verification failed while setting up logstash to connect with elasticsearch](https://discuss.elastic.co/t/hostname-verification-failed-while-setting-up-logstash-to-connect-with-elasticsearch/209450)
**Author:** [@Vikas\_Yadav](https://discuss.elastic.co/u/Vikas_Yadav)\ **Replies:** 1\ **Last updated:** [November 26, 2019, 9:01am UTC](https://discuss.elastic.co/t/hostname-verification-failed-while-setting-up-logstash-to-connect-with-elasticsearch/209450 "2019-11-26T09:01:03Z")
i'm setting up the ELK security using X-Pack, I generated the CA and Certs as suggested by the docs: bin/elasticsearch-certutil ca bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 Shipped them to all the ela… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=439) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=441)