# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=623 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 624 --- ## [Data mismatch in elasticsearch when loading data from database using logstash](https://discuss.elastic.co/t/data-mismatch-in-elasticsearch-when-loading-data-from-database-using-logstash/161513)
**Author:** [@balumurari1](https://discuss.elastic.co/u/balumurari1)\ **Replies:** 17\ **Last updated:** [January 7, 2019, 7:32am UTC](https://discuss.elastic.co/t/data-mismatch-in-elasticsearch-when-loading-data-from-database-using-logstash/161513 "2019-01-07T07:32:42Z")
Hello, I have loaded 6lakhs+ of data from database into elasticsearch using logstash. Observed the count of the number of rows in database is equal to the number of records in elasticsearch(as highlighted in yellow col… --- ## [Streaming Spring Boot Application Logs to ELK Stack with outFilebeat](https://discuss.elastic.co/t/streaming-spring-boot-application-logs-to-elk-stack-with-outfilebeat/163179)
**Author:** [@manickalai](https://discuss.elastic.co/u/manickalai)\ **Replies:** 3\ **Last updated:** [January 7, 2019, 10:48am UTC](https://discuss.elastic.co/t/streaming-spring-boot-application-logs-to-elk-stack-with-outfilebeat/163179 "2019-01-07T10:48:03Z")
Hello Peoples, I am exploring on Streaming Log Analysis project. So i used spring boot application to logback implementation with filebeat concept. could help withe best support in creating logback.xml and logstack.co… --- ## [Can not set up beats along with logstash](https://discuss.elastic.co/t/can-not-set-up-beats-along-with-logstash/161562)
**Author:** [@bertolis](https://discuss.elastic.co/u/bertolis)\ **Replies:** 2\ **Last updated:** [January 7, 2019, 10:17am UTC](https://discuss.elastic.co/t/can-not-set-up-beats-along-with-logstash/161562 "2019-01-07T10:17:37Z")
Hi, i am trying to configure beats working with logstash, using docker compose. If i don't use logstash then everything works perfectly in kibana's dashboard, but when i use logstash and i open kibana, i get some errors … --- ## [Only write the newest data into ES](https://discuss.elastic.co/t/only-write-the-newest-data-into-es/163168)
**Author:** [@ErikWarming](https://discuss.elastic.co/u/ErikWarming)\ **Replies:** 0\ **Last updated:** [January 7, 2019, 9:01am UTC](https://discuss.elastic.co/t/only-write-the-newest-data-into-es/163168 "2019-01-07T09:01:00Z")
I'm pulling a datafile into elasticsearch I'm using http\_poller with an API and I'm including the metadata where I get a last-modified timestamp for my datafile. What I want to do is compare the http\_poller timestamp w… --- ## [Nested objects in Elasticsearch and Kibana](https://discuss.elastic.co/t/nested-objects-in-elasticsearch-and-kibana/163166)
**Author:** [@amitnegi6190](https://discuss.elastic.co/u/amitnegi6190)\ **Replies:** 1\ **Last updated:** [January 7, 2019, 8:41am UTC](https://discuss.elastic.co/t/nested-objects-in-elasticsearch-and-kibana/163166 "2019-01-07T08:41:50Z")
Hi, I have a sample JSON file test.json with the following content: {"name":"Jonathan","score":"9.9","address":"New Delhi","lastUpdated":null, "firstUpdated":"86400","official": \[{"id": "ABCD1","uploader": {"AGS": 154… --- ## [Parse UNIX timestamp as human readable date while moving the data into elasticsearch](https://discuss.elastic.co/t/parse-unix-timestamp-as-human-readable-date-while-moving-the-data-into-elasticsearch/162317)
**Author:** [@amitnegi6190](https://discuss.elastic.co/u/amitnegi6190)\ **Replies:** 2\ **Last updated:** [January 7, 2019, 8:30am UTC](https://discuss.elastic.co/t/parse-unix-timestamp-as-human-readable-date-while-moving-the-data-into-elasticsearch/162317 "2019-01-07T08:30:19Z")
Hi, I have a JSON file which has a field named lastUpdated. It has the UNIX timestamp value for the time when the specified record was last updated. I wish to parse this JSON file and feed it to elasticsearch where this … --- ## [Elasticsearch score query with logstash does not work](https://discuss.elastic.co/t/elasticsearch-score-query-with-logstash-does-not-work/163165)
**Author:** [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Replies:** 0\ **Last updated:** [January 7, 2019, 8:23am UTC](https://discuss.elastic.co/t/elasticsearch-score-query-with-logstash-does-not-work/163165 "2019-01-07T08:23:23Z")
Hello. I have a working scoring query with decay exponential function calculated from a date. I want to use scheduler in logstash to run the query every 1h, score the results, and export it to json or csv. My logstash … --- ## [Logstash cannot handle huge amount of data](https://discuss.elastic.co/t/logstash-cannot-handle-huge-amount-of-data/162906)
**Author:** [@longnx](https://discuss.elastic.co/u/longnx)\ **Replies:** 5\ **Last updated:** [January 7, 2019, 8:02am UTC](https://discuss.elastic.co/t/logstash-cannot-handle-huge-amount-of-data/162906 "2019-01-07T08:02:00Z")
Hi Experts, We are using Logstash to get data from Oracle database using JDBC with SQL query base on ID and time stamp. The Oracle table has around 15 billions/ month and the data inserted continuous to that table. It … --- ## [Gorkparsefailure](https://discuss.elastic.co/t/gorkparsefailure/163147)
**Author:** [@Sripal](https://discuss.elastic.co/u/Sripal)\ **Replies:** 8\ **Last updated:** [January 7, 2019, 7:18am UTC](https://discuss.elastic.co/t/gorkparsefailure/163147 "2019-01-07T07:18:48Z")
Dear all, I am getting gorkparsefailure error for my log lines. Kindly help, 70% of my datas are not getting structured. Conf file file{ path =\> \["/my/path/files/13\_70/", "/my/path/files/13\_71/"\] exclude =\> "\*.gz" … --- ## [Problem ingest data filebeat with multi-pipeline](https://discuss.elastic.co/t/problem-ingest-data-filebeat-with-multi-pipeline/162337)
**Author:** [@hermann102](https://discuss.elastic.co/u/hermann102)\ **Replies:** 5\ **Last updated:** [January 7, 2019, 5:32am UTC](https://discuss.elastic.co/t/problem-ingest-data-filebeat-with-multi-pipeline/162337 "2019-01-07T05:32:14Z")
Hello I have successfully configured a multi-pipeline. Each pipeline normally loads the data. My multi-pipeline consists of three jdbc pipeline and one filebeat entry. This morning I was facing a problem loading data … --- ## [Logstash, kibana day early date issue](https://discuss.elastic.co/t/logstash-kibana-day-early-date-issue/163084)
**Author:** [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Replies:** 2\ **Last updated:** [January 6, 2019, 10:29pm UTC](https://discuss.elastic.co/t/logstash-kibana-day-early-date-issue/163084 "2019-01-06T22:29:57Z")
I am loading a csv file into elasticsearch using logstash. I used the date filter plugin to convert my date column to date type in my config file : date { match =\> \["DAY", "M/d/yyyy", "M/dd/yyyy", "MM/d/yyyy", "MM/dd/y… --- ## [How can update logstash field value?](https://discuss.elastic.co/t/how-can-update-logstash-field-value/162110)
**Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 3\ **Last updated:** [January 6, 2019, 8:12pm UTC](https://discuss.elastic.co/t/how-can-update-logstash-field-value/162110 "2019-01-06T20:12:49Z")
Logstash field: "keywords" =\> \[ \[0\] "Audit Success" \] I wish: "keywords" =\> "Audit Success" --- ## [How to configure "Logstash" syslog](https://discuss.elastic.co/t/how-to-configure-logstash-syslog/162074)
**Author:** [@kuckaprozova](https://discuss.elastic.co/u/kuckaprozova)\ **Replies:** 3\ **Last updated:** [January 6, 2019, 8:09pm UTC](https://discuss.elastic.co/t/how-to-configure-logstash-syslog/162074 "2019-01-06T20:09:55Z")
I have a question. I started using docker-elk but ı have a problem because Logstash syslog is not working.Actually the default conf file was working. This using : https://github.com/deviantony/docker-elk But I was work… --- ## [Logstash CSV Filter: Define Columns from Event Data/Fields](https://discuss.elastic.co/t/logstash-csv-filter-define-columns-from-event-data-fields/163014)
**Author:** [@dm90](https://discuss.elastic.co/u/dm90)\ **Replies:** 2\ **Last updated:** [January 6, 2019, 4:12pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-define-columns-from-event-data-fields/163014 "2019-01-06T16:12:45Z")
This is related to: https://discuss.elastic.co/t/csv-filter-column/122752 Use case: The S3 input plugin can return AWS CloudFront logs from an S3 bucket. The subsequent events include the message (a tab delimited log … --- ## [Dissect errors post upgrade](https://discuss.elastic.co/t/dissect-errors-post-upgrade/163081)
**Author:** [@NerdSec](https://discuss.elastic.co/u/NerdSec)\ **Replies:** 0\ **Last updated:** [January 6, 2019, 3:28pm UTC](https://discuss.elastic.co/t/dissect-errors-post-upgrade/163081 "2019-01-06T15:28:59Z")
Hi, Post upgrade to 6.5.4 from 6.3.2, we are seeing a lot of warnings in the dissect filter. We are handling these using multiple dissects in conditionals, but the warning messages are often distracting in the log file… --- ## [One or more required cgroup files not found of cgroup](https://discuss.elastic.co/t/one-or-more-required-cgroup-files-not-found-of-cgroup/163054)
**Author:** [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Replies:** 2\ **Last updated:** [January 6, 2019, 11:11am UTC](https://discuss.elastic.co/t/one-or-more-required-cgroup-files-not-found-of-cgroup/163054 "2019-01-06T11:11:02Z")
hello, I am using elasticsearch and kibana and logstash windows version without docker. whenever I want to load a file to elasticsearch I get endless loop of this error \[DEBUG\]\[logstash.instrument.periodicpoller.cgroup\]… --- ## [InstanceAlreadyExistsException for logstash id](https://discuss.elastic.co/t/instancealreadyexistsexception-for-logstash-id/163047)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 2\ **Last updated:** [January 6, 2019, 9:40am UTC](https://discuss.elastic.co/t/instancealreadyexistsexception-for-logstash-id/163047 "2019-01-06T09:40:19Z")
I have multiple pipelines in my Logstash configuration. Each pipeline has a dedicated Kafka topic. Should the client id be same for pipelines? It seems the client id is accepted for a few but creates an InstanceAlreadyEx… --- ## [Logstash multiple instances running together error](https://discuss.elastic.co/t/logstash-multiple-instances-running-together-error/162759)
**Author:** [@Apple23](https://discuss.elastic.co/u/Apple23)\ **Replies:** 2\ **Last updated:** [January 6, 2019, 6:09am UTC](https://discuss.elastic.co/t/logstash-multiple-instances-running-together-error/162759 "2019-01-06T06:09:22Z")
\[LogStash::Runner\] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting. h… --- ## [CloudWatch Integration to ELK Stack](https://discuss.elastic.co/t/cloudwatch-integration-to-elk-stack/163064)
**Author:** [@Mayank\_Khudia](https://discuss.elastic.co/u/Mayank_Khudia)\ **Replies:** 3\ **Last updated:** [January 6, 2019, 5:39am UTC](https://discuss.elastic.co/t/cloudwatch-integration-to-elk-stack/163064 "2019-01-06T05:39:26Z")
Hello Team, I am trying to integrate CloudWatch Logs to ELK stack. I have successfully managed to install all 3 components of the stack. But still unable to monitor and visualize log streams from CloudWatch. I have set … --- ## [Logstash is freaking out!](https://discuss.elastic.co/t/logstash-is-freaking-out/163021)
**Author:** [@OpSecMonkey](https://discuss.elastic.co/u/OpSecMonkey)\ **Replies:** 4\ **Last updated:** [January 6, 2019, 4:44am UTC](https://discuss.elastic.co/t/logstash-is-freaking-out/163021 "2019-01-06T04:44:13Z")
So I left the house for a little bit and I came back to seeing this on the screen. I am not sure what happened. \[2019-01-04T20:52:50,872\]\[INFO \]\[filewatch.observingtail \] START, creating Discoverer, Watch with file and… --- ## [I Can't Parse My Logs with Grok or Dissect Filter](https://discuss.elastic.co/t/i-cant-parse-my-logs-with-grok-or-dissect-filter/163052)
**Author:** [@Daniel\_Gonzalez1](https://discuss.elastic.co/u/Daniel_Gonzalez1)\ **Replies:** 5\ **Last updated:** [January 6, 2019, 12:37am UTC](https://discuss.elastic.co/t/i-cant-parse-my-logs-with-grok-or-dissect-filter/163052 "2019-01-06T00:37:45Z")
For Real! This is one of thousand reasons why Splunk in much much better and easy to use than Elastic. And that's why we can't sell this freakin tool with any customer for the hard of use. With Splunk I parsed a freakin… --- ## [What is wrong with my JSON file? Why this is not being parsed?](https://discuss.elastic.co/t/what-is-wrong-with-my-json-file-why-this-is-not-being-parsed/163026)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 12\ **Last updated:** [January 5, 2019, 8:43am UTC](https://discuss.elastic.co/t/what-is-wrong-with-my-json-file-why-this-is-not-being-parsed/163026 "2019-01-05T08:43:59Z")
Hi team, I am writing my logs in json file and and input filter is json directly but those are being parsed correctly in elasticsearch? Any clue what I am missing here? Here is my json file {"org\_name": "google.com", … --- ## [Logstash filter mutate modifies elasticsearch documents from input source](https://discuss.elastic.co/t/logstash-filter-mutate-modifies-elasticsearch-documents-from-input-source/162999)
**Author:** [@Sebastian\_Treu](https://discuss.elastic.co/u/Sebastian_Treu)\ **Replies:** 7\ **Last updated:** [January 5, 2019, 4:46am UTC](https://discuss.elastic.co/t/logstash-filter-mutate-modifies-elasticsearch-documents-from-input-source/162999 "2019-01-05T04:46:54Z")
When using elasticsearch as input and adding fields in logstash filter, the field is being added to the document in the sourced elasticsearch index. This is not an expecting behaviour, take into account this configurati… --- ## [I'm getting an error in logstash](https://discuss.elastic.co/t/im-getting-an-error-in-logstash/162878)
**Author:** [@Hamed\_khosravi](https://discuss.elastic.co/u/Hamed_khosravi)\ **Replies:** 4\ **Last updated:** [January 4, 2019, 6:34pm UTC](https://discuss.elastic.co/t/im-getting-an-error-in-logstash/162878 "2019-01-04T18:34:58Z")
i'm using logstash for parsing tomcat log file but since i wrote grok pattern for this log 127.0.0.1 - - \[11/Dec/2018:11:01:59 +0330\] "GET /parabank/template.css HTTP/1.1" 200 9571 and my grok pattern is ^%{IPORHOST:clie… --- ## [Add current time in logstash, when logs flow from filebeat -\> logstash -\> ES](https://discuss.elastic.co/t/add-current-time-in-logstash-when-logs-flow-from-filebeat-logstash-es/162995)
**Author:** [@Chaitra\_Ranganna](https://discuss.elastic.co/u/Chaitra_Ranganna)\ **Replies:** 0\ **Last updated:** [January 4, 2019, 5:52pm UTC](https://discuss.elastic.co/t/add-current-time-in-logstash-when-logs-flow-from-filebeat-logstash-es/162995 "2019-01-04T17:52:08Z")
Hi, We are pushing logs using below ELK stack filebeat -\> logstash -\> ES we would like to add a field when filebeat sees logs, when logstash sees the logs. I understand filebeat already adds @timestamp by this we c… --- ## [Twitter input not working on Debian build in Google compute node](https://discuss.elastic.co/t/twitter-input-not-working-on-debian-build-in-google-compute-node/162989)
**Author:** [@schmorgs](https://discuss.elastic.co/u/schmorgs)\ **Replies:** 0\ **Last updated:** [January 4, 2019, 4:21pm UTC](https://discuss.elastic.co/t/twitter-input-not-working-on-debian-build-in-google-compute-node/162989 "2019-01-04T16:21:23Z")
Bit of a strange one here. I'm trying to get the twitter input working for Logstash 6.5.4 running on Debian in Google compute node. Everything seems to start ok, but Logstash never actually collects any tweets and does… --- ## [GeoIP Maxmind Database Question](https://discuss.elastic.co/t/geoip-maxmind-database-question/162968)
**Author:** [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Replies:** 1\ **Last updated:** [January 4, 2019, 2:19pm UTC](https://discuss.elastic.co/t/geoip-maxmind-database-question/162968 "2019-01-04T14:19:26Z")
Hi all, Just had a quick question regarding using a custom (paid for) Maxmind GeoIP database, rather than the bundled one in Logstash. My question is if Logstash reads against the database each time the plugin is calle… --- ## [Logstash .conf file run forever as a service?](https://discuss.elastic.co/t/logstash-conf-file-run-forever-as-a-service/162952)
**Author:** [@inandi](https://discuss.elastic.co/u/inandi)\ **Replies:** 5\ **Last updated:** [January 4, 2019, 1:38pm UTC](https://discuss.elastic.co/t/logstash-conf-file-run-forever-as-a-service/162952 "2019-01-04T13:38:12Z")
i want to run logstash.conf file forever in server . Everytime i have to run this command sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf but i want run some command which will run forever… --- ## [Geoip country\_code3 always 2 chars](https://discuss.elastic.co/t/geoip-country-code3-always-2-chars/162962)
**Author:** [@Lorenzo\_Bagni](https://discuss.elastic.co/u/Lorenzo_Bagni)\ **Replies:** 0\ **Last updated:** [January 4, 2019, 1:33pm UTC](https://discuss.elastic.co/t/geoip-country-code3-always-2-chars/162962 "2019-01-04T13:33:25Z")
I've noticed that the field "country\_code3" of geoip filter always returns the code with only 2 chars instead 3. Configuration: input { stdin { } filter { geoip { source =\> "message" } } output { stdout { codec =\> ru… --- ## [Logstash cannot open file located in /etc/logstash](https://discuss.elastic.co/t/logstash-cannot-open-file-located-in-etc-logstash/162855)
**Author:** [@bigdamhero](https://discuss.elastic.co/u/bigdamhero)\ **Replies:** 2\ **Last updated:** [January 4, 2019, 12:59pm UTC](https://discuss.elastic.co/t/logstash-cannot-open-file-located-in-etc-logstash/162855 "2019-01-04T12:59:03Z")
I am trying (for the first time) to get an instance of logstash (running on ubuntu) to connect with my secured cluster. I am confused a bit on the security certificate formats but I am receiving this error so I am worki… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=622) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=624)