# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=873 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 874 --- ## [Sync elasticsearch data to another elasticsearch using logstash](https://discuss.elastic.co/t/sync-elasticsearch-data-to-another-elasticsearch-using-logstash/108065)
**Author:** [@prabhu\_pandurangan](https://discuss.elastic.co/u/prabhu_pandurangan)\ **Replies:** 0\ **Last updated:** [November 17, 2017, 5:49am UTC](https://discuss.elastic.co/t/sync-elasticsearch-data-to-another-elasticsearch-using-logstash/108065 "2017-11-17T05:49:30Z")
Hi I am a newbie to logstash, i would like your help here, to sync data every minute from one elasticsearch to another elasticsearch using logstash. Please help me here. --- ## [Does First Input First Output in logstash](https://discuss.elastic.co/t/does-first-input-first-output-in-logstash/107850)
**Author:** [@lsyoung](https://discuss.elastic.co/u/lsyoung)\ **Replies:** 6\ **Last updated:** [November 17, 2017, 3:23am UTC](https://discuss.elastic.co/t/does-first-input-first-output-in-logstash/107850 "2017-11-17T03:23:56Z")
Can logstash ensure the message first read by input-plugin can be output firstly with output-plugin? AND If if the output-plugin is elasticsearch, does the the docs write into elasticsearch by the same sequence with … --- ## [Could not retrieve remote IP address for beats input](https://discuss.elastic.co/t/could-not-retrieve-remote-ip-address-for-beats-input/104815)
**Author:** [@gotjoshua](https://discuss.elastic.co/u/gotjoshua)\ **Replies:** 5\ **Last updated:** [November 17, 2017, 2:43am UTC](https://discuss.elastic.co/t/could-not-retrieve-remote-ip-address-for-beats-input/104815 "2017-11-17T02:43:23Z")
"Could not retrieve remote IP address for beats input." https://github.com/logstash-plugins/logstash-input-beats/blob/master/lib/logstash/inputs/beats/message\_listener.rb#L34 I see this warning regularly. (even though … --- ## [Measuring multiple event durations with elapsed plugin](https://discuss.elastic.co/t/measuring-multiple-event-durations-with-elapsed-plugin/107809)
**Author:** [@joconner](https://discuss.elastic.co/u/joconner)\ **Replies:** 1\ **Last updated:** [November 17, 2017, 1:04am UTC](https://discuss.elastic.co/t/measuring-multiple-event-durations-with-elapsed-plugin/107809 "2017-11-17T01:04:34Z")
My event stream log items look like this: \, \, \, \ My event\_id fields can be several strings that look like this: FOO\_BEGIN FOO\_END BAR\_BEGIN BAR\_END BUZZ\_BEGIN BUZZ\_END … --- ## [Logstash is not converting a syslog date to date](https://discuss.elastic.co/t/logstash-is-not-converting-a-syslog-date-to-date/107985)
**Author:** [@ajstark123](https://discuss.elastic.co/u/ajstark123)\ **Replies:** 4\ **Last updated:** [November 16, 2017, 8:58pm UTC](https://discuss.elastic.co/t/logstash-is-not-converting-a-syslog-date-to-date/107985 "2017-11-16T20:58:56Z")
We are using the following: %{SYSLOGTIMESTAMP:syslog\_time} to extract the syslog date from the logs. The format of this field is Nov 16 11:43:38 in the logs. we are trying to convert syslogtime from a string to a da… --- ## [CSV conf ig does not parse properly](https://discuss.elastic.co/t/csv-conf-ig-does-not-parse-properly/107847)
**Author:** [@pgervais](https://discuss.elastic.co/u/pgervais)\ **Replies:** 9\ **Last updated:** [November 16, 2017, 8:10pm UTC](https://discuss.elastic.co/t/csv-conf-ig-does-not-parse-properly/107847 "2017-11-16T20:10:38Z")
I use a simple csv config file as shown below. A input { file { path =\> "/Users/pgervais/Elk/mach.csv” start\_position =\> “beginning” sincedb\_path =\> “/dev/null” } } filter { csv { convert =\> { "ObsDate" =\> “d… --- ## [How to update a document in elasticsearch with logstash http output plugin](https://discuss.elastic.co/t/how-to-update-a-document-in-elasticsearch-with-logstash-http-output-plugin/107995)
**Author:** [@kabali12345](https://discuss.elastic.co/u/kabali12345)\ **Replies:** 4\ **Last updated:** [November 16, 2017, 8:01pm UTC](https://discuss.elastic.co/t/how-to-update-a-document-in-elasticsearch-with-logstash-http-output-plugin/107995 "2017-11-16T20:01:47Z")
Hi all.... Here i am trying to update a document in elasticsearch with the help of output logstash http plugin here is my logstash conf input { elasticsearch { hosts =\> "192.168.1.59:9233" index =\> "titan… --- ## [Config has duplicate Ids](https://discuss.elastic.co/t/config-has-duplicate-ids/107991)
**Author:** [@bmh.01](https://discuss.elastic.co/u/bmh.01)\ **Replies:** 2\ **Last updated:** [November 16, 2017, 5:36pm UTC](https://discuss.elastic.co/t/config-has-duplicate-ids/107991 "2017-11-16T17:36:27Z")
Has anyone ran into this before? The only thing I can find is this bug report that seems to be untouched from the alpha days. https://github.com/elastic/logstash/issues/7403 Migrating my ELK stack from 5 to 6 and this … --- ## [Logstash Parsing Help](https://discuss.elastic.co/t/logstash-parsing-help/107988)
**Author:** [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\ **Replies:** 3\ **Last updated:** [November 16, 2017, 5:19pm UTC](https://discuss.elastic.co/t/logstash-parsing-help/107988 "2017-11-16T17:19:05Z")
I am trying to parse NASA IIS logs. I get the error below: \[2017-11-16T16:48:48,905\]\[INFO \]\[logstash.modules.scaffold\] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/confi… --- ## [Logstash systemd issue](https://discuss.elastic.co/t/logstash-systemd-issue/107992)
**Author:** [@kashif.alig](https://discuss.elastic.co/u/kashif.alig)\ **Replies:** 0\ **Last updated:** [November 16, 2017, 5:07pm UTC](https://discuss.elastic.co/t/logstash-systemd-issue/107992 "2017-11-16T17:07:01Z")
Hi I am setting up ELK stack for bro. I am testing logstash with file input and output plugin and using logstash-5.6.4-1. I can start logtash with /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/bro.conf --c… --- ## [Filebeat / logstash "merging" events?](https://discuss.elastic.co/t/filebeat-logstash-merging-events/107603)
**Author:** [@TheUnderdogr](https://discuss.elastic.co/u/TheUnderdogr)\ **Replies:** 1\ **Last updated:** [November 16, 2017, 3:58pm UTC](https://discuss.elastic.co/t/filebeat-logstash-merging-events/107603 "2017-11-16T15:58:15Z")
Howdy. I have setup ELK stack on one machine, and feed logs from filebeat from another machine, the one that hosts the actual logs, to logstash. Log entries are quite diverse, however they conform to the following stru… --- ## [Logstash-input-jdbc empty records](https://discuss.elastic.co/t/logstash-input-jdbc-empty-records/107972)
**Author:** [@mecbure](https://discuss.elastic.co/u/mecbure)\ **Replies:** 3\ **Last updated:** [November 16, 2017, 3:44pm UTC](https://discuss.elastic.co/t/logstash-input-jdbc-empty-records/107972 "2017-11-16T15:44:38Z")
I am trying to get db records from postgres to elastic but I am not able to get first three columns. Pdate and record\_id are coming but others(text) not. input { jdbc { jdbc\_connection\_string =\> "jdbc:postgresql:/… --- ## [How to convert String to interger](https://discuss.elastic.co/t/how-to-convert-string-to-interger/107956)
**Author:** [@saad](https://discuss.elastic.co/u/saad)\ **Replies:** 2\ **Last updated:** [November 16, 2017, 2:55pm UTC](https://discuss.elastic.co/t/how-to-convert-string-to-interger/107956 "2017-11-16T14:55:15Z")
Hi The input is cef and I have two fields (bytesIn and bytesOut). I want to convert these two fields from string to numeric fields such as interger or long. I tried to do the following in mutate as shown below: convert… --- ## [Setting document type with dynamic mapping to 'strict'](https://discuss.elastic.co/t/setting-document-type-with-dynamic-mapping-to-strict/107959)
**Author:** [@makibroshett](https://discuss.elastic.co/u/makibroshett)\ **Replies:** 0\ **Last updated:** [November 16, 2017, 2:43pm UTC](https://discuss.elastic.co/t/setting-document-type-with-dynamic-mapping-to-strict/107959 "2017-11-16T14:43:32Z")
Hi, I'm using a template with strict dynamic mapping turned on (i.e. ES output plugin will reject documents that don't have a defined mapping, or if the fields do not correspond to the mapping definition). I use this in… --- ## [Upgraded to 6.0.0, where is my latest data!?](https://discuss.elastic.co/t/upgraded-to-6-0-0-where-is-my-latest-data/107926)
**Author:** [@cosmo](https://discuss.elastic.co/u/cosmo)\ **Replies:** 2\ **Last updated:** [November 16, 2017, 1:55pm UTC](https://discuss.elastic.co/t/upgraded-to-6-0-0-where-is-my-latest-data/107926 "2017-11-16T13:55:10Z")
Hello I upgraded from 5.6 to 6.0 yesterday. I have a cron to download a daily log, then pass that into logstash to index all the events. eg bin/logstash -f custom-config.yml \< /logs/log-2017-11-15.log I haven't chang… --- ## [Parsing JSON Array results in JSON parse error](https://discuss.elastic.co/t/parsing-json-array-results-in-json-parse-error/107904)
**Author:** [@Robert\_Andersson](https://discuss.elastic.co/u/Robert_Andersson)\ **Replies:** 2\ **Last updated:** [November 16, 2017, 1:07pm UTC](https://discuss.elastic.co/t/parsing-json-array-results-in-json-parse-error/107904 "2017-11-16T13:07:56Z")
I am trying to parse a JSON array by using the json codec. I have been looking at split. But all the examples assume that the array is within a JSON object. But in my situation, the array is not within an object. The da… --- ## [Jdbc input scheduler in logstash configuration](https://discuss.elastic.co/t/jdbc-input-scheduler-in-logstash-configuration/107940)
**Author:** [@anchal](https://discuss.elastic.co/u/anchal)\ **Replies:** 1\ **Last updated:** [November 16, 2017, 1:04pm UTC](https://discuss.elastic.co/t/jdbc-input-scheduler-in-logstash-configuration/107940 "2017-11-16T13:04:49Z")
I am using a jdbc input in logstash configuration file. I dont want to schedule it regularly but want it to run only when some insert, update , delete operation happens in the DB. Is there a way i can do this ? --- ## [Eager\_global\_ordinals](https://discuss.elastic.co/t/eager-global-ordinals/105063)
**Author:** [@Mojster](https://discuss.elastic.co/u/Mojster)\ **Replies:** 1\ **Last updated:** [November 16, 2017, 12:49pm UTC](https://discuss.elastic.co/t/eager-global-ordinals/105063 "2017-11-16T12:49:21Z")
Continuing the discussion from Cannot set eager in ES as by ES guide: Can I set date to eagar\_global\_ordinals? My first search is still quite slow. This would be the last condition filed that is not under eager. And I… --- ## [JSON filter, query](https://discuss.elastic.co/t/json-filter-query/107902)
**Author:** [@runtman](https://discuss.elastic.co/u/runtman)\ **Replies:** 7\ **Last updated:** [November 16, 2017, 12:29pm UTC](https://discuss.elastic.co/t/json-filter-query/107902 "2017-11-16T12:29:46Z")
Hello, I am receiving alerts from SOPHOS in clear JSON, below is an example: {"end": "2017-11-15T21:35:18.000Z", "customer\_id": "x", "endpoint\_type": "computer", "dhost": "x", "type": "Event::Endpoint::WebControlViolati… --- ## [Issue with converting date/time log entry into timestamp field](https://discuss.elastic.co/t/issue-with-converting-date-time-log-entry-into-timestamp-field/107908)
**Author:** [@uher3805](https://discuss.elastic.co/u/uher3805)\ **Replies:** 1\ **Last updated:** [November 16, 2017, 12:00pm UTC](https://discuss.elastic.co/t/issue-with-converting-date-time-log-entry-into-timestamp-field/107908 "2017-11-16T12:00:02Z")
Hi, while there are many topics on this I have not been successful to identify the right solution. I am using logstash to parse (GROK) application log entries. Each line start with 2017.11.15 09:10:04.168 (i.e. YYYY.MM.… --- ## [RabbitMQ and Zabbix Plugins](https://discuss.elastic.co/t/rabbitmq-and-zabbix-plugins/107057)
**Author:** [@demudrol](https://discuss.elastic.co/u/demudrol)\ **Replies:** 7\ **Last updated:** [November 16, 2017, 11:38am UTC](https://discuss.elastic.co/t/rabbitmq-and-zabbix-plugins/107057 "2017-11-16T11:38:48Z")
Hi there! I've got two plugins: Rabbitmq input plugin Zabbix output plugin Took some data from RabbitMQ and now i've got responseAnswer.Code filed. Now i want to send this filed to zabbix: if \[type\] == "ld.api.lo… --- ## [Constantly repeating error: A plugin had an unrecoverable error. Will restart this plugin](https://discuss.elastic.co/t/constantly-repeating-error-a-plugin-had-an-unrecoverable-error-will-restart-this-plugin/107104)
**Author:** [@ajstark123](https://discuss.elastic.co/u/ajstark123)\ **Replies:** 2\ **Last updated:** [November 16, 2017, 10:58am UTC](https://discuss.elastic.co/t/constantly-repeating-error-a-plugin-had-an-unrecoverable-error-will-restart-this-plugin/107104 "2017-11-16T10:58:01Z")
We are getting the following error in the logs. We run fine for about 40 minutes then we start to get the following errors. It is unclear as to what went wrong. \[2017-11-09T15:03:38,289\]\[INFO \]\[logstash.pipeline … --- ## [Logstash logs are not found in /var/log/logstash](https://discuss.elastic.co/t/logstash-logs-are-not-found-in-var-log-logstash/107825)
**Author:** [@kabali12345](https://discuss.elastic.co/u/kabali12345)\ **Replies:** 1\ **Last updated:** [November 16, 2017, 10:56am UTC](https://discuss.elastic.co/t/logstash-logs-are-not-found-in-var-log-logstash/107825 "2017-11-16T10:56:36Z")
Hi all i have installed logstash with rpm i am trying to check it's logs in /var/log/logstash directory but i have found no files in that\[/var/log/logstash\] directory here i am using default logstash.yml file how ca… --- ## [Logstash error {:reason=\>"Expected one of #, \\", ', }](https://discuss.elastic.co/t/logstash-error-reason-expected-one-of/107752)
**Author:** [@Gaoo\_Nin\_Rice](https://discuss.elastic.co/u/Gaoo_Nin_Rice)\ **Replies:** 5\ **Last updated:** [November 16, 2017, 10:51am UTC](https://discuss.elastic.co/t/logstash-error-reason-expected-one-of/107752 "2017-11-16T10:51:47Z")
hi, I have a problem for my logstash config. Need to help what's happen from my config my config input { file { \["host", "127.0.0.1"\] path =\> "/drives/d/logstash-5.6.3/bin/data/Topping/Trigger/\*.csv" } } filter … --- ## [How to add ignore node attribute when using xml filter plugin to translate xml to json?](https://discuss.elastic.co/t/how-to-add-ignore-node-attribute-when-using-xml-filter-plugin-to-translate-xml-to-json/107903)
**Author:** [@yunmao](https://discuss.elastic.co/u/yunmao)\ **Replies:** 0\ **Last updated:** [November 16, 2017, 10:32am UTC](https://discuss.elastic.co/t/how-to-add-ignore-node-attribute-when-using-xml-filter-plugin-to-translate-xml-to-json/107903 "2017-11-16T10:32:45Z")
I use http input plugin to listen on port 8080, then I passing in POST http request with XML body. Then I use xml plugin to translate it into json. But for some nodes of the XML, they have attributes with namespace infor… --- ## [My grok parse error skip some value](https://discuss.elastic.co/t/my-grok-parse-error-skip-some-value/106429)
**Author:** [@Gaoo\_Nin\_Rice](https://discuss.elastic.co/u/Gaoo_Nin_Rice)\ **Replies:** 2\ **Last updated:** [November 16, 2017, 10:15am UTC](https://discuss.elastic.co/t/my-grok-parse-error-skip-some-value/106429 "2017-11-16T10:15:56Z")
i have problem to parse this value My input is this: 2013-02-28 09:57:56,SERVICEID|0863591797|topup|C00000001||10.0|20170110|N|aaa|bbb|ccc|aaaaabbb|ccccc|kkkkk|hhhhh My grok filter is this: ^%{TIMESTAMP\_ISO8601:ti… --- ## [How to get information from CVS file(working like jdbc\_streaming) in filter section for enrichment?](https://discuss.elastic.co/t/how-to-get-information-from-cvs-file-working-like-jdbc-streaming-in-filter-section-for-enrichment/107892)
**Author:** [@dsakpal](https://discuss.elastic.co/u/dsakpal)\ **Replies:** 4\ **Last updated:** [November 16, 2017, 10:01am UTC](https://discuss.elastic.co/t/how-to-get-information-from-cvs-file-working-like-jdbc-streaming-in-filter-section-for-enrichment/107892 "2017-11-16T10:01:49Z")
Scenario: Taking particular value from log file & depending on that value I need to get information from static CSV file. So we are getting ASSETID from log file. Using that ASSETID I need to find ASSETNAME & ASSETGRP … --- ## [Sending Logstash's logs to /var/log/logstash but not to elasticsearch](https://discuss.elastic.co/t/sending-logstashs-logs-to-var-log-logstash-but-not-to-elasticsearch/107051)
**Author:** [@Ferrow](https://discuss.elastic.co/u/Ferrow)\ **Replies:** 6\ **Last updated:** [November 16, 2017, 9:14am UTC](https://discuss.elastic.co/t/sending-logstashs-logs-to-var-log-logstash-but-not-to-elasticsearch/107051 "2017-11-16T09:14:14Z")
Hi guys.. Im very new to this, but i dont understand why I cant load my .conf file in /usr/share/logstash/bin.. using ./logstash -f file.conf getting a message saying: "Sending Logstash's logs to /var/log/logstash whic… --- ## [About the Logstash data output](https://discuss.elastic.co/t/about-the-logstash-data-output/107857)
**Author:** [@lightlevin](https://discuss.elastic.co/u/lightlevin)\ **Replies:** 1\ **Last updated:** [November 16, 2017, 9:00am UTC](https://discuss.elastic.co/t/about-the-logstash-data-output/107857 "2017-11-16T09:00:06Z")
I try to use metricbeat + logstash + influxdb for data collection, Logstash conf: influxdb { host =\> \["xxx.xxx.xxx.xxx"\] db =\> "metricbeat" codec =\> json data\_points =\> { "platform" =\> "%{platform}" "ho… --- ## [Elasticsearch aggregate Query from logstash](https://discuss.elastic.co/t/elasticsearch-aggregate-query-from-logstash/107543)
**Author:** [@ranjith](https://discuss.elastic.co/u/ranjith)\ **Replies:** 1\ **Last updated:** [November 16, 2017, 8:55am UTC](https://discuss.elastic.co/t/elasticsearch-aggregate-query-from-logstash/107543 "2017-11-16T08:55:17Z")
Hi All, I want to aggregate an index in elasticsearch using logstash and store that aggregate result in another index. input { elasticsearch { hosts =\> "http://50.254.92.204:9200" user =\> "elastic" pass… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=872) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=874)