Export elasticsearch data into csv file

Nikparab · March 25, 2017, 5:45am

Hi
I want to export data which is indexed in elasticsearch
Here is my config file

input {
elasticsearch {
hosts => ["localhost:9200"]
index => "mongolog1"
query => '
{
"query": {
"bool": {
"must": [
{
"match":{
"content": "notification"
}
}
],
"must_not": [
{
"range": {
"duration": {
"gte": 500
}
}
}
]
}
}
}'
}
}
output {
csv {
fields => ["@timestamp", "content", "duration"]
path => "D:/ELKstack/elastic/mongo_log1.csv"
}
}

output-
logstash-5.1.1\bin>logstash -f output-csv.conf
nd log4j2 configuration at path /ELKstack/logstash-5.1.1/config/log4j2.properties. Using default config which logs to console
[[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pip
flight"=>500}
[[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
[Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
[[main]>worker0] INFO logstash.outputs.csv - Opening file {:path=>"D:/ELKstack/elastic/mongo_log1.csv"}
[LogStash::Runner] WARN logstash.agent - stopping pipeline {:id=>"main"}

this error i got.

Nikparab · March 25, 2017, 5:47am

how to resolve it???

teebu · March 25, 2017, 6:17pm

start it with --log.level debug to see more info

Nikparab · March 27, 2017, 4:51am

can you please tell where can i write this command?

I write this command as following
C:\ELK\logstash-5.2.2>bin\logstash -V --log.level debug -f bin\mongo-csv.conf
is this right way?
It giving me error

Could not find log4j2 configuration at path /ELK/logstash-5.2.2/config/log4j2.properties. Using default config which logs to console
logstash 5.2.2
jruby 1.7.25 (1.9.3p551) 2016-04-13 867cb81 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_121-b13 +jit [Windows 10-amd64]
java 1.8.0_121 (Oracle Corporation)
jvm Java HotSpot(TM) 64-Bit Server VM / 25.121-b13
gem addressable 2.3.8
gem atomic 1.1.99
gem avl_tree 1.2.1
gem awesome_print 1.7.0
gem jmespath 1.3.1
gem aws-sdk-core 2.3.22
gem aws-sdk-resources 2.3.22
gem aws-sdk 2.3.22
gem json 1.8.6
gem nokogiri 1.7.0.1
gem aws-sdk-v1 1.66.0
gem bindata 2.3.5
gem bson 4.2.1
gem buftok 0.2.0
gem builder 3.2.3
gem bundler 1.9.10
gem cabin 0.9.0
gem numerizer 0.1.1
gem chronic_duration 0.10.6
gem cinch 2.3.3
gem clamp 0.6.5
gem coderay 1.1.1
gem concurrent-ruby 1.0.0
gem unf 0.1.4
gem domain_name 0.5.20161129
gem dotenv 2.2.0
gem edn 1.1.1
gem multi_json 1.12.1
gem elasticsearch-api 5.0.3
gem multipart-post 2.0.0
gem faraday 0.9.2
gem elasticsearch-transport 5.0.3
gem elasticsearch 5.0.3
gem equalizer 0.0.10
gem ffi 1.9.17
gem minitar 0.5.4
gem file-dependencies 0.1.6
gem filesize 0.0.4
gem filewatch 0.9.0
gem gelfd 0.2.0
gem gems 0.8.3
gem hitimes 1.2.4
gem http-cookie 1.0.3
gem http-form_data 1.0.1
gem http_parser.rb 0.6.0
gem http 0.9.9
gem i18n 0.6.9
gem insist 1.0.0
gem jar-dependencies 0.3.10
gem jdbc-sqlite3 3.8.10.1
gem jls-grok 0.11.4
gem jls-lumberjack 0.0.26
gem jrjackson 0.4.2
gem jrmonitor 0.4.2
gem jruby-openssl 0.9.16
gem jruby-stdin-channel 0.2.0
gem ruby-maven-libs 3.3.9
gem ruby-maven 3.3.12
gem logstash-core-event-java 5.2.2
gem logstash-core-queue-jruby 5.2.2
gem method_source 0.8.2
gem slop 3.6.0
gem spoon 0.0.6
gem pry 0.10.4
gem puma 2.16.0
gem rubyzip 1.1.7
gem rack 1.6.5
gem rack-protection 1.5.3
gem tilt 2.0.6
gem sinatra 1.4.8
gem stud 0.0.22
gem thread_safe 0.3.5
gem polyglot 0.3.5
gem treetop 1.4.15
gem logstash-core 5.2.2
gem logstash-core-plugin-api 2.1.12
gem logstash-codec-cef 4.1.2
gem logstash-codec-collectd 3.0.3
gem logstash-codec-dots 3.0.2
gem logstash-codec-edn 3.0.2
gem logstash-codec-line 3.0.2
gem logstash-codec-edn_lines 3.0.2
gem logstash-codec-es_bulk 3.0.3
gem msgpack-jruby 1.4.1
gem logstash-codec-fluent 3.0.2
gem logstash-codec-graphite 3.0.2
gem logstash-codec-json 3.0.2
gem logstash-codec-json_lines 3.0.2
gem logstash-codec-msgpack 3.0.2
gem logstash-patterns-core 4.0.2
gem logstash-codec-multiline 3.0.3
gem logstash-codec-netflow 3.3.0
gem logstash-codec-plain 3.0.2
gem logstash-codec-rubydebug 3.0.2
gem logstash-filter-clone 3.0.2
gem logstash-filter-csv 3.0.2
gem logstash-filter-date 3.1.3
gem logstash-filter-dissect 1.0.8
gem lru_redux 1.1.0
gem logstash-filter-dns 3.0.3
gem logstash-filter-drop 3.0.2
gem murmurhash3 0.1.6
gem logstash-filter-fingerprint 3.0.2
gem logstash-filter-geoip 4.0.4
gem logstash-filter-grok 3.3.1
gem logstash-filter-json 3.0.2
gem logstash-filter-kv 3.1.1
gem metriks 0.9.9.7
gem logstash-filter-metrics 4.0.2
gem logstash-filter-mutate 3.1.3
gem logstash-filter-ruby 3.0.2
gem logstash-filter-sleep 3.0.3
gem logstash-filter-split 3.1.1
gem logstash-filter-syslog_pri 3.0.2
gem logstash-filter-throttle 4.0.1
gem logstash-filter-urldecode 3.0.2
gem user_agent_parser 2.3.0
gem logstash-filter-useragent 3.0.3
gem logstash-filter-uuid 3.0.2
gem xml-simple 1.1.5
gem logstash-filter-xml 4.0.2
gem logstash-input-beats 3.1.12
gem logstash-input-couchdb_changes 3.1.0
gem logstash-input-elasticsearch 4.0.2
gem logstash-input-exec 3.1.2
gem logstash-input-file 4.0.0
gem logstash-input-ganglia 3.1.0
gem logstash-input-gelf 3.0.2
gem logstash-input-generator 3.0.2
gem logstash-input-tcp 4.1.0
gem logstash-input-graphite 3.0.2
gem logstash-input-heartbeat 3.0.2
gem logstash-input-http 3.0.3
gem manticore 0.6.1
gem logstash-mixin-http_client 4.0.3
gem tzinfo 1.2.2
gem rufus-scheduler 3.0.9
gem logstash-input-http_poller 3.1.1
gem mime-types 2.6.2
gem mail 2.6.4
gem logstash-input-imap 3.0.2
gem logstash-input-irc 3.0.2
gem sequel 4.43.0
gem tzinfo-data 1.2016.10
gem logstash-input-jdbc 4.1.3
gem logstash-input-kafka 5.1.6
gem logstash-input-log4j 3.0.3
gem logstash-input-lumberjack 3.1.1
gem mongo 2.4.1
gem logstash-input-mongodb 0.4.1
gem logstash-input-pipe 3.0.2
gem march_hare 2.22.0
gem logstash-mixin-rabbitmq_connection 4.2.2
gem logstash-input-rabbitmq 5.2.2
gem redis 3.3.3
gem logstash-input-redis 3.1.2
gem logstash-mixin-aws 4.2.0
gem logstash-input-s3 3.1.2
gem snmp 1.2.0
gem logstash-input-snmptrap 3.0.2
gem logstash-input-sqs 3.0.2
gem logstash-input-stdin 3.2.2
gem logstash-input-syslog 3.2.0
gem memoizable 0.4.2
gem naught 1.1.0
gem simple_oauth 0.3.1
gem twitter 5.15.0
gem logstash-input-twitter 3.0.3
gem logstash-input-udp 3.1.0
gem logstash-input-unix 3.0.2
gem xmpp4r 0.5
gem logstash-input-xmpp 3.1.1
gem logstash-output-cloudwatch 3.0.3
gem logstash-output-file 4.0.1
gem logstash-output-csv 3.0.2
gem logstash-output-elasticsearch 6.2.6
gem logstash-output-graphite 3.1.1
gem logstash-output-http 4.1.0
gem logstash-output-irc 3.0.2
gem logstash-output-kafka 5.1.5
gem logstash-output-nagios 3.0.2
gem logstash-output-null 3.0.2
gem logstash-output-pagerduty 3.0.3
gem logstash-output-pipe 3.0.2
gem logstash-output-rabbitmq 4.0.6
gem logstash-output-redis 3.0.3
gem logstash-output-s3 4.0.5
gem logstash-output-sns 4.0.2
gem logstash-output-sqs 4.0.0
gem statsd-ruby 1.2.0
gem logstash-output-statsd 3.1.1
gem logstash-output-stdout 3.1.0
gem logstash-output-tcp 4.0.0
gem logstash-output-udp 3.0.2
gem snappy-jars 1.1.0.1.2
gem snappy 0.0.12
gem webhdfs 0.8.0
gem logstash-output-webhdfs 3.0.2
gem logstash-output-xmpp 3.0.2
gem mustache 0.99.8
gem paquet 0.2.0
gem pleaserun 0.0.28
gem ruby-progressbar 1.8.1

this is output.
I am not able to see logs in csv file

Nikparab · March 27, 2017, 11:33am

Is there any way to export data from elasticsearch to csv using kibana 5

ajnfde · March 28, 2017, 1:22pm

Hello, did you find anything ? I have a particular interest for this question.

Nikparab · March 29, 2017, 4:06am

yes @ajnfde
i install following plugin for logstash
bin/logstash-plugin install --version 3.0.3 logstash-output-csv

I got expected result by running conf file by using command

bin/logstash --log.level debug -f output-csv.conf

ajnfde · March 29, 2017, 8:01am

thanks @Nikparab. I understood you needed to export data from kibana 5 interface.

Nikparab · March 29, 2017, 10:16am

Yes.You welcom @ajnfde

system · April 26, 2017, 10:17am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.