Json load in elasticserach thru logstash

Elastic Stack Logstash
1

My SImple JSON FILE IS as below :-

{"hello":"world", "goodnight": "moon"}

configuration file in bin/logstash.conf as below :-
input
{
file
{
path => ["/home/cloudera/Desktop/Data/LOGSTASH_DATA/test.json"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter
{
mutate
{
replace => [ "message", "%{message}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}

}

output
{
elasticsearch {
protocol => "http"
codec => json
host => "localhost"
index => "json"
embedded => true
}

stdout { codec => rubydebug }

}
}

when I run ./logstash -f logstash.conf

I get the below error can some one please help me :-

fetched an invalid config {:config=>"input \n{\n file \n {\n path => ["/home/cloudera/Desktop/Data/LOGSTASH_DATA/test.json"]\n start_position => "beginning"\n sincedb_path => "/dev/null"\n }\n}\n\nfilter \n{\n mutate\n {\n replace => [ "message", "%{message}" ]\n gsub => [ 'message','\n','']\n }\n if [message] =~ /^{.*}$/\n {\n json { source => message }\n }\n\n}\n\noutput\n{ \n elasticsearch {\n protocol => "http"\n codec => json\n host => "localhost"\n index => "json"\n embedded => true\n }\n\n stdout { codec => rubydebug }\n}\n}\n\n", :reason=>"Expected one of #, input, filter, output at line 37, column 1 (byte 553) after ", :level=>:error}

KIndly provide me with an answer .

Thanks
Prosenjit

2

It looks like you're closing the output section twice.

3

Thanks for replying back
Hi When I don't close it I get the below error

as ./logstash -f logstash.conf
Settings: Default pipeline workers: 1
Pipeline aborted due to error {:exception=>"LogStash::ConfigurationError", :backtrace=>["/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/config/mixin.rb:88:in config_init'", "org/jruby/RubyHash.java:1342:ineach'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/config/mixin.rb:72:in config_init'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/outputs/base.rb:79:ininitialize'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/output_delegator.rb:74:in register'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:181:instart_workers'", "org/jruby/RubyArray.java:1613:in each'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:181:instart_workers'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:in run'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:instart_pipeline'"], :level=>:error}
stopping pipeline {:id=>"main"}

changed the .conf file to below :-
input
{
file
{
path => ["/home/cloudera/Desktop/Data/LOGSTASH_DATA/test.json"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter
{
mutate
{
replace => [ "message", "%{message}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}

}

output
{
elasticsearch {
protocol => "http"
codec => json
host => "localhost"
index => "json"
embedded => true
}

stdout { codec => rubydebug }

}

4

Please format your log snippet and configuration file as preformatted text (use the </> toolbar button).

5

Hi Magnus,

I did't get the point can u please send me an example .

Thanks
Prosenjit

6 
["/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/config/mixin.rb:88:in config_init'", "org/jruby/RubyHash.java:1342:ineach'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/config/mixin.rb:72:in config_init'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/outputs/base.rb:79:ininitialize'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/output_delegator.rb:74:in register'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:181:instart_workers'", "org/jruby/RubyArray.java:1613:in each'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:181:instart_workers'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:in run'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:instart_pipeline'"], :level=>:error}

stopping pipeline {:id=>"main"}

changed the .conf file to below :-

input
{
file
{
path => ["/home/cloudera/Desktop/Data/LOGSTASH_DATA/test.json"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter
{
mutate
{
replace => [ "message", "%{message}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}

}

output
{
elasticsearch {
protocol => "http"
codec => json
host => "localhost"
index => "json"
embedded => true
}

stdout { codec => rubydebug }
}

As you told .
Thanks
Prosenjit

7

Hi Magnusbaeck,

I am yet to get a reply from your side may ypu ,please reply so that I can resolve the issue :-

    `input 
{
file 
{
path => ["/home/cloudera/Desktop/Data/LOGSTASH_DATA/test.json"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter 
{
mutate
{
replace => [ "message", "%{message}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}

}

output
{ 
elasticsearch {
protocol => "http"
codec => json
host => "localhost"
index => "json"
embedded => true
}

stdout { codec => rubydebug }
}`

Kindly help .

8

You're using several deprecated options for the elasticsearch output (host, protocol, and embedded). Consult the documentation for your version of Logstash to see what options are available to you.

9

Hi Magnus,

Thanks for the response ,I have changed the logstash.conf file as below :-
input
{
file
{
path => ["/home/cloudera/Desktop/Data/LOGSTASH_DATA/test.json"]
start_position => "beginning"
sincedb_path => "/dev/null"
codec => "json"
}
}

filter
{
mutate
{
replace => [ "message", "%{message}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}

}

output
{
elasticsearch {
codec => json
index => "json"
}

stdout { debug => true}

}

However I am getting a different error know :-

 [cloudera@quickstart bin]$ ./logstash --verbose -f logstash.conf

starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 1
Registering file input {:path=>["/home/cloudera/Desktop/Data/LOGSTASH_DATA/test.json"], :level=>:info}
Using mapping template from {:path=>nil, :level=>:info}
Attempting to install template {:manage_template=>{"template"=>"logstash-", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"string", "index"=>"not_analyzed"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"float"}, "longitude"=>{"type"=>"float"}}}}}}}, :level=>:info}
New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["127.0.0.1"], :level=>:info}
Unknown setting 'debug' for stdout {:level=>:error}
Pipeline aborted due to error {:exception=>"LogStash::ConfigurationError", :backtrace=>["/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/config/mixin.rb:135:in config_init'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/outputs/base.rb:79:ininitialize'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/output_delegator.rb:74:in register'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:181:instart_workers'", "org/jruby/RubyArray.java:1613:in each'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:181:instart_workers'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:in run'", "/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:instart_pipeline'"], :level=>:error}
stopping pipeline {:id=>"main"}
Closing inputs {:level=>:info}
Closed inputs {:level=>:info}

Kindly help me for the same .

Thanks
Prosenjit.

10

Unknown setting 'debug' for stdout {:level=>:error}

Here's your problem.

11

Hi Magnus,

I have removed the code for stdout {debug => true } from the logstash.conf file and I receive the below output

Settings: Default pipeline workers: 1

Registering file input {:path=>["/home/cloudera/Desktop/Data/LOGSTASH_DATA/test.json"], :level=>:info}
Using mapping template from {:path=>nil, :level=>:info}
Attempting to install template {:manage_template=>{"template"=>"logstash-", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"string", "index"=>"not_analyzed"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"float"}, "longitude"=>{"type"=>"float"}}}}}}}, :level=>:info}
New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["127.0.0.1"], :level=>:info}
Starting pipeline {:id=>"main", :pipeline_workers=>1, :batch_size=>125, :batch_delay=>5, :max_inflight=>125, :level=>:info}
Pipeline main started

I am trying to load JSON data from logstash to elasticsearch

the data in the JSON file is like this "{"hello":"world", "goodnight": "moon"}".

Kindly help as after Pipeline main started it doesn't moves ahead .

Thanks
Prosenjit

12

I'm not sure you can expect to get that much more output. Try starting Logstash with --debug to get more clues.

13

Hi magnus,

I have run logstash with --debug option i get this type of option interval=>1, :level=>:debug, :file=>"logstash/outputs/elasticsearch/buffer.rb", :line=>"90", :method=>"interval_flush"} Flushing buffer at interval {:instance=>"#<LogStash::Outputs::ElasticSearch::Buffer:0x4dbfa7d0 @stopping=#<Concurrent::AtomicBoolean:0x61a27090>, @last_flush=2016-10-25 06:45:00 -0700, @flush_thread=#<Thread:0x37f4330d run>, @max_size=500, @operations_lock=#<Java::JavaUtilConcurrentLocks::ReentrantLock:0x143b72e>, @submit_proc=#<Proc:0x8de72ff@/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:57>, @flush_interval=1, ,

I have a secons question do we have a option to create a mapping in logstash.conf file ,please let me know.

Thanks
Prosenjit

14

Hmm, wait.

codec => json

Remove this. It might not do any harm but it won't help either. Apart from that I don't know what's up.

I have a secons question do we have a option to create a mapping in logstash.conf file

The elasticsearch output has a number of options related to index templates. What, specifically, do you want to accomplish?

15

Hi Magnus,

I have removed codec => json from the conf file ,wanted to know if I can create an mapping associated with the index ,is that a possibility .

Thanks
Prosenjit

16

Hi,

when I go the path /home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch and open the common.rb file , i see error in that file at the end of file `# For all other errors print out full connection issues
@logger.error(
"Attempted to send a bulk request to Elasticsearch configured at '#{@client.client_options[:hosts]}'," +
" but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using " +
"the configuration provided?",
:error_message => e.message,
:error_class => e.class.name,
:backtrace => e.backtrace
)

  @logger.debug("Failed actions for last bad bulk request!", :actions => actions)

  raise e
end

end
end; end; end`.

Please help.

Thanks
Prosenjit

17

Hi Magnus,

Kindly respond.

Thanks
Prosenjit

18

wanted to know if I can create an mapping associated with the index ,is that a possibility .

Yes, use index templates to control how fields are mapped. Again, see the related options for the elasticsearch output.

when I go the path /home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch and open the common.rb file , i see error in that file at the end of file

Yes? Are you getting stacktraces in your logs related to that piece of code, or why are you inspecting common.rb?

19

Hi Magnus,

As you told to use index templates to control ,how fields are mapped , I don't get a good URL ,I have searched over the net ,but I am unable to find a good tutorial ,can you help me on this.

I go to the path as/home/cloudera/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch and open the common.rb file

as when I run logstash with --debug mode I see that ,this is place where the error is there ,so wanted to check with you.

Thanks
Prosenjit.

20

As you told to use index templates to control ,how fields are mapped , I don't get a good URL ,I have searched over the net ,but I am unable to find a good tutorial ,can you help me on this.

I'm afraid I don't have time to write tailored tutorials, but I can answer specific questions. If the ES index template and mapping documentation as well as old posts here doesn't get you started I don't think I can help.