I want solution for location mapping for my dataset. Following is the link for my dataset:
I am not able to get location type geo_point for mapping. Kindly advise.
Hi @aaditya_alekar,
It looks like that data set you linked to includes location data with a lat and long, so you should be able to ingest that as a geo_point fairly easily. There are several different formats that will work with Elasticsearch (object, string, geohash, array).
You'll need to use logstash or ingest pipeline to make sure the lat and long are changed to the geo_point shape at ingest time. See this thread for an example of how that works: Problem converting latitude and longitude into a geo point for Kibana
I am not able to create logstash config file using mac terminal. Kindly advise
You may want to start with the configuring logstash docs to get an idea as to how this process looks. It involves creating a .conf file, which you can do from the command line.
Also for your case, perhaps this thread, though old, would be even more useful: How to process "Lat" & "Long" fields using default Logstash config and mapping to use in Kibana 4 tile map
It includes a link to this blog post which gives some concrete examples of using mutate to turn separate latitude and longitude values into a geo_point location.
(I'm going to re-categorize this discussion under Logstash since it's more relevant to ingesting data... hope that's okay!) 
input{
file{
path => " /Users/aadi2391/Desktop/Documents/CIS5900\ Project/311_Service_Requests_from_2010_to_Present.csv "
start position => "beginning"
"sincedb_path" => "/dev/null"
}
}
filter {
csv {
seperator => ","
columns => { }
}
date {
locale => "eng"
match => {"Created Date" , "MM/dd/yyyy HH:mm:ss aa" , "ISO8601"
target => "Date"
remove_field => ["Created Date"]
}
mutate { convert => {"Latitude" => "float"} }
mutate { convert => {"Longitude" => "float"} }
mutate { rename => {"Latitude" => "[location][lat]"} }
mutate { rename => {"Longitude" => "[location][lon]"} }
}
output{
elasticsearch {
Hosts => ["localhost:9200"]
index => "nyc311calls"
document_type => "calls"
user => "elastic"
password => "changeme"}
stdout { codec => rubydebug { metadata => true } }
stdout { codec => dots }
}
is this correct?
can someone please help?
Hi you need to put a mapping in first if you have not done that ....
Example
PUT nyc311calls
{
"mappings": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
You need to do that before you run logstash and ingest documents otherwise you will not end up with a geo_point data type in the indexed documents
Also my question was how can do a logstash config in mac?
Whats the command for that?
Also with my dataset which all possible visualization i could do? and how to find out the CPU usage?
In the above mapping that you have provided what Index name should i give?
When you create the mapping it should have the same name as the index you create.
Perhaps you should read the docs on mapping
@aaditya_alekar you are asking many different types of questions... It seems that perhaps you should work one step at a time.... And reference the excellent documents.
Please see
bin/logstash -f /Users/aadi2391/Desktop/Documents/logstash-7.2.0/bin/logstash-Boston_Crimes.conf
-bash: bin/logstash: No such file or directory
I am not able to run .conf using logstash? Kindly help
It does not have to do with the .conf file ... bash cannot find logstash.
- You need to run that command from the home directory where you installed logstash
- or use the full path to the logstash executable
- or add the path to the logstash executable to your path.
sir can you give an example?
How did you install logstash on the Mac? Logstash does not come native on the Mac you need to install it.
Installation Instructions
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
I installed logstash from the elastic.co by first downloading then unzipping it
Aadityas-MacBook:~ aadi2391$ logstash
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/local/Cellar/logstash-full/7.3.0/libexec/logs which is now configured via log4j2.properties
ERROR: Pipelines YAML file is empty. Location: /usr/local/Cellar/logstash-full/7.3.0/libexec/config/pipelines.yml
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
[2019-08-04T11:24:31,016][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
i tried to run logstash but it didnt work
Looks like you installed with brew as well I can see that by the error message.
You may have 2 logstash installed one from brew and one you downloaded. I think that's where your confusion is. I would remove one or the other.
So to run the brew version which is already in your path.
logstash -f /Users/aadi2391/Desktop/Documents/logstash-7.2.0/bin/logstash-Boston_Crimes.conf
Please show the exact command you are running.
Aadityas-MacBook:documents aadi2391$ logstash -f /Users/aadi2391/Desktop/Documents/logstash-7.2.0/bin/logstash-Boston_Crimes.conf
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/local/Cellar/logstash-full/7.3.0/libexec/logs which is now configured via log4j2.properties
[2019-08-04T12:30:01,387][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-04T12:30:01,417][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.3.0"}
[2019-08-04T12:30:02,696][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 4, column 12 (byte 111) after input{\n file{\n path => " /Users/aadi2391/Desktop/Documents/CIS5900\ Project/crime\ 2.csv "\n start ", :backtrace=>["/usr/local/Cellar/logstash-full/7.3.0/libexec/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/local/Cellar/logstash-full/7.3.0/libexec/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "/usr/local/Cellar/logstash-full/7.3.0/libexec/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:inmap'", "/usr/local/Cellar/logstash-full/7.3.0/libexec/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:ininitialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/usr/local/Cellar/logstash-full/7.3.0/libexec/logstash-core/lib/logstash/java_pipeline.rb:24:ininitialize'", "/usr/local/Cellar/logstash-full/7.3.0/libexec/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/local/Cellar/logstash-full/7.3.0/libexec/logstash-core/lib/logstash/agent.rb:325:inblock in converge_state'"]}
[2019-08-04T12:30:03,220][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-08-04T12:30:08,215][INFO ][logstash.runner ] Logstash shut down.
I tried to run the file from the home directory where the logstash is installed and it gives error
Logstash is starting now... Now you have errors in your .conf file that you're going to need to work through.
The error is right there..
Expected one of #, => at line 4, column 12 (byte 111) after input{\n file{\n path => " /Users/aadi2391/Desktop/Documents/CIS5900\ Project/crime\ 2.csv "\n start ", :backtrace=>
I can see you right there you have a backslash\ instead of a forward slash /
Start working through your errors...
You can read the documents here
https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html
You can also add the -t option at the end of your current command and it will just test the configuration when it's correct it'll tell you.
Aadityas-MacBook:documents aadi2391$ logstash -f /Users/aadi2391/Desktop/Documents/logstash-7.2.0/config/logstash-Boston_Crimes.conf -t
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/local/Cellar/logstash-full/7.3.0/libexec/logs which is now configured via log4j2.properties
[2019-08-04T13:35:38,680][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-04T13:35:39,247][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of #, => at line 4, column 12 (byte 110) after input{
file {
path => " /Users/aadi2391/Desktop/Documents/CIS5900/Project/crime 2.csv "
start
[2019-08-04T13:35:39,262][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
This is the error now?