Search query problem

Gabriele_Angeli · September 6, 2013, 1:00pm

After this search query curl
localhost:9200/logstash-2013.09.06/_search?pretty=true i have this
results:

"_index" : "logstash-2013.09.06",
"_type" : "domain_alert",
"_id" : "82-WdTekRUOMPCxYrr7Ceg",
"_score" : 1.0, "_source" : {"@fields":{}
,"@timestamp":"2013-09-06T10:43:35.335Z","@message":""
,"@type":"domain_alert"}
}

The @fields and @message fields should be full of informations
because I'm be able to display the informations on Kibana dashboard.
Someone can help me?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

javanna · September 6, 2013, 1:45pm

Hi Gabriele,
can you see some information on Kibana about the very same document/log (id
82-WdTekRUOMPCxYrr7Ceg)?

Are you missing that information under @fields and @message only for that
specific document when querying elasticsearch? Or for all documents?

Cheers
Luca

On Friday, September 6, 2013 3:00:33 PM UTC+2, Gabriele Angeli wrote:

After this search query curl
localhost:9200/logstash-2013.09.06/_search?pretty=true i have this
results:

"_index" : "logstash-2013.09.06",
"_type" : "domain_alert",
"_id" : "82-WdTekRUOMPCxYrr7Ceg",
"_score" : 1.0, "_source" : {"@fields":{}
,"@timestamp":"2013-09-06T10:43:35.335Z","@message":""
,"@type":"domain_alert"}
}

The @fields and @message fields should be full of informations
because I'm be able to display the informations on Kibana dashboard.
Someone can help me?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Gabriele_Angeli · September 6, 2013, 2:00pm

Hi Luca,
in Kibana everythings is fine. I see all fields for all documents.

In elasticsearch i don't see @fields and @message informations for all
documents.

Regards

Gabriele

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · September 6, 2013, 2:34pm

Hey Gabriele,

How did you configure logstash?
What does your logstash.conf file looks like?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 6 sept. 2013 à 16:00, Gabriele Angeli g.angeli8214@gmail.com a écrit :

Hi Luca,
in Kibana everythings is fine. I see all fields for all documents.

In elasticsearch i don't see @fields and @message informations for all documents.

Regards

Gabriele

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Gabriele_Angeli · September 7, 2013, 9:52am

Hi David,

I post my configure file :

input {
file {
type => "data"
path => "/home/gabe/Deskop/data.log"
format => plain
start_position => "beginning"

filter {
grok {
type => "data"
pattern =>
"domain=%{GREEDYDATA:domain_name},TLD=%{GREEDYDATA:tld},Registrant_country=%{GREEDYDATA:Registrant_country},create_date=%{DATE:create_date},update_date=%{DATE:update_date},expiration_date=%{DATE:expiration_date},owner=%{GREEDYDATA:owner},associated_email=%{GREEDYDATA:associated_email},desc=%{GREEDYDATA:desc},category=%{GREEDYDATA:category},severity=%{INT:severity},score_level=%{INT:score_level},source_severity=%{INT:source_severity},false_positive=%{INT:false_positive}"
} }

output {
elasticsearch {
type => "data"
index => "logstash-%{+YYYY.MM.dd}"
}
}

I want to repeat that in Kibana3 i'm be able to view every single fields
but not in elasticsearch
In elasticserach I do the following query
localhost:9200/"index"/_search?pretty=true

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · September 7, 2013, 10:31am

Oh? Do you mean that curl commands does not work for you?

On almost every panel in Kibana, you have a small "eye" icon that helps to see the exact query Kibana does behind the scene.
I don't remember if the "spy" option exists on table panel. If it does, could you run the same query?

HTH

David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 7 sept. 2013 à 11:52, Gabriele Angeli g.angeli8214@gmail.com a écrit :

Hi David,

I post my configure file :

input {
file {
type => "data"
path => "/home/gabe/Deskop/data.log"
format => plain
start_position => "beginning"

filter {
grok {
type => "data"
pattern => "domain=%{GREEDYDATA:domain_name},TLD=%{GREEDYDATA:tld},Registrant_country=%{GREEDYDATA:Registrant_country},create_date=%{DATE:create_date},update_date=%{DATE:update_date},expiration_date=%{DATE:expiration_date},owner=%{GREEDYDATA:owner},associated_email=%{GREEDYDATA:associated_email},desc=%{GREEDYDATA:desc},category=%{GREEDYDATA:category},severity=%{INT:severity},score_level=%{INT:score_level},source_severity=%{INT:source_severity},false_positive=%{INT:false_positive}"
} }

output {
elasticsearch {
type => "data"
index => "logstash-%{+YYYY.MM.dd}"
}
}

I want to repeat that in Kibana3 i'm be able to view every single fields but not in elasticsearch
In elasticserach I do the following query localhost:9200/"index"/_search?pretty=true

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · September 7, 2013, 10:54am

I think Kibana uses field option when querying.

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 7 sept. 2013 à 12:31, David Pilato david@pilato.fr a écrit :

Oh? Do you mean that curl commands does not work for you?

On almost every panel in Kibana, you have a small "eye" icon that helps to see the exact query Kibana does behind the scene.
I don't remember if the "spy" option exists on table panel. If it does, could you run the same query?

HTH

David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 7 sept. 2013 à 11:52, Gabriele Angeli g.angeli8214@gmail.com a écrit :

Hi David,

I post my configure file :

input {
file {
type => "data"
path => "/home/gabe/Deskop/data.log"
format => plain
start_position => "beginning"

filter {
grok {
type => "data"
pattern => "domain=%{GREEDYDATA:domain_name},TLD=%{GREEDYDATA:tld},Registrant_country=%{GREEDYDATA:Registrant_country},create_date=%{DATE:create_date},update_date=%{DATE:update_date},expiration_date=%{DATE:expiration_date},owner=%{GREEDYDATA:owner},associated_email=%{GREEDYDATA:associated_email},desc=%{GREEDYDATA:desc},category=%{GREEDYDATA:category},severity=%{INT:severity},score_level=%{INT:score_level},source_severity=%{INT:source_severity},false_positive=%{INT:false_positive}"
} }

output {
elasticsearch {
type => "data"
index => "logstash-%{+YYYY.MM.dd}"
}
}

I want to repeat that in Kibana3 i'm be able to view every single fields but not in elasticsearch
In elasticserach I do the following query localhost:9200/"index"/_search?pretty=true

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Gabriele_Angeli · September 7, 2013, 2:36pm

SOLVED!!!

Hi David, thanks for yours precious informations!!!
My original query was incomplete. The problem was that.

Regards

Gabriele

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Gabriele_Angeli · September 8, 2013, 12:08pm

Hi David I saw kibana query, but i didn't understand which is the syntax to
extract @fields properties.
My question is: Is there a method to do this with a simple query? Because
kibana query is very complex.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · September 8, 2013, 3:42pm

Hey Gabriele,

Have a look at: Elasticsearch Platform — Find real-time answers at scale | Elastic

It should help.

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 8 sept. 2013 à 14:08, Gabriele Angeli g.angeli8214@gmail.com a écrit :

Hi David I saw kibana query, but i didn't understand which is the syntax to extract @fields properties.
My question is: Is there a method to do this with a simple query? Because kibana query is very complex.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Gabriele_Angeli · September 9, 2013, 7:56am

Thank you David,

  for the moment I resolved using the keyword "highlight".

Thank you very much for your time.
One of this day will add you on my twitter account.

Regards

Gabriele

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.