Example needed for Perl Search::Elasticsearch

Elastic Stack Elasticsearch
1

Hi,

Simple question, but there seems to be a lack of detailed examples for
using the otherwise very useful Search::Elasticsearch CPAN module !

I'm getting syslog data into elasticsearch via fluentd.

What I'd like to do now is run a perl search that will give me results for
notice, emerg and crit events. As a test (seeing as I don't get many
emerg/crit events !), I've tried the below, but it only seems to pick up
notice events and doesn't return any info events !

Help welcome !

Thanks.

Tim

#!/usr/bin/perl

use 5.014;
use strict;
use warnings;
use autodie;

use Data::Dumper;
use Search::Elasticsearch;

my $e = Search::Elasticsearch->new();

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {
"bool" => {
"must" => {match => { severity => 'notice'},match =>
{ severity => 'info'}}
}
}
}
);

print Dumper($results);

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/42e60034-655f-46ca-979e-308b0e7532e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Try this to search "notice" or "info" severity.

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {
"bool" => {
"should" => [
{ match => { severity => 'notice'} },
{ match => { severity => 'info'} }
]
}
}
}
);

Jörg

On Wed, Aug 13, 2014 at 12:01 PM, Log Muncher railroaderslament@gmail.com
wrote:

Hi,

Simple question, but there seems to be a lack of detailed examples for
using the otherwise very useful Search::Elasticsearch CPAN module !

I'm getting syslog data into elasticsearch via fluentd.

What I'd like to do now is run a perl search that will give me results for
notice, emerg and crit events. As a test (seeing as I don't get many
emerg/crit events !), I've tried the below, but it only seems to pick up
notice events and doesn't return any info events !

Help welcome !

Thanks.

Tim

#!/usr/bin/perl

use 5.014;
use strict;
use warnings;
use autodie;

use Data::Dumper;
use Search::Elasticsearch;

my $e = Search::Elasticsearch->new();

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {
"bool" => {
"must" => {match => { severity => 'notice'},match
=> { severity => 'info'}}
}
}
}
);

print Dumper($results);

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/42e60034-655f-46ca-979e-308b0e7532e3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/42e60034-655f-46ca-979e-308b0e7532e3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFrb%3Dz246SNk4GGgbuZTSv-dh-GgPXdN%3DPOP1jhVhxZow%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Well the the Perl module certainly doesn't complain about the syntax, but
it stil doesn't manage to output anything other than the "notice" severity ?

$ perl test.pl | fgrep "severity"
'severity' => 'notice'
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice'
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',

$ cat test.pl
#!/usr/bin/perl

use 5.014;
use strict;
use warnings;
use autodie;

use Data::Dumper;
use Search::Elasticsearch;

my $e = Search::Elasticsearch->new();

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {
#match => { severity => 'notice'}
"bool" => {
"should" => [
{match => { severity => 'notice'}},
{match => { severity => 'info'}}
]
}
}
}
);

print Dumper($results);

On Wednesday, 13 August 2014 11:40:42 UTC+1, Jörg Prante wrote:

Try this to search "notice" or "info" severity.

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {
"bool" => {
"should" => [
{ match => { severity => 'notice'} },
{ match => { severity => 'info'} }
]
}
}
}
);

Jörg

On Wed, Aug 13, 2014 at 12:01 PM, Log Muncher <railroad...@gmail.com
<javascript:>> wrote:

Hi,

Simple question, but there seems to be a lack of detailed examples for
using the otherwise very useful Search::Elasticsearch CPAN module !

I'm getting syslog data into elasticsearch via fluentd.

What I'd like to do now is run a perl search that will give me results
for notice, emerg and crit events. As a test (seeing as I don't get many
emerg/crit events !), I've tried the below, but it only seems to pick up
notice events and doesn't return any info events !

Help welcome !

Thanks.

Tim

#!/usr/bin/perl

use 5.014;
use strict;
use warnings;
use autodie;

use Data::Dumper;
use Search::Elasticsearch;

my $e = Search::Elasticsearch->new();

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {
"bool" => {
"must" => {match => { severity => 'notice'},match
=> { severity => 'info'}}
}
}
}
);

print Dumper($results);

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/42e60034-655f-46ca-979e-308b0e7532e3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/42e60034-655f-46ca-979e-308b0e7532e3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1967d9c9-e53e-4037-803c-586dce6a6568%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

A reason may be that your result set size is too small for containing both
severity values. You could either try a larger result set size, or boost
the "info" clause so you get docs with "info" before "notice.

Jörg

On Wed, Aug 13, 2014 at 12:51 PM, Log Muncher railroaderslament@gmail.com
wrote:

Well the the Perl module certainly doesn't complain about the syntax, but
it stil doesn't manage to output anything other than the "notice" severity ?

$ perl test.pl | fgrep "severity"
'severity' => 'notice'
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice'
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',

$ cat test.pl
#!/usr/bin/perl

use 5.014;
use strict;
use warnings;
use autodie;

use Data::Dumper;
use Search::Elasticsearch;

my $e = Search::Elasticsearch->new();

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {

            #match => { severity => 'notice'}

            "bool" => {
                    "should" => [
                            {match => { severity => 'notice'}},
                            {match => { severity => 'info'}}
                    ]
            }
        }

}
);

print Dumper($results);

On Wednesday, 13 August 2014 11:40:42 UTC+1, Jörg Prante wrote:

Try this to search "notice" or "info" severity.

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {
"bool" => {
"should" => [
{ match => { severity => 'notice'} },
{ match => { severity => 'info'} }
]
}
}
}
);

Jörg

On Wed, Aug 13, 2014 at 12:01 PM, Log Muncher railroad...@gmail.com
wrote:

Hi,

Simple question, but there seems to be a lack of detailed examples for
using the otherwise very useful Search::Elasticsearch CPAN module !

I'm getting syslog data into elasticsearch via fluentd.

What I'd like to do now is run a perl search that will give me results
for notice, emerg and crit events. As a test (seeing as I don't get many
emerg/crit events !), I've tried the below, but it only seems to pick up
notice events and doesn't return any info events !

Help welcome !

Thanks.

Tim

#!/usr/bin/perl

use 5.014;
use strict;
use warnings;
use autodie;

use Data::Dumper;
use Search::Elasticsearch;

my $e = Search::Elasticsearch->new();

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {
"bool" => {
"must" => {match => { severity => 'notice'},match
=> { severity => 'info'}}
}
}
}
);

print Dumper($results);

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/42e60034-655f-46ca-979e-308b0e7532e3%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/42e60034-655f-46ca-979e-308b0e7532e3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1967d9c9-e53e-4037-803c-586dce6a6568%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1967d9c9-e53e-4037-803c-586dce6a6568%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGT62WWavGGDcB4gVkcW%2BdAF4jbYd81oTHiVPUw1ZXKQA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

Aahh.. newbie mistake ! I didn't realise the results were limited by
default. :wink:

Thanks !

On Wednesday, 13 August 2014 12:09:43 UTC+1, Jörg Prante wrote:

A reason may be that your result set size is too small for containing both
severity values. You could either try a larger result set size, or boost
the "info" clause so you get docs with "info" before "notice.

Jörg

On Wed, Aug 13, 2014 at 12:51 PM, Log Muncher <railroad...@gmail.com
<javascript:>> wrote:

Well the the Perl module certainly doesn't complain about the syntax, but
it stil doesn't manage to output anything other than the "notice" severity ?

$ perl test.pl | fgrep "severity"
'severity' => 'notice'
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice'
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',

$ cat test.pl
#!/usr/bin/perl

use 5.014;
use strict;
use warnings;
use autodie;

use Data::Dumper;
use Search::Elasticsearch;

my $e = Search::Elasticsearch->new();

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {

            #match => { severity => 'notice'}

            "bool" => {
                    "should" => [
                            {match => { severity => 'notice'}},
                            {match => { severity => 'info'}}
                    ]
            }
        }

}
);

print Dumper($results);

On Wednesday, 13 August 2014 11:40:42 UTC+1, Jörg Prante wrote:

Try this to search "notice" or "info" severity.

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {
"bool" => {
"should" => [
{ match => { severity => 'notice'} },
{ match => { severity => 'info'} }
]
}
}
}
);

Jörg

On Wed, Aug 13, 2014 at 12:01 PM, Log Muncher railroad...@gmail.com
wrote:

Hi,

Simple question, but there seems to be a lack of detailed examples for
using the otherwise very useful Search::Elasticsearch CPAN module !

I'm getting syslog data into elasticsearch via fluentd.

What I'd like to do now is run a perl search that will give me results
for notice, emerg and crit events. As a test (seeing as I don't get many
emerg/crit events !), I've tried the below, but it only seems to pick up
notice events and doesn't return any info events !

Help welcome !

Thanks.

Tim

#!/usr/bin/perl

use 5.014;
use strict;
use warnings;
use autodie;

use Data::Dumper;
use Search::Elasticsearch;

my $e = Search::Elasticsearch->new();

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {
"bool" => {
"must" => {match => { severity => 'notice'},match
=> { severity => 'info'}}
}
}
}
);

print Dumper($results);

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/42e60034-655f-46ca-979e-308b0e7532e3%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/42e60034-655f-46ca-979e-308b0e7532e3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1967d9c9-e53e-4037-803c-586dce6a6568%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1967d9c9-e53e-4037-803c-586dce6a6568%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/64102621-766c-4071-8e0c-5b6fdf7c5146%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

Would this be the correct syntax ?

{match => { severity => {query=>'info',boost=>20}}}

Even with the agressive boost, I'm still getting "notice" as the
prioritised results ?

On Wednesday, 13 August 2014 12:09:43 UTC+1, Jörg Prante wrote:

A reason may be that your result set size is too small for containing both
severity values. You could either try a larger result set size, or boost
the "info" clause so you get docs with "info" before "notice.

Jörg

On Wed, Aug 13, 2014 at 12:51 PM, Log Muncher <railroad...@gmail.com
<javascript:>> wrote:

Well the the Perl module certainly doesn't complain about the syntax, but
it stil doesn't manage to output anything other than the "notice" severity ?

$ perl test.pl | fgrep "severity"
'severity' => 'notice'
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice'
'severity' => 'notice',
'severity' => 'notice',
'severity' => 'notice',

$ cat test.pl
#!/usr/bin/perl

use 5.014;
use strict;
use warnings;
use autodie;

use Data::Dumper;
use Search::Elasticsearch;

my $e = Search::Elasticsearch->new();

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {

            #match => { severity => 'notice'}

            "bool" => {
                    "should" => [
                            {match => { severity => 'notice'}},
                            {match => { severity => 'info'}}
                    ]
            }
        }

}
);

print Dumper($results);

On Wednesday, 13 August 2014 11:40:42 UTC+1, Jörg Prante wrote:

Try this to search "notice" or "info" severity.

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {
"bool" => {
"should" => [
{ match => { severity => 'notice'} },
{ match => { severity => 'info'} }
]
}
}
}
);

Jörg

On Wed, Aug 13, 2014 at 12:01 PM, Log Muncher railroad...@gmail.com
wrote:

Hi,

Simple question, but there seems to be a lack of detailed examples for
using the otherwise very useful Search::Elasticsearch CPAN module !

I'm getting syslog data into elasticsearch via fluentd.

What I'd like to do now is run a perl search that will give me results
for notice, emerg and crit events. As a test (seeing as I don't get many
emerg/crit events !), I've tried the below, but it only seems to pick up
notice events and doesn't return any info events !

Help welcome !

Thanks.

Tim

#!/usr/bin/perl

use 5.014;
use strict;
use warnings;
use autodie;

use Data::Dumper;
use Search::Elasticsearch;

my $e = Search::Elasticsearch->new();

my $results = $e->search(
index => 'logstash-2014.08.13',
body => {
query => {
"bool" => {
"must" => {match => { severity => 'notice'},match
=> { severity => 'info'}}
}
}
}
);

print Dumper($results);

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/42e60034-655f-46ca-979e-308b0e7532e3%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/42e60034-655f-46ca-979e-308b0e7532e3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1967d9c9-e53e-4037-803c-586dce6a6568%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1967d9c9-e53e-4037-803c-586dce6a6568%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d61847eb-8fb8-46d8-b371-96f23e17fcd7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

Hiya

Simple question, but there seems to be a lack of detailed examples for
using the otherwise very useful Search::Elasticsearch CPAN module !

The idea was that the API of the module maps very closely to all of the
REST APIs in Elasticsearch, so that anything that works with raw curl
statements should be easy to translate into requests with Search::ES.

Btw, you can always see the equivalent curl statement output to STDERR with
the following:

$e = Search::Elasticsearch->new( trace_to => 'Stderr')

Would this be the correct syntax ?

{match => { severity => {query=>'info',boost=>20}}}

Even with the agressive boost, I'm still getting "notice" as the
prioritised results ?

That is the correct syntax. Perhaps try just searching for "info" to see
if you actually have matching results?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/46ddc9bf-5bc4-417f-a26d-82c1c5679eb0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

Clinton,

I get that, but for some reason it's not that easy to map them for a novice
to the API. I'm used to using kibana and have recently just started
messing with the Perl API to produce some automated reports. I find the
API to be very robust and has lots of features, but the lack of more
complex examples is inhibiting. I have a simple boolean query that I
struggled to get working from the examples and trying to use the query DSL
to make it work and I'm constantly getting parse errors. It wasn't until I
came across this thread that I actually got the thing to "work" without
parse errors. But it still doesn't work the way I would expect and that's
probably something flawed in my understanding. For example,

My query:

    body => {
            'query' =>  {
                    "bool" => {
                            "must" => [
                    { 'match' => { '@message' => 'session opened for

user' } },
{ 'match' => { 'tags' => 'syslog' } } ]
}
}
}

This returns a number of results that I wasn't expecting. For example:

in "@message" I get:

"Accepted publickey for from xxx.xxx.xxx.xxx port 1415 ssh2

I'm not sure why.

Does it match ANY word in my message? In this case "for". How do I get it
to match the entire string?

Thanks, and sorry for jumping in on someone else's thread. It just seemed
appropriate.

On Wednesday, August 13, 2014 9:42:38 AM UTC-4, Clinton Gormley wrote:

Hiya

Simple question, but there seems to be a lack of detailed examples for
using the otherwise very useful Search::Elasticsearch CPAN module !

The idea was that the API of the module maps very closely to all of the
REST APIs in Elasticsearch, so that anything that works with raw curl
statements should be easy to translate into requests with Search::ES.

Btw, you can always see the equivalent curl statement output to STDERR
with the following:

$e = Search::Elasticsearch->new( trace_to => 'Stderr')

Would this be the correct syntax ?

{match => { severity => {query=>'info',boost=>20}}}

Even with the agressive boost, I'm still getting "notice" as the
prioritised results ?

That is the correct syntax. Perhaps try just searching for "info" to see
if you actually have matching results?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e03c668d-2801-43b1-99a1-0053106f202d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.