Visual Basic Examples for ElasticSearch.net and NEST

kvickery · April 3, 2020, 2:07am

Hi. I am trying to update a Visual Basic project to use the new Elasticsearch.Net and NEST api versions. Unfortunately whenever I try to find examples all I can find is c#.

Can anyone show me some examples of the following calls in Visual Basic?:

Indicies.Create
Ingest.Putpipeline
Search

Below are examples of the syntax issues I am having:

#1 - Indicies.Create - method Mappings is obsolete as of version 7.x

            Dim result = client.Indices.Create(index_name,
                                                Function(c) c.Mappings(Function(ms) ms.Map(Of IndexFile)(Function(m) m.AutoMap().AllField(Function(al) al.Enabled(False)).Properties(Function(ps) ps.Object(Of Attachment)(Function(a) a.Name(Function(n) n.Attachment).AutoMap())))
                                                                           ).Settings(Function(a) a.Analysis(Function(aa) aa.Analyzers(Function(am) am.Custom("filename_analyzier", Function(ca) ca.Filters("standard", "lowercase", "stop").Tokenizer("whitespace"))))))

#2 - Ingest.PutPipline - no version of PutPipeline found matching the parameters I am passing

            Return Me.client.Ingest.PutPipeline("attachments",
                    Function(p) p.Description("Document Attacment pipeline").Processors(
                        Function(pr) pr.Attachment(Of IndexFile)(
                            Function(a) a.Field(
                                Function(f) f.FileContent).TargetField(
                                    Function(f) f.Attachment)).Remove(Of IndexFile)(Function(r) r.Field(Function(f) f.FileContent)))).IsValid

#3 - Search - method Mappings is obsolete as of version 7.x

            Dim queryResult = Me.client.Search(Of IndexFile)(Function(d) d.Index(indexName).AllTypes().Query(Function(q) q.Match(Function(m) m.Field(Function(a) a.Attachment.Content).Query(queryTerm))).From(0).Size(1000))

forloop · April 13, 2020, 8:15am

Remove the .Mappings(..) call and use .Map(Of IndexFile) on c i.e.

Dim result = client.Indices.Create(index_name,
		Function(c) c.Map(Of IndexFile)(Function(m)

Field on RemoveProcessor takes a Fields or a Func<FieldsDescriptor, IPromise<Fields>>. Assuming FileContent is a String property, something like

	Dim result = client.Ingest.PutPipeline("attachments",
					Function(p) p.
						Description("Document Attacment pipeline").
						Processors(
							Function(pr) pr.
								Attachment(Of IndexFile)(
									Function(a) a.Field(
										Function(f) f.FileContent).TargetField(
											Function(f) f.Attachment)).
								Remove(Of IndexFile)(Function(r) r.Field(Function(f) f.Field(Of String)(Function (ff) ff.FileContent)))))

I think you mean AllTypes() is obsolete? Type mappings are obsolete and will be removed in a future major version. As such, the 7.x client does not have an AllTypes() method. If you remove this, it should be fine

Dim indexName As String = "my_index"
Dim queryTerm As String = "query"

Dim queryResult = client.Search(Of IndexFile)(Function(d) d.Index(indexName).Query(Function(q) q.Match(Function(m) m.Field(Function(a) a.Attachment.Content).Query(queryTerm))).From(0).Size(1000))

You may want to look at a C# to Visual Basic converter; there are quite a few examples in C# in the documentation, on here and stackoverflow, which may be helpful.

kvickery · April 14, 2020, 4:42pm

Thanks for the help Russ. I have made progress. The code converter you suggested is very useful.

I'm hoping you can help me out with a couple more api usages I have not been able to figure out yet.

First, in the Indices.Create call below I am getting a warning saying AllField is obsolete. How can I remove the AllField but still have the search work the same?

            Dim result = client.Indices.Create(index_name,
                                                Function(ms) ms.Map(Of IndexFile)(
                                                    Function(m) m.AutoMap().AllField(
                                                        Function(al) al.Enabled(False)).Properties(
                                                            Function(ps) ps.Object(Of Attachment)(
                                                                Function(a) a.Name(
                                                                    Function(n) n.Attachment).AutoMap()))
                                        ).Settings(
                                            Function(a) a.Analysis(
                                                Function(aa) aa.Analyzers(
                                                    Function(am) am.Custom("filename_analyzier",
                                                        Function(ca) ca.Filters("standard", "lowercase", "stop").Tokenizer("whitespace"))))))

Second, I have a search call with a function that builds the query. On the following line of code I get the "overload resolution failed" error unless I remove the ".Operator([Operator].And)", though this works fine with the Match method. Why can't I use the Operator suffix with MatchPhrase?

query = query And q.MatchPhrase(
    Function(m) m.Field("attachment.content").Query(keyword_str).Operator([Operator].And))

forloop · April 15, 2020, 12:58am

You can remove this. The _all field has been removed in 6.0.0+ and the replacement for it is to use copy_to on fields that you wish to search across, to copy values into an _all-like field that you define. Since your code is disabling the _all field however, we can simply remove this, as there is no _all field to disable anymore

You can remove this. This operator isn't valid for a match_phrase query, which is looking for the query input as a phrase within documents.

You may have been setting it in an earlier version of NEST such as 5.x, because of the hierarchy of query types in the client allowing it. It would have simply been ignored by Elasticsearch though. In NEST 7.x, the match family of queries were refactored in 6.x client, to match a change in query DSL structure in Elasticsearch.

kvickery · April 15, 2020, 1:56am

Thanks for your help again.

For the match_phrase query, it was being used in conjunction with other match and match_phrase queries, so my understanding was the And Operator was needed to build the query properly.

For example, a search might involve a combination of the term "swamp" and "pacific chorus frog", which would result in a match query combined with a match_phrase query. I will make sure the search combination works during my testing.

Thanks.

kvickery · April 15, 2020, 2:11am

I just realized I was probably not understanding the intent of the Operator method. I though it was relating the different query functions as they got combined, but I see now it probably dictates to the Match function if for a set of terms if just one term needs to be found or all.

So yes, it makes sense the MatchPattern function would have no use for the Operator method.

Sorry - I have never worked with Elasticsearch before and am having to learning quickly to upgrade an application to use the newer version of the server.

forloop · April 15, 2020, 11:40am

No worries, happy learning!

kvickery · April 15, 2020, 6:52pm

Hi again Russ. One last question (I hope).

My call to Ingest.PutPipelineRequest is failing with "No processor type exists with name [attachment]".

It sounds like I need to install the ingest-attachment plugin. I saw a post saying to use the command below for Windows but the suggested bat file is not present in my OpenDistro installation - maybe old instructions. Do you know how I can install or enable the plugin? I thought the OpenDistro package included plugins.

elasticsearch-plugin.bat install ingest-attachment

This is how I am starting Elasticsearch, under Windows 10:

docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" amazon/opendistro-for-elasticsearch:1.4.0

Here is my Indices.Create call:

            Dim result = client.Indices.Create(index_name,
                                                Function(ms) ms.Map(Of IndexFile)(
                                                    Function(m) m.AutoMap().AllField(
                                                        Function(al) al.Enabled(False)).Properties(
                                                            Function(ps) ps.Object(Of Attachment)(
                                                                Function(a) a.Name(
                                                                    Function(n) n.Attachment).AutoMap()))
                                        ).Settings(
                                            Function(a) a.Analysis(
                                                Function(aa) aa.Analyzers(
                                                    Function(am) am.Custom("filename_analyzier",
                                                        Function(ca) ca.Filters("standard", "lowercase", "stop").Tokenizer("whitespace"))))))

And here is my Ingest.PutPipelineRequest call:

            Dim putPipelineResponse = client.Ingest.PutPipeline("attachments",
                                                Function(p) p.Description("Document Attachment Pipeline").Processors(
                                                    Function(pr) pr.Attachment(Of IndexFile)(
                                                        Function(a) a.Field(Function(f) f.FileContent).TargetField(
                                                            Function(f) f.Attachment)).Remove(Of IndexFile)(
                                                                Function(r) r.Field(
                                                                    Function(f) f.Field(Of String)(
                                                                        Function(ff) ff.FileContent)))))

forloop · April 16, 2020, 1:00am

You'll need to install the ingest-attachment plugin. I'm not familiar enough with the OpenDistro fork to help here. You're right, that normally, one would call elasticsearch-plugin.bat to install the plugin. You may want to ask on the Amazon forums about this

kvickery · April 16, 2020, 1:05am

Update: I did some research and finally found what I think are the rights steps, though the outcome is not a success.

I created a Dockerfile as:

FROM elasticsearch:6.6.2 RUN bin/elasticsearch-plugin install --batch ingest-attachment

and then performed:

docker build -t elasticsearch-wao .

There was a warning from the build saying "WARNING: plugin requires additional permissions".

Then I created a new bat file to launch my build as:

docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" elasticsearch-wao:latest

I don't see any errors during startup but I don't see the final "node initialized" message either. "docker ps" show the server as up, but "curl" commands fail with the message:

curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_INVALID_TOKEN (0x80090308) - The token supplied to the function is invalid.

And communication with my client application is failing now ("handshake failed due to an unexpected packet format".

I feel like I am almost there but I am still missing a step (or two).

forloop · April 16, 2020, 1:21am

I created a docker image and ran it using your steps. I was able to curl Elasticsearch without problems

$ curl http://localhost:9200
{
  "name" : "Yl39C87",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "kXDQhIcUQsOxgo50gW8_2w",
  "version" : {
    "number" : "6.6.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "3bd3e59",
    "build_date" : "2019-03-06T15:16:26.864148Z",
    "build_snapshot" : false,
    "lucene_version" : "7.6.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

The error message implies some issue in establishing a security context. What curl command are you running?

An aside, if you're using 6.6.2 Elasticsearch, you'll want to use the latest 6.x version of the .NET client, currently 6.8.5.

kvickery · April 16, 2020, 1:41am

Below is a screen shot showing my curl command results. Let me know if you have trouble reading it.

forloop · April 16, 2020, 2:03am

The 6.6.2 docker image does not have basic authentication or TLS enabled. These features are available for free in

6.8.0+ (and less than 7.0.0)
7.1.0+

Check out the blog post for more details.

There's an example of configuring a cluster with Docker Compose in the docs, with TLS and configuring a bootstrap password

kvickery · April 17, 2020, 5:32pm

Hi Russ.

I thought I would give you an update. Silly mistake on my part that. When I setup my Dockerfile to create my custom build with the plugin I created my build from elasticsearch:7.6.1 - because OpenDistro 1.4.0 relates to elasticsearch 7.6.1. Made sense at the time.

FROM elasticsearch:7.6.1 RUN bin/elasticsearch-plugin install --batch ingest-attachment

But I should have been basing my build on the OpenDistro build that I had already been working with.

FROM amazon/opendistro-for-elasticsearch:1.4.0 RUN bin/elasticsearch-plugin install --batch ingest-attachment

This build starts up properly (node initialized), responds to my curl commands, and best of all the Ingest.PutPipelineRequest call works now.

Thanks again for your guidance.

kvickery · April 17, 2020, 10:30pm

Hi again.

All of my API calls are working now. The only problem I seem to have left is when I stop my Elasticsearch container (via docker stop) and start it up again the repository (indexes, etc.) are gone. Why are my created indexes etc. not being saved?

forloop · April 17, 2020, 11:00pm

Hey @kvickery, this is probably better asked as a new discussion topic, as it's about Docker and not .NET client with Visual Basic.

I suspect that the issue is path.data used by Elasticsearch to persist data is not pointing to a persistent location e.g. a docker volume, but to the writable container layer which doesn't persist when the container no longer exists..

See the documentation for an example of setting up a volume with Docker compose.

kvickery · April 17, 2020, 11:56pm

Thanks for pointing me in the right direction. I just needed to use docker-compose so I could define persistent volume.

system · May 15, 2020, 11:56pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.