How to interact with .NET NEST object methods from powershell?

dreic · June 21, 2021, 5:37pm

How to call method of .NET object in powershell (Elasticsearch NEST .NET client). This is basic example of .NET code for NEST client and I am trying to write PS equivalent but I am failing in setting object value. Is this even possible to do with .NET object in powershell?

.NET example

var config = new Nest.ConnectionSettings(pool, httpConnection)
.DefaultIndex("employee_data"); //complete extensions omitted for brevity

Powershell

$config = new-object -Type Nest.ConnectionSettings($pool, $httpConnection)
$config.DefaultIndex("mate")
Method invocation failed because [Nest.ConnectionSettings] does not contain a method named 'DefaultIndex'.

$config | gm



   TypeName: Nest.ConnectionSettings

Name                                MemberType Definition                                                                                                                                                      
----                                ---------- ----------                                                                                                                                                      
ApiKeyAuthentication                Method     Nest.ConnectionSettings ApiKeyAuthentication(string id, securestring apiKey), Nest.ConnectionSettings ApiKeyAuthentication(string id, string apiKey), Nest.Co...
BasicAuthentication                 Method     Nest.ConnectionSettings BasicAuthentication(string username, string password), Nest.ConnectionSettings BasicAuthentication(string username, securestring pass...
ClientCertificate                   Method     Nest.ConnectionSettings ClientCertificate(X509Certificate certificate), Nest.ConnectionSettings ClientCertificate(string certificatePath)                       
ClientCertificates                  Method     Nest.ConnectionSettings ClientCertificates(System.Security.Cryptography.X509Certificates.X509CertificateCollection certificates)                                
ConnectionLimit                     Method     Nest.ConnectionSettings ConnectionLimit(int connectionLimit)                                                                                                    
DeadTimeout                         Method     Nest.ConnectionSettings DeadTimeout(timespan timeout)                                                                                                           
DefaultDisableIdInference           Method     Nest.ConnectionSettings DefaultDisableIdInference(bool disable)                                                                                                 
DefaultFieldNameInferrer            Method     Nest.ConnectionSettings DefaultFieldNameInferrer(System.Func[string,string] fieldNameInferrer)                                                                  
DefaultIndex                        Method     Nest.ConnectionSettings DefaultIndex(string defaultIndex)                                                                                                       
DefaultMappingFor                   Method     Nest.ConnectionSettings DefaultMappingFor[TDocument](System.Func[Nest.ClrTypeMappingDescriptor[TDocument],Nest.IClrTypeMapping[TDocument]] selector), Nest.Co...
DisableAutomaticProxyDetection      Method     Nest.ConnectionSettings DisableAutomaticProxyDetection(bool disable)                                                                                            
DisableDirectStreaming              Method     Nest.ConnectionSettings DisableDirectStreaming(bool b)                                                                                                          
DisableMetaHeader                   Method     Nest.ConnectionSettings DisableMetaHeader(bool disable)                                                                                                         
DisablePing                         Method     Nest.ConnectionSettings DisablePing(bool disable)                                                                                                               
Dispose                             Method     void IDisposable.Dispose()                                                                                                                                      
DnsRefreshTimeout                   Method     Nest.ConnectionSettings DnsRefreshTimeout(timespan timeout)                                                                                                     
EnableApiVersioningHeader           Method     Nest.ConnectionSettings EnableApiVersioningHeader(bool enable)                                                                                                  
EnableDebugMode                     Method     Nest.ConnectionSettings EnableDebugMode(System.Action[Elasticsearch.Net.IApiCallDetails] onRequestCompleted)                                                    
EnableHttpCompression               Method     Nest.ConnectionSettings EnableHttpCompression(bool enabled)                                                                                                     
EnableHttpPipelining                Method     Nest.ConnectionSettings EnableHttpPipelining(bool enabled)                                                                                                      
EnableTcpKeepAlive                  Method     Nest.ConnectionSettings EnableTcpKeepAlive(timespan keepAliveTime, timespan keepAliveInterval)                                                                  
EnableTcpStats                      Method     Nest.ConnectionSettings EnableTcpStats(bool enableTcpStats)                                                                                                     
EnableThreadPoolStats               Method     Nest.ConnectionSettings EnableThreadPoolStats(bool enableThreadPoolStats)                                                                                       
Equals                              Method     bool Equals(System.Object obj)                                                                                                                                  
GetHashCode                         Method     int GetHashCode()                                                                                                                                               
.
.
.

So as you see, Method DefaultIndex is there, so I am confused by this error.

forloop · June 22, 2021, 2:15am

This is a weird one and I'm not sure what the underlying cause is.

ConnectionSettings is implemented with a couple of patterns that might contribute:

Curiously recurring template pattern: ConnectionSettings derives from ConnectionSettingsBase<ConnectionSettings> where the methods like DefaultIndex are defined. The generic parameter to ConnectionSettingsBase<> is the return type of those methods.
Explicit interface implementation: ConnectionSettings explicitly implements IConnectionSettingsValues that defines a set of readonly properties for connection settings. ConnectionSettings then implements methods with the same names as the explicitly implemented properties, that set private fields from which the explicitly implemented properties read their values.

Although not a strongly typed API to Elasticsearch, have you seen Elastic.Console PowerShell package? It can provide a better experience than Invoke-WebRequest et. al for interacting with Elasticsearch from PowerShell.

dreic · June 22, 2021, 7:06am

Hello,

I am working with Elasticseach AWS service where I need to authenticate to it before I can do anything. Found GitHub - bcuff/elasticsearch-net-aws: Add-on to Elasticsearch.Net & NEST for using AWS's elasticsearch service. that can help me do that so I managed to auth and list indexes. Now I wanted to create actual logic that I will be using to achieve what I need but I am stuck on this. My PS skills are pretty decent, no .NET skills but I didn't expect this to be so hard

forloop · June 22, 2021, 7:48am

Can drop down to reflection to call it

$config.GetType().GetMethod("DefaultIndex").Invoke($config, "employee_data")

dreic · June 22, 2021, 7:58am

Well thank you very much, thank worked! I would never think of that, since I didn't know it even existed.

dreic · June 23, 2021, 4:57pm

I have 1 more problem, I believe it might be related to date type filed. Until I formatted time properties to match date type pattern, Kibana create index pattern for those as string. That is the only change I made and this stopped working.

Also I can't figure out how to use Bulk function. I have powershell object which is array of similar objects in json and if I try $client.Bulk($esrecords_json) then I get:

PS C:\Users\svcacct-adaxes> $client.Bulk($esrecords_json)
Cannot find an overload for "Bulk" and the argument count: "1".

This is indexDocument that isn't working. From DEV Tools this same json works when I use API to send document .

PS C:\Users\svcacctaccount> $tosendDoc
{
    "StartTime":  "2021-06-23T06:00:00",
    "CompletionTime":  "2021-06-23T06:00:46",
    "InitiatorName":  "UploadPasswordLogsToS3",
    "TargetObjectName":  "svcacctaccount",
    "TargetObjectType":  "user",
    "Description":  "Run PowerShell script 'uploadlogs' for 'svcacctaccount (domain.com\\Service Accounts)'"
}
PS C:\Users\svcacctaccount>  $client.IndexDocument($tosendDoc)


IsValid           : False
Id                : 
Index             : 
PrimaryTerm       : 0
Result            : Error
SequenceNumber    : 0
Shards            : 
Type              : 
Version           : 0
ApiCall           : Unsuccessful (400) low level call on POST: /index-test/_doc
DebugInformation  : Invalid NEST response built from a unsuccessful (400) low level call on POST: /index-test/_doc
                    # Audit trail of this API call:
                     - [1] BadResponse: Node: https://vpc-ctse-es-vsldctp.us-west-2.es.amazonaws.com/ Took: 00:00:00.0473368
                    # OriginalException: Elasticsearch.Net.ElasticsearchClientException: The remote server returned an error: (400) Bad Request. Call: Status code 400 from: POST /index-test/_doc. 
                    ServerError: Type: mapper_parsing_exception Reason: "failed to parse" CausedBy: "Type: not_x_content_exception Reason: "Compressor detection can only be called on some xcontent 
                    bytes or compressed xcontent bytes"" ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
                       at System.Net.HttpWebRequest.GetResponse()
                       at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData)
                       --- End of inner exception stack trace ---
                    # Request:
                    "{\r\n    \"StartTime\":  \"2021-06-23T06:00:00\",\r\n    \"CompletionTime\":  \"2021-06-23T06:00:46\",\r\n    \"InitiatorName\":  \"UploadPasswordLogsToS3\",\r\n    
                    \"TargetObjectName\":  \"svcacctaccount\",\r\n    \"TargetObjectType\":  \"user\",\r\n    \"Description\":  \"Run PowerShell script 'uploadlogs' for 'svcacctaccount 
                    (domain.com\\\\Service Accounts)'\"\r\n}"
                    # Response:
                    {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to 
                    parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}},"status":400}
                    
OriginalException : Elasticsearch.Net.ElasticsearchClientException: The remote server returned an error: (400) Bad Request. Call: Status code 400 from: POST /index-test/_doc. ServerError: Type: 
                    mapper_parsing_exception Reason: "failed to parse" CausedBy: "Type: not_x_content_exception Reason: "Compressor detection can only be called on some xcontent bytes or compressed 
                    xcontent bytes"" ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
                       at System.Net.HttpWebRequest.GetResponse()
                       at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData)
                       --- End of inner exception stack trace ---
ServerError       : ServerError: 400Type: mapper_parsing_exception Reason: "failed to parse" CausedBy: "Type: not_x_content_exception Reason: "Compressor detection can only be called on some xcontent 
                    bytes or compressed xcontent bytes""

I appreciate your help, would be nice if there was a documentation with examples that could help others like me.

dreic · June 23, 2021, 5:42pm

I figured out single document, it needs to be an object. I was converting it to json with
$tosendDoc = $esrecords | Select-Object -First 1 | ConvertTo-Json | % { [System.Text.RegularExpressions.Regex]::Unescape($_) }

PS C:\Users\svcacct-adaxes> $tosendDoc | gm


   TypeName: System.Object

Name             MemberType   Definition                                                                                                   
----             ----------   ----------                                                                                                   
Equals           Method       bool Equals(System.Object obj)                                                                               
GetHashCode      Method       int GetHashCode()                                                                                            
GetType          Method       type GetType()                                                                                               
ToString         Method       string ToString()                                                                                            
CompletionTime   NoteProperty string CompletionTime=2021-06-23T06:00:46                                                                    
Description      NoteProperty string Description=Run PowerShell script 'uploadlogs' for 'svcacct-adaxes (upworkcorp.com\\Service Accounts)'
InitiatorName    NoteProperty string InitiatorName=UploadPasswordLogsToS3                                                                  
StartTime        NoteProperty string StartTime=2021-06-23T06:00:00                                                                         
TargetObjectName NoteProperty string TargetObjectName=svcacct-adaxes                                                                       
TargetObjectType NoteProperty string TargetObjectType=user

Edit: Found out that what I need is .NET Lambda expression written in Powershell.

$client.Bulk({param($b) $b.IndexMany($esrecords)}) this didn't work

# Response:
                    {"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}

dreic · July 1, 2021, 4:53pm

I gave up on this, it was too cumbersome to do it powershell. Writing everything in C#.

system · July 29, 2021, 4:54pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.