Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Binary Encoding - Use Non-Binary Serializer for Query/ ReadManyItems/ Change Feed APIs #5043

Closed
Tracked by #4718
kundadebdatta opened this issue Mar 3, 2025 · 0 comments · Fixed by #5058
Closed
Tracked by #4718

Binary Encoding - Use Non-Binary Serializer for Query/ ReadManyItems/ Change Feed APIs #5043

kundadebdatta opened this issue Mar 3, 2025 · 0 comments · Fixed by #5058
Assignees
@aavasthy
Labels
BinaryEncoding binary encoding in .NET sdk needs-investigation

Comments

@kundadebdatta
Copy link
Member

kundadebdatta commented Mar 3, 2025 •
edited
Loading

Background:

Today, the CosmosClient uses a default version of CosmosJsonDotNetSerializer as a propertiesSerializer, which, by definition uses the ConfigurationManager.IsBinaryEncodingEnabled() to determine whether to use binary encoding during serialization and de-serialization.

This propertiesSerializer is hooked into multiple serializers (for query spec, change feed etc.) to use as internal serializers. This might cause some unexpected behaviors for APIs like ReadManyItems, ChangeFeed and cause a regression.

For example, during ReadManyItems invocation, a sanity test is throwing the below exception:

Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: ({"Errors":["One of the specified inputs is invalid"]}););
---> Microsoft.Azure.Cosmos.Query.Core.Exceptions.ExpectedQueryPartitionProviderException: {"Errors":["One of the specified inputs is invalid"]} ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x800A0B00
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Cosmos.CosmosQueryClientCore.<TryGetPartitionedQueryExecutionInfoAsync>d__9.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.CosmosQueryClientCore.TryGetPartitionedQueryExecutionInfoAsync(SqlQuerySpec sqlQuerySpec, ResourceType resourceType, PartitionKeyDefinition partitionKeyDefinition, VectorEmbeddingPolicy vectorEmbeddingPolicy, Boolean requireFormattableOrderByQuery, Boolean isContinuationExpected, Boolean allowNonValueAggregateQuery, Boolean hasLogicalPartitionKey, Boolean allowDCount, Boolean useSystemPrefix, GeospatialType geospatialType, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.QueryPlan.QueryPlanHandler.<TryGetQueryPlanAsync>d__2.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.QueryPlan.QueryPlanHandler.TryGetQueryPlanAsync(SqlQuerySpec sqlQuerySpec, ResourceType resourceType, PartitionKeyDefinition partitionKeyDefinition, VectorEmbeddingPolicy vectorEmbeddingPolicy, Boolean hasLogicalPartitionKey, Boolean useSystemPrefix, GeospatialType geospatialType, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.QueryPlan.QueryPlanRetriever.<GetQueryPlanWithServiceInteropAsync>d__5.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.QueryPlan.QueryPlanRetriever.GetQueryPlanWithServiceInteropAsync(CosmosQueryClient queryClient, SqlQuerySpec sqlQuerySpec, ResourceType resourceType, PartitionKeyDefinition partitionKeyDefinition, VectorEmbeddingPolicy vectorEmbeddingPolicy, Boolean hasLogicalPartitionKey, GeospatialType geospatialType, Boolean useSystemPrefix, ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.<GetPartitionedQueryExecutionInfoAsync>d__15.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.GetPartitionedQueryExecutionInfoAsync(CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, ContainerQueryProperties containerQueryProperties, ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.<TryCreateCoreContextAsync>d__8.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateCoreContextAsync(DocumentContainer documentContainer, CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.<>c__DisplayClass7_0.<Create>b__1(ITrace trace, CancellationToken innerCancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.AsyncLazy`1.<GetValueAsync>d__7.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.AsyncLazy`1.GetValueAsync(ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.LazyQueryPipelineStage.<MoveNextAsync>d__7.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.LazyQueryPipelineStage.MoveNextAsync(ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.NameCacheStaleRetryQueryPipelineStage.<MoveNextAsync>d__10.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.NameCacheStaleRetryQueryPipelineStage.MoveNextAsync(ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.CatchAllQueryPipelineStage.<MoveNextAsync>d__1.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.CatchAllQueryPipelineStage.MoveNextAsync(ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.QueryIterator.<ReadNextAsync>d__14.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.Query.QueryIterator.ReadNextAsync(ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.FeedIteratorInlineCore.ReadNextAsync(ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.ReadManyQueryHelper.<GenerateStreamResponsesForPartitionAsync>d__16.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.ReadManyQueryHelper.GenerateStreamResponsesForPartitionAsync(QueryDefinition queryDefinition, PartitionKeyRange partitionKeyRange, ReadManyRequestOptions readManyRequestOptions, ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.ReadManyQueryHelper.<>c__DisplayClass9_2.<<ReadManyTaskHelperAsync>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Cosmos.ReadManyQueryHelper.<>c__DisplayClass9_2.<ReadManyTaskHelperAsync>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
 
--- Cosmos Diagnostics ---{"Summary":{},"name":"ReadManyItemsAsync","start datetime":"2025-03-01T13:07:08.511Z","duration in milliseconds":1.3099,"data":{"Client Configuration":{"Client Created Time Utc":"2025-03-01T03:56:11.0197149Z","MachineId":"vmId:68908161-9010-4971-871c-ed1f4bbad241","VM Region":"westus","NumberOfClientsCreated":342,"NumberOfActiveClients":51,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.47.2|7|X64|Microsoft Windows 10.0.20348 |.NET Framework 4.8.4775.0|N|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: False)","rntbd":"(cto: 5, icto: 1200, mrpc: 30, mcpe: 65535, erd: True, pr: PrivatePortPool)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: Session, prgns:[], apprgn: )","ProcessorCount":8}},"children":[{"name":"Execute query for a partitionkeyrange","duration in milliseconds":1.0566,"data":{"Query Correlated ActivityId":"1db5602c-a4e6-4a56-99e6-1d94bbcbe8ba"},"children":[{"name":"Create Query Pipeline","duration in milliseconds":0.7758,"children":[{"name":"Get Container Properties","duration in milliseconds":0.0063,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.001}]},{"name":"Service Interop Query Plan","duration in milliseconds":0.6801}]}]}]}

Acceptance Criteria:

  • Update the CosmosSerializerCore to gracefully handle the binarySerializer and use it only with the operations, that are currently in scope for binary encoding.

  • Add test coverage to run the ReadManyItems API with binary encoding.
@kundadebdatta kundadebdatta added the BinaryEncoding binary encoding in .NET sdk label Mar 3, 2025
@kundadebdatta kundadebdatta moved this to In Progress in Azure Cosmos SDKs Mar 3, 2025
@kundadebdatta kundadebdatta mentioned this issue Mar 5, 2025
Open
This was referenced Mar 7, 2025
Closed
Merged
@github-project-automation github-project-automation bot moved this from In Progress to Done in Azure Cosmos SDKs Mar 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aavasthy aavasthy

Labels
BinaryEncoding binary encoding in .NET sdk needs-investigation
Projects
Azure Cosmos SDKs
Status: Done
Milestone
No milestone
2 participants
@kundadebdatta @aavasthy