Error accessing Key Vault while executing Azure Synapse Notebook via Synapse Pipelines - runtime-error

Question

I'm trying to execute an Azure Synapse Notebook using Notebook Activity in Synapse Pipelines and it keeps coming up with errors while debuging the Pipelines, the Notebook is using TokenLibrary.getSecret() and it looks like accessing key vault is the problem.
The notebook runs fine when executed its when added as pipeline activity the error comes up.
The key vault access policies are set to give both me and the synapse app get and list on secrets.
Many thanks for any help in advance.
Mariusz
Error msg for ref:
Microsoft Azure
Synapse Analytics
Search
/
feature/fixError6002 branch
Develop
Filter resources by name
SQL scripts
1
Notebooks
40
Data flows
1
Activities
Search activities
Synapse
Move & transform
Azure Data Explorer
Azure Function
Batch Service
Databricks
Data Lake Analytics
General
HDInsight
Iteration & conditionals
Machine Learning
Pipeline run ID:
58ff181a-37e6-47b8-b3d8-fe94295b9ec7
View debug run consumption
Name
Type
Run start
Duration
Status
Integration runtime
Run ID
Invoke Notebook
Notebook
2021-09-29T10:44:24.3680765Z
00:02:04
Failed
DefaultIntegrationRuntime (UK South)
9a6773d8-69e7-46f6-bc99-25878637b7b7
Log Pipeline Start
Azure Function
2021-09-29T10:44:19.9601216Z
00:00:03
Succeeded
IR-AZ-Runtime (UK South)
8280f852-1915-49ab-a6a8-35bf22a716bb
Error
{
"errorCode": "6002",
"message": "Py4JJavaError: An error occurred while calling z:com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.getSecret.\n: java.lang.Exception: Access token couldn't be obtained {\"result\":\"DependencyError\",\"errorId\":\"BadRequest\",\"errorMessage\":\"LSRServiceException is [{\\\"StatusCode\\\":400,\\\"ErrorResponse\\\":{\\\"code\\\":\\\"LSRResolveFailure\\\",\\\"message\\\":\\\"Cannot acquire MSI token for a Vault audience.\\\",\\\"target\\\":\\\"Vault\\\"},\\\"Message\\\":\\\"Cannot acquire MSI token for a Vault audience.\\\",\\\"Data\\\":{},\\\"InnerException\\\":null,\\\"StackTrace\\\":\\\" at Microsoft.Marlin.Common.ADF.Impl.LSRClient.CheckForFailures(HttpResponseMessage response, String responseContent) in C:\\\\\\\\source\\\\\\\\Common\\\\\\\\Microsoft.Marlin.Common.ADF\\\\\\\\Impl\\\\\\\\LSRClient.cs:line 272\\\\r\\\\n at Microsoft.Marlin.Common.ADF.Impl.LSRClient.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken, String traceId) in C:\\\\\\\\source\\\\\\\\Common\\\\\\\\Microsoft.Marlin.Common.ADF\\\\\\\\Impl\\\\\\\\LSRClient.cs:line 288\\\\r\\\\n at Microsoft.Marlin.Common.ADF.Impl.LSRClient.ResolveAudienceAsync(String audience, ResolveAudienceRequest request, String traceId, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\Common\\\\\\\\Microsoft.Marlin.Common.ADF\\\\\\\\Impl\\\\\\\\LSRClient.cs:line 164\\\\r\\\\n at Microsoft.Marlin.TokenService.Token.LSRAudienceTokenProvider.GetToken(Boolean isLinkedService, String audience, String sessionToken, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\TokenService\\\\\\\\Microsoft.Marlin.TokenService\\\\\\\\Token\\\\\\\\LSRAudienceTokenProvider.cs:line 140\\\\r\\\\n at Microsoft.Marlin.TokenService.Token.LSRAudienceTokenProvider.GetTokenForAudienceAsync(Boolean isLinkedService, String audience, String account, String sessionToken, SignaturePayload signaturePayload, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\TokenService\\\\\\\\Microsoft.Marlin.TokenService\\\\\\\\Token\\\\\\\\LSRAudienceTokenProvider.cs:line 67\\\\r\\\\n at Microsoft.Marlin.TokenService.Controllers.TokenController.GetTokenAsync(TokenRequest request, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\TokenService\\\\\\\\Microsoft.Marlin.TokenService\\\\\\\\Controllers\\\\\\\\TokenController.cs:line 67\\\\r\\\\n at lambda_method(Closure , Object )\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()\\\\r\\\\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)\\\\r\\\\n at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)\\\\r\\\\n at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)\\\\r\\\\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\\\\r\\\\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\\\\r\\\\n at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)\\\\r\\\\n at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)\\\\r\\\\n at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)\\\",\\\"HelpLink\\\":null,\\\"Source\\\":\\\"Microsoft.Marlin.Common.ADF\\\",\\\"HResult\\\":-2146233088}]. TraceId : 8158a0b1-4612-48de-aaa5-c443c564d25b. Error Component : LSR\"}\n\tat com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$$anonfun$9.apply(TokenLibrary.scala:382)\n\tat com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$$anonfun$9.apply(TokenLibrary.scala:374)\n\tat com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:1808)\n\tat com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:1807)\n\tat com.twitter.util.Promise$FutureTransformer.liftedTree1$1(Promise.scala:240)\n\tat com.twitter.util.Promise$FutureTransformer.k(Promise.scala:240)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:215)\n\tat com.twitter.util.Promise$WaitQueue.com$twitter$util$Promise$WaitQueue$$run(Promise.scala:91)\n\tat com.twitter.util.Promise$WaitQueue$$anon$4.run(Promise.scala:86)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:198)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:157)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:274)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:109)\n\tat com.twitter.util.Promise$WaitQueue.runInScheduler(Promise.scala:86)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:778)\n\tat com.twitter.util.Promise.update(Promise.scala:750)\n\tat com.twitter.util.Promise.setValue(Promise.scala:726)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:123)\n\tat com.twitter.finagle.netty4.transport.ChannelTransport$$anon$2.channelRead(ChannelTransport.scala:168)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat com.twitter.finagle.netty4.http.handler.UnpoolHttpHandler$.channelRead(UnpoolHttpHandler.scala:32)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat com.twitter.finagle.netty4.http.handler.ClientExceptionMapper$.channelRead(ClientExceptionMapper.scala:35)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat com.twitter.finagle.netty4.http.handler.HeaderValidatorHandler$.channelRead(HeaderValidatorHandler.scala:51)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)\n\tat io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)\n\tat io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)\n\tat io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:383)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:23)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:748)\n\nTraceback (most recent call last):\n\n File \"<stdin>\", line 9, in get_cloudinary_data\n\n File \"/opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py\", line 1257, in __call__\n answer, self.gateway_client, self.target_id, self.name)\n\n File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 69, in deco\n return f(*a, **kw)\n\n File \"/opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py\", line 328, in get_return_value\n format(target_id, \".\", name), value)\n\npy4j.protocol.Py4JJavaError: An error occurred while calling z:com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.getSecret.\n: java.lang.Exception: Access token couldn't be obtained {\"result\":\"DependencyError\",\"errorId\":\"BadRequest\",\"errorMessage\":\"LSRServiceException is [{\\\"StatusCode\\\":400,\\\"ErrorResponse\\\":{\\\"code\\\":\\\"LSRResolveFailure\\\",\\\"message\\\":\\\"Cannot acquire MSI token for a Vault audience.\\\",\\\"target\\\":\\\"Vault\\\"},\\\"Message\\\":\\\"Cannot acquire MSI token for a Vault audience.\\\",\\\"Data\\\":{},\\\"InnerException\\\":null,\\\"StackTrace\\\":\\\" at Microsoft.Marlin.Common.ADF.Impl.LSRClient.CheckForFailures(HttpResponseMessage response, String responseContent) in C:\\\\\\\\source\\\\\\\\Common\\\\\\\\Microsoft.Marlin.Common.ADF\\\\\\\\Impl\\\\\\\\LSRClient.cs:line 272\\\\r\\\\n at Microsoft.Marlin.Common.ADF.Impl.LSRClient.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken, String traceId) in C:\\\\\\\\source\\\\\\\\Common\\\\\\\\Microsoft.Marlin.Common.ADF\\\\\\\\Impl\\\\\\\\LSRClient.cs:line 288\\\\r\\\\n at Microsoft.Marlin.Common.ADF.Impl.LSRClient.ResolveAudienceAsync(String audience, ResolveAudienceRequest request, String traceId, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\Common\\\\\\\\Microsoft.Marlin.Common.ADF\\\\\\\\Impl\\\\\\\\LSRClient.cs:line 164\\\\r\\\\n at Microsoft.Marlin.TokenService.Token.LSRAudienceTokenProvider.GetToken(Boolean isLinkedService, String audience, String sessionToken, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\TokenService\\\\\\\\Microsoft.Marlin.TokenService\\\\\\\\Token\\\\\\\\LSRAudienceTokenProvider.cs:line 140\\\\r\\\\n at Microsoft.Marlin.TokenService.Token.LSRAudienceTokenProvider.GetTokenForAudienceAsync(Boolean isLinkedService, String audience, String account, String sessionToken, SignaturePayload signaturePayload, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\TokenService\\\\\\\\Microsoft.Marlin.TokenService\\\\\\\\Token\\\\\\\\LSRAudienceTokenProvider.cs:line 67\\\\r\\\\n at Microsoft.Marlin.TokenService.Controllers.TokenController.GetTokenAsync(TokenRequest request, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\TokenService\\\\\\\\Microsoft.Marlin.TokenService\\\\\\\\Controllers\\\\\\\\TokenController.cs:line 67\\\\r\\\\n at lambda_method(Closure , Object )\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()\\\\r\\\\n at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()\\\\r\\\\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)\\\\r\\\\n at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)\\\\r\\\\n at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)\\\\r\\\\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\\\\r\\\\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\\\\r\\\\n at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)\\\\r\\\\n at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)\\\\r\\\\n at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)\\\",\\\"HelpLink\\\":null,\\\"Source\\\":\\\"Microsoft.Marlin.Common.ADF\\\",\\\"HResult\\\":-2146233088}]. TraceId : 8158a0b1-4612-48de-aaa5-c443c564d25b. Error Component : LSR\"}\n\tat com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$$anonfun$9.apply(TokenLibrary.scala:382)\n\tat com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$$anonfun$9.apply(TokenLibrary.scala:374)\n\tat com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:1808)\n\tat com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:1807)\n\tat com.twitter.util.Promise$FutureTransformer.liftedTree1$1(Promise.scala:240)\n\tat com.twitter.util.Promise$FutureTransformer.k(Promise.scala:240)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:215)\n\tat com.twitter.util.Promise$WaitQueue.com$twitter$util$Promise$WaitQueue$$run(Promise.scala:91)\n\tat com.twitter.util.Promise$WaitQueue$$anon$4.run(Promise.scala:86)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:198)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:157)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:274)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:109)\n\tat com.twitter.util.Promise$WaitQueue.runInScheduler(Promise.scala:86)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:778)\n\tat com.twitter.util.Promise.update(Promise.scala:750)\n\tat com.twitter.util.Promise.setValue(Promise.scala:726)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:123)\n\tat com.twitter.finagle.netty4.transport.ChannelTransport$$anon$2.channelRead(ChannelTransport.scala:168)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat com.twitter.finagle.netty4.http.handler.UnpoolHttpHandler$.channelRead(UnpoolHttpHandler.scala:32)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat com.twitter.finagle.netty4.http.handler.ClientExceptionMapper$.channelRead(ClientExceptionMapper.scala:35)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat com.twitter.finagle.netty4.http.handler.HeaderValidatorHandler$.channelRead(HeaderValidatorHandler.scala:51)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)\n\tat io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)\n\tat io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)\n\tat io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:383)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:23)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:748)\n\n",
"failureType": "UserError",
"target": "Invoke Notebook",
"details": []
}

Answer 1

Please make sure to enable MSI(managed service identity) under identity > system /user assigned(ON) and make sure to add access policies properly to the KeyVault secrets, and also add a LinkedService to the Azure Synapse pointing to the KeyVault.
(Then restart again)
If your url is something like below
url = TokenLibrary.getSecret("mykeyvault", "ConnectionString")
Try to also linked service as a third parameter like below
url = TokenLibrary.getSecret("mykeyvault", "ConnectionString", "AzureKeyVaultLinkedServiceName")
References:
secure-credentials-with-tokenlibrary | Microsoft docs
access-secret-from-vault-using-synapse | microsoft Q&A
securely-access-Article-dzone

Answer 2

Running it in parallel, maybe with a for loop that is not sequentialmight be problematic. I seem to have a similar issue, when running the same notebook multiple times in parallel (each sending requests to the keyvault at the same time).