"trustAnchors parameter must be non-empty" failure in httpClient
Description
Environment
Activity
Scott CantorMay 22, 2023 at 12:26 PM
I guess in the vein of perhaps explaining why I’ve never seen this, I don’t automate patching so I do always end up restarting Jetty after I update Java along with other patches.
Paul B. HensonMay 20, 2023 at 3:24 AM
I wouldn’t have to maintain it, I’d just cut out all the symlinks and point it directly to the final destination file /etc/pki/ca-trust/extracted/java/cacerts :). But given java was presumably updated for some reason not restarting jetty when that happens is a problem even if that’s not actually causing this failure.
Brent PutmanMay 20, 2023 at 2:07 AM
The other option would be to point your JVM at a trust store at a different path that you maintain manually and that won’t change unexpectedly due to RPM or platform updates. You’d just add a -Djavax.net.ssl.trustStore to your container start process for that. Obviously puts the burden on you to then maintain that file, so not 100% optimal. But at least gives more predictability.
Paul B. HensonMay 20, 2023 at 1:44 AM
I wouldn’t necessarily consider issues resulting from upgrading java and continuing to run the old version a bug. If something was running that had never instantiated a trust store before I’d expect it to fail when it did as the path it would use didn’t exist anymore. What’s weird is that it seems if you already have a valid trust store it will keep working for some indeterminate amount of time then suddenly unexpectedly break 8-/. Assuming that is the underlying root cause it made it kinda hard to correlate <sigh>. I guess I could downgrade/update java on my dev instances a lot and see if it happened more, but I think my best bet is to actually restart jetty when java gets updated and if it never happens again assume that’s what it was.
Scott CantorMay 19, 2023 at 12:39 PM
Well, given how patching systems happens nowadays, it’s definitely a bug if that’s really the trigger, but I tend to doubt it’s all that simple.
The HC5 comment had to do with the situation re: getting our designs adapted, I’ll raise it on today’s call.
As far as your question, the way we’d enforce it isn’t in the Spring parser per se (apart from requiring one of the two mechanisms) but in the component init check, it would flag the absence of the TrustEngine there if it’s not set to disregard.
Every now and again, not very frequently and not reproducible, downloads of metadata fail with this error:
2023-05-12 12:56:56,518 - 2620:df:8000:4785:0:2:d68:3254/node01rgbckgke63px17gt9q8t468e0781623 - ERROR [org.opensaml.saml.metadata.resolver.impl.AbstractDynamicMetadataResolver:869] - Metadata Resolver FunctionDrivenDynamicHTTPMetadataResolver incommon-mdq: Error fetching metadata from origin sourcejavax.net.ssl.SSLException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-emptyThis failure and error persist until I restart jetty. AFAICT there are no changes that occur before this happens, it just randomly starts to fail. Something somewhere somehow goes wiggy :). Here is the full stack trace:
2023-05-12 12:56:56,518 - 2620:df:8000:4785:0:2:d68:3254/node01rgbckgke63px17gt9q8t468e0781623 - ERROR [org.opensaml.saml.metadata.resolver.impl.AbstractDynamicMetadataResolver:869] - Metadata Resolver FunctionDrivenDynamicHTTPMetadataResolver incommon-mdq: Error fetching metadata from origin sourcejavax.net.ssl.SSLException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-emptyat java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:353)at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:296)at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:291)at java.base/sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1690)at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:471)at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)at net.shibboleth.utilities.java.support.httpclient.TLSSocketFactory.createLayeredSocket(TLSSocketFactory.java:324)at net.shibboleth.utilities.java.support.httpclient.TLSSocketFactory.connectSocket(TLSSocketFactory.java:280)at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)at org.apache.http.impl.client.cache.CachingExec.callBackend(CachingExec.java:592)at org.apache.http.impl.client.cache.CachingExec.handleCacheMiss(CachingExec.java:356)at org.apache.http.impl.client.cache.CachingExec.execute(CachingExec.java:275)at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)at net.shibboleth.utilities.java.support.httpclient.ContextHandlingHttpClient.doExecute(ContextHandlingHttpClient.java:132)at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)at org.opensaml.saml.metadata.resolver.impl.AbstractDynamicHTTPMetadataResolver.fetchFromOriginSource(AbstractDynamicHTTPMetadataResolver.java:274)at org.opensaml.saml.metadata.resolver.impl.AbstractDynamicMetadataResolver.resolveFromOriginSourceWithEntityID(AbstractDynamicMetadataResolver.java:842)at org.opensaml.saml.metadata.resolver.impl.AbstractDynamicMetadataResolver.resolveFromOriginSource(AbstractDynamicMetadataResolver.java:802)at org.opensaml.saml.metadata.resolver.impl.AbstractDynamicMetadataResolver.resolve(AbstractDynamicMetadataResolver.java:705)at org.opensaml.saml.metadata.resolver.impl.AbstractDynamicMetadataResolver.resolve(AbstractDynamicMetadataResolver.java:90)at org.opensaml.saml.metadata.resolver.ChainingMetadataResolver.resolve(ChainingMetadataResolver.java:175)at org.opensaml.saml.metadata.resolver.ChainingMetadataResolver.resolve(ChainingMetadataResolver.java:53)at net.shibboleth.idp.saml.metadata.impl.ReloadingRelyingPartyMetadataProvider.resolve(ReloadingRelyingPartyMetadataProvider.java:75)at net.shibboleth.idp.saml.metadata.impl.ReloadingRelyingPartyMetadataProvider.resolve(ReloadingRelyingPartyMetadataProvider.java:44)at org.opensaml.saml.metadata.resolver.impl.PredicateRoleDescriptorResolver.resolve(PredicateRoleDescriptorResolver.java:260)at org.opensaml.saml.metadata.resolver.impl.PredicateRoleDescriptorResolver.resolveSingle(PredicateRoleDescriptorResolver.java:245)at org.opensaml.saml.metadata.resolver.impl.PredicateRoleDescriptorResolver.resolveSingle(PredicateRoleDescriptorResolver.java:73)at org.opensaml.saml.common.binding.impl.SAMLMetadataLookupHandler.doInvoke(SAMLMetadataLookupHandler.java:158)at org.opensaml.messaging.handler.AbstractMessageHandler.invoke(AbstractMessageHandler.java:93)at net.shibboleth.idp.profile.impl.WebFlowMessageHandlerAdaptor.doExecute(WebFlowMessageHandlerAdaptor.java:195)at org.opensaml.profile.action.AbstractProfileAction.execute(AbstractProfileAction.java:180)at net.shibboleth.idp.profile.AbstractProfileAction.doExecute(AbstractProfileAction.java:154)at net.shibboleth.idp.profile.AbstractProfileAction.execute(AbstractProfileAction.java:126)at org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51)at org.springframework.webflow.action.EvaluateAction.doExecute(EvaluateAction.java:77)at org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188)at org.springframework.webflow.execution.AnnotatedAction.execute(AnnotatedAction.java:145)at org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51)at org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:101)at org.springframework.webflow.engine.State.enter(State.java:194)at org.springframework.webflow.engine.Transition.execute(Transition.java:228)at org.springframework.webflow.engine.impl.FlowExecutionImpl.execute(FlowExecutionImpl.java:395)at org.springframework.webflow.engine.impl.RequestControlContextImpl.execute(RequestControlContextImpl.java:214)at org.springframework.webflow.engine.TransitionableState.handleEvent(TransitionableState.java:116)at org.springframework.webflow.engine.Flow.handleEvent(Flow.java:547)at org.springframework.webflow.engine.impl.FlowExecutionImpl.handleEvent(FlowExecutionImpl.java:390)at org.springframework.webflow.engine.impl.RequestControlContextImpl.handleEvent(RequestControlContextImpl.java:210)at org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:105)at org.springframework.webflow.engine.State.enter(State.java:194)at org.springframework.webflow.engine.Flow.start(Flow.java:527)at org.springframework.webflow.engine.impl.FlowExecutionImpl.start(FlowExecutionImpl.java:368)at org.springframework.webflow.engine.impl.FlowExecutionImpl.start(FlowExecutionImpl.java:223)at org.springframework.webflow.executor.FlowExecutorImpl.launchExecution(FlowExecutorImpl.java:139)at org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:264)at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1071)at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:964)at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)at net.shibboleth.idp.log.SLF4JMDCServletFilter.doFilter(SLF4JMDCServletFilter.java:76)at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)at net.shibboleth.utilities.java.support.net.RequestResponseContextFilter.doFilter(RequestResponseContextFilter.java:61)at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)at net.shibboleth.utilities.java.support.net.DynamicResponseHeaderFilter.doFilter(DynamicResponseHeaderFilter.java:125)at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)at net.shibboleth.utilities.java.support.net.CookieBufferingFilter.doFilter(CookieBufferingFilter.java:68)at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)at net.shibboleth.utilities.java.support.net.SameSiteCookieHeaderFilter.doFilter(SameSiteCookieHeaderFilter.java:204)at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)at org.eclipse.jetty.server.Server.handle(Server.java:516)at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)at java.base/java.lang.Thread.run(Thread.java:829)Caused by: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: thetrustAnchors parameter must be non-emptyat java.base/sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:102)at java.base/sun.security.validator.Validator.getInstance(Validator.java:181)at java.base/sun.security.ssl.X509TrustManagerImpl.getValidator(X509TrustManagerImpl.java:300)at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrustedInit(X509TrustManagerImpl.java:176)at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:189)at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341)at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507)at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417)at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)... 128 common frames omittedCaused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-emptyat java.base/java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)at java.base/java.security.cert.PKIXParameters.<init>(PKIXParameters.java:120)at java.base/java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:104)at java.base/sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:99)