Unhandled Exception in iis7_shib.dll
Description
Environment
Windows Server 2016 on VMWare
Our website is running mostly Classic ASP, I have set useHeaders="true" useVariables="false"
Activity
Scott Cantor August 31, 2020 at 12:54 PM
Release is done and advisory sent.
Scott Cantor August 28, 2020 at 4:07 PM
Installers are up in downloads/prerelease/
Rod Widdowson August 27, 2020 at 2:47 PM
Immediate fix for the most egregious bits .
Catch everything that ISAPI caught
including special case for catch_all
Walked the changes (forcing the error) and watch IIS return a 500 instead.
Before the next real patch I would like to
Bottom out why we need to throw an exception here. Why log and return null (for instance)
And if we cannot for structural reasons I would prefer to throw a defined exception which we can then handle better at the outer level.
Clean up the logging so HRESULTs go to the log in hex.
I suspect that that will engender some more code tidy now we have more insight into the way this stuff is put together.
Rod Widdowson August 27, 2020 at 6:00 AM
Yea, I’m not sure where that came from it may well be that my IIS install caught this. Who knows.
When I have myC++ head back on, my plan is to add the catch all, but also recast the internal use into using a specialized HRESULT-carrying Exception so I can just inject the result back into the request. Not my favorite paradigm but...
Scott Cantor August 26, 2020 at 11:23 PM(edited)
I see now that in fact this is a pretty expansive issue. Many error paths are calling a throwError method, so the exception being raised is in fact ours. I did not realize that. Obviously that's not viable unless we have an exception trap.
I believe the mistake was in thinking that we did, inside the various ServiceProvider methods (doAuthentication, etc.) because they do have try/catch logic. Unfortunately, I think what's happening here is recursive due to the error handling code in the catches. That code implements template-based error page handling, and I suspect that code may be ending up parsing the CGI state of the request and ending up back inside getRequestBody(), throwing, and that's where it escapes.
That said, as with the old ISAPI module, the lowest layer still should have try/catch block for all the standard exceptions, and then the configurable catch all.
A few months ago we started using the Shibboleth Service Provider v3.1.0.1 on IIS on Windows Server 2016 (with all recent Windows updates)
It can be running fine for days but then sometimes w3wp.exe crashes (a few times on a day) with no apparent cause.
Windows Error Reporting shows:
Faulting application name: w3wp.exe, version: 10.0.14393.0, time stamp: 0x57899b8a Faulting module name: KERNELBASE.dll, version: 10.0.14393.3659, time stamp: 0x5e9140ed Exception code: 0xe06d7363 Fault offset: 0x0000000000034f38 Faulting process id: 0xc44c Faulting application start time: 0x01d65efa426943f8 Faulting application path: c:\windows\system32\inetsrv\w3wp.exe Faulting module path: C:\Windows\System32\KERNELBASE.dllThe 0xe06d7363 translates to EXCEPTION_UNCAUGHT_CXX_EXCEPTION
Next step is to catch a minidump. The Call Stack is shown below
Iis7_shib.dll is very high in the call-stack with IIS7Request::throwError at the top.
It is called by IIS7Request::getRequestBody because a call to m_request->ReadEntityBody fails.
First of all: I am not sure Shibboleth is *causing* the crash, maybe it is just collateral damage. But it may help in finding the root cause
My questions:
A message goes into the IOException. Is there any way to retrieve that message (some logging perhaps? I could not find it in shibd.log or any other log)
--> In the minidump I can see the error is ERROR_NETNAME_DELETED
ReadEntityBody sound like it is receiving data. Is it fair to generate an (unhandled) exception for ‘just’ a network error?
If it is only collateral damage: is there a way to catch the exception?
I do see the <InProcess> element supports a (disadvised) catchAll but if I read the sourcecode correctly it may not apply to the IOException
-->Currently testing
(callstack and shibd.log follows)
KERNELBASE.dll!RaiseException-() Unknown [External Code] iis7_shib.dll!IIS7Request::throwError(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & operation, HRESULT hr) Line 452 C++ iis7_shib.dll!IIS7Request::getRequestBody() Line 332 C++ shibsp-lite3_1.dll!std::_Tree<std::_Tmap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,boost::shared_ptr<shibsp::DOMPropertySet>,std::less<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<shibsp::DOMPropertySet> > >,0> >::_Insert_hint<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<shibsp::DOMPropertySet> > &,std::_Tree_node<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<shibsp::DOMPropertySet> >,void *> *>(std::_Tree_const_iterator<std::_Tree_val<std::_Tree_simple_types<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<shibsp::DOMPropertySet> > > > > _Where, std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<shibsp::DOMPropertySet> > & _Val, std::_Tree_node<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,boost::shared_ptr<shibsp::DOMPropertySet> >,void *> * _Newnode) Line 1703 C++ shibsp-lite3_1.dll!shibsp::AbstractSPRequest::getHandlerURL(const char * resource) Line 247 C++ xmltooling-lite3_1.dll!xmltooling::TemplateEngine::TemplateParameters::getParameter(const char * name) Line 61 C++ shibsp-lite3_1.dll!shibsp::ServerThread::job() Line 597 C++ xmltooling-lite3_1.dll!xmltooling::TemplateEngine::process(bool visible, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & buf, const char * & lastpos, std::basic_ostream<char,std::char_traits<char> > & os, const xmltooling::TemplateEngine::TemplateParameters & parameters, const std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,std::basic_string<char,std::char_traits<char>,std::allocator<char> > > & loopentry, const xmltooling::XMLToolingException * e) Line 157 C++ xmltooling-lite3_1.dll!xmltooling::TemplateEngine::run(std::basic_istream<char,std::char_traits<char> > & is, std::basic_ostream<char,std::char_traits<char> > & os, const xmltooling::TemplateEngine::TemplateParameters & parameters, const xmltooling::XMLToolingException * e) Line 286 C++ shibsp-lite3_1.dll!shibsp::ServiceProvider::doHandler(shibsp::SPRequest & request) Line 684 C++ [External Code] shibsp-lite3_1.dll!shibsp::DOMPropertySet::load(const xercesc_3_2::DOMElement * e, log4shib::Category * log, xercesc_3_2::DOMNodeFilter * filter, const shibsp::DOMPropertySet::Remapper * remapper, const xmltooling::QName * unsetter) Line 151 C++ iis7_shib.dll!ShibHttpModule::DoHandler(IHttpContext * pHttpContext, IHttpEventProvider * pProvider) Line 62 C++ [External Code]Also: this is what shibd.log shows during the crash:
2020-08-19 17:13:17 ERROR Shibboleth.Listener [45]: error reading size of input message 2020-08-19 17:13:17 ERROR Shibboleth.Listener [45]: failed socket call (unknown), result (0): No error 2020-08-19 17:13:17 ERROR Shibboleth.Listener [45]: I/O failure processing request on socket (636) 2020-08-19 17:13:17 ERROR Shibboleth.Listener [47]: error reading size of input message 2020-08-19 17:13:17 ERROR Shibboleth.Listener [47]: failed socket call (unknown), result (0): No error 2020-08-19 17:13:17 ERROR Shibboleth.Listener [47]: I/O failure processing request on socket (440) 2020-08-19 17:13:17 ERROR Shibboleth.Listener [46]: error reading size of input message 2020-08-19 17:13:17 ERROR Shibboleth.Listener [46]: failed socket call (unknown), result (0): No error 2020-08-19 17:13:17 ERROR Shibboleth.Listener [46]: I/O failure processing request on socket (648)