All work
- orphaned DiscoveryFeed cache files accumulateSSPCPP-731Resolved issue: SSPCPP-731Scott Cantor
- Successfully cached metadata documents containing ` ` subquently fail signature validationSSPCPP-684Resolved issue: SSPCPP-684Rod Widdowson
- Metadata AttributeExtractor unnecessarily escapes semicolon in attribute valuesSSPCPP-631Resolved issue: SSPCPP-631Scott Cantor
- missing shibsp1_5D.dllSSPCPP-626Resolved issue: SSPCPP-626Scott Cantor
- Dynamic metadata provider in SP should avoid unmarshalling non-EntityDescriptor resultsSSPCPP-598Resolved issue: SSPCPP-598Scott Cantor
- postTemplat.html form submission bugSSPCPP-595Resolved issue: SSPCPP-595Scott Cantor
- DiscoFeed Content-Type header lacks charsetSSPCPP-551Resolved issue: SSPCPP-551Scott Cantor
- shibd on Windows missing a version optionSSPCPP-505Resolved issue: SSPCPP-505Scott Cantor
- RequestInitiator metadata generated in a case where it shouldn't beSSPCPP-445Resolved issue: SSPCPP-445Scott Cantor
- Apache 2.4 supportSSPCPP-430Resolved issue: SSPCPP-430Scott Cantor
- Memcache build on RH6 and error handling fixesSSPCPP-420Resolved issue: SSPCPP-420Scott Cantor
- Add option to shibd to set uid and gid at startupSSPCPP-389Resolved issue: SSPCPP-389Scott Cantor
- RPM packaging problem - /etc/httpd/conf.d/shib.conf is overwritten on upgrade, but comments in file say "will be preserved across upgrades"SSPCPP-385Resolved issue: SSPCPP-385Scott Cantor
- DiscoFeed should return empty feed with no metadata providerSSPCPP-379Resolved issue: SSPCPP-379Scott Cantor
- metagen.sh creates PAOS ACS elements twiceSSPCPP-374Resolved issue: SSPCPP-374Scott Cantor
- Shibd: high cpu load, memory leakSSPCPP-373Resolved issue: SSPCPP-373Scott Cantor
- SSL_CHECK_SERVERHELLO_TLSEXTSSPCPP-370Resolved issue: SSPCPP-370Scott Cantor
- generated metadata should include cryptographic algorithmsSSPCPP-366Resolved issue: SSPCPP-366Scott Cantor
- add 'metadata last refresh' to SP's status pageSSPCPP-362Resolved issue: SSPCPP-362Scott Cantor
- Session handler with better parseable and accessable (X)HTML codeSSPCPP-361Resolved issue: SSPCPP-361Scott Cantor
- metagen.sh includes xmlns for NAKEDHOSTSSSPCPP-359Resolved issue: SSPCPP-359Scott Cantor
- Export info to application about how attributes, etc are being provided.SSPCPP-323Scott Cantor
- Implement token exchange for delegation within a callable handler.SSPCPP-308Scott Cantor
- Metadata Fetch with a UserAgent StringSSPCPP-297Resolved issue: SSPCPP-297Scott Cantor
- Expose SPNameQualifier as header variableSSPCPP-295Resolved issue: SSPCPP-295Scott Cantor
- Externalize security policy config and add plugin interface for itSSPCPP-293Resolved issue: SSPCPP-293Scott Cantor
- Metadata handler should generate both validUntil and cacheDurationSSPCPP-288Resolved issue: SSPCPP-288Scott Cantor
- Give dynamic metadata plugin ability to regex-transform entityID into a URLSSPCPP-284Resolved issue: SSPCPP-284Scott Cantor
- RFE: Load sysconfig file from init.d script.SSPCPP-280Resolved issue: SSPCPP-280Scott Cantor
- Isolate schema files in a versioned directorySSPCPP-276Resolved issue: SSPCPP-276Scott Cantor
- New endpoint to produce JSON data used by new Discovery ServiceSSPCPP-254Resolved issue: SSPCPP-254Scott Cantor
- Signed metadata uses whole doc reference instead of ID.SSPCPP-251Resolved issue: SSPCPP-251Scott Cantor
- Status handler could provide more informationSSPCPP-222Resolved issue: SSPCPP-222Scott Cantor
- Make resolvertest output more script "friendly"SSPCPP-211Resolved issue: SSPCPP-211Scott Cantor
- shibd ignores the -f optionSSPCPP-206Resolved issue: SSPCPP-206Scott Cantor
- shibd should properly daemonize itselfSSPCPP-204Resolved issue: SSPCPP-204Scott Cantor
- AuthType not being setSSPCPP-193Resolved issue: SSPCPP-193Scott Cantor
- Metadata generation duplicates indexes when multiple base URLs are supplied.SSPCPP-184Resolved issue: SSPCPP-184Scott Cantor
- keygen script under Windows doesn't obey -y paremeterSSPCPP-176Resolved issue: SSPCPP-176Scott Cantor
- NSAPI handler responses carry multiple content-type headersSSPCPP-175Resolved issue: SSPCPP-175Scott Cantor
- Allow Metadata Generator to populate a validUntil or cacheDuration attribute on the generated EntityDescriptorSSPCPP-160Resolved issue: SSPCPP-160Scott Cantor
- NSAPI module has numerous regressionsSSPCPP-139Resolved issue: SSPCPP-139Scott Cantor
- Invalid child element; metadata failure for non-standard RoleDescriptorSSPCPP-126Resolved issue: SSPCPP-126Scott Cantor
- Use of key name to select TLS or signing cred for SOAP client breaks trust resolutionSSPCPP-113Resolved issue: SSPCPP-113Scott Cantor
- Integrate a Memcache backed StorageService pluginSSPCPP-94Resolved issue: SSPCPP-94Scott Cantor
- Assertion not exported (again?)SSPCPP-66Resolved issue: SSPCPP-66Scott Cantor
- Pretty Printing for Unsigned XML generated by automatic HandlerSSPCPP-55Resolved issue: SSPCPP-55Scott Cantor
- V2 shibd daemon should not fprintf to stderr in normal circumstancesSSPCPP-38Resolved issue: SSPCPP-38Scott Cantor
- shibd init script should set selinux context on pid fileSSPCPP-35Resolved issue: SSPCPP-35Scott Cantor
- siterefresh is not proxy awareSSPCPP-30Resolved issue: SSPCPP-30Scott Cantor
50 of 52
orphaned DiscoveryFeed cache files accumulate
Fixed
Basics
Technical
Logistics
Basics
Technical
Logistics
Description
Environment
None
Attachments
1
Created March 21, 2017 at 6:29 PM
Updated June 22, 2021 at 9:06 PM
Resolved April 3, 2018 at 5:07 PM
Activity
Scott Cantor April 3, 2018 at 5:07 PM
Scott Cantor
April 3, 2018 at 5:07 PM
That was a whole lot of work for a one character fix. Fun times.
Scott Cantor April 3, 2018 at 5:00 PM
Scott Cantor
April 3, 2018 at 5:00 PM
Both of these bugs were regressions caused by the "fix" that corrected the problem with overlapping applications stepping on each others feeds, which was done in 2.5.6. So ever since that version, the original cleanup logic was broken and there's been more overhead. Fix is simple. If we have to do another security fix, I'll backport this.
This may still leave a few orphans, but nothing like it's doing now.
Scott Cantor April 3, 2018 at 4:48 PM
Scott Cantor
April 3, 2018 at 4:48 PM
On top of that, I believe the entire caching mechanism was broken and not actually tracking what it was meant to be tracking, so every client without an ETag was causing the feed to be re-written to the same filename over and over, and I believe no cleanup of any files was actually ever happening.
Scott Cantor April 3, 2018 at 1:20 PM
Scott Cantor
April 3, 2018 at 1:20 PM
The main bug here is that the shutdown logic isn't computing the filenames correctly, so anything not cleaned up during actual system usage gets left behind unintentionally.
Scott Cantor March 29, 2018 at 1:52 PM
Scott Cantor
March 29, 2018 at 1:52 PM
I would be curious what kind of performance you see if you turn the disk caching off. It's not really ever been tested much. The feed is still "cached" in the client and I don't know how much extra overhead there is from serving it directly from shibd memory and out to the client, though I imagine at these sizes it could be noticeable.
The current SP code in the file DiscoveryFeed.cpp includes this comment:
// Remove any files unused for more than a couple of minutes.
// Anything left will be orphaned, but that shouldn't happen too often.
We are finding with our 2.6.0 deployments (both CentOS 7 and Debian 8) that enough of the cache files are orphaned each day to be a nuisance. The number of orphans appears to be correlated with the load on the system. The busiest service accumulates about 10 orphan files a day. Each one is 1.5 MB in size (due to eduGAIN, these are widely federated SPs), so about 15 MB of orphaned files a day.
We are requesting an enhancement that would reap the orphan files more robustly.
A simple cron job to remove them nightly solves the issue so this is not a high priority, but it is a nuisance for an otherwise "well behaved" system.