Part of our server move was prompted by the ever-increasing indexing of our many, many activity feeds.
In addition to the overall history feed on the front page, each vocabulary has a history feed:
Metadata Registry Change History for Element Set: ‘MARC 21 Elements 00X’
Each element or concept also has its own history feed:
Metadata Registry Change History for Element: ‘Biography of Books’ in Element Set: ‘MARC 21 Elements 00X’
Each individual statement has a history feed (yes we are indeed as crazy as you may suspect we are at this point):
Metadata Registry Change History for Property: ‘label’ of Element: ‘Biography of Books’ in Element Set: ‘MARC 21 Elements 00X’
We believe that it’s important for vocabulary management systems to track changes (what, who, when) right down to the statement level and while we have no idea if there’s any utility to that fine-grained a history feed, we do track it so it seemed reasonable to offer a feed, even at that level, in RSS1 (rdf), RSS2, and atom no less.
This means we have thousands of feeds and these are being parsed heavily by various search engines, much more so than the resource changes they represent, and this generates a lot of bandwidth and processing overhead. We’re working on a number of different solutions, but the first one we tried — implementing app-level managed caching — didn’t do the trick. and we spent quite a bit of time fiddling with different configuration options. And in tinkering with that we broke the feeds in a way that was very hard to track down, since they passed our integration tests, worked on our staging server, and then broke on the production server… sometimes.
So they’re fixed now, and we’re exploring other caching options.
