In the three years since I created it, my Flickr Set Manager has  turned out to be amazingly popular, with one of the most-liked features being the automatic updating of sets. Of course, the problem with being popular, is that there are then an awful lot of sets to update. And I do mean an awful lot.

I am sometimes asked if sets can be automatically updated more often than they currently are – perhaps every hour, or if I can arrange for them to be updated at a specific time of day. The answer is very simple – no, that’s just not really possible. The script that keeps the sets updated runs continuously, and just about manages to keep up with the amount of work it currently has to do.

There are a couple of things that need to be considered here. The first is the processing power required to keep these sets up to date. Yesterday, I moved dopiaza.org over to a new server that has plenty of spare capacity – that should help keep things running smoothly for the forseeable future. The second thing that needs consideration is the Set Manager’s use of the Flickr API.

For some time now, I have been stretching the limits of my Flickr API key somewhat – Flickr allow free non-commercial use of the API, but they do ask you to limit calls to the API to an acceptable rate. As the popularity of the set manager has increased over time, so has its usage of the Flickr API, and the time has come to restructure things a little.

I have spent some time recently rewriting large chunks of the Set Manager, and the first set of changes were rolled out yesterday as part of the site move. This particular batch of changes are all about making the application more Flickr-friendly.

• The set generation code has been extensively rewritten and is now much more efficient – particularly for random sets.
• There are quite a few sets in the system that are still marked as “automatically regenerate” but are actually now redundant (for example, the searches used to construct them no longer ever return any matching photos). With immediate effect, automatically generated sets that continually fail to find matching photos will have automatic regeneration disabled.
• The rebuilding of sets is now throttled at a much lower level than previously. This means that it may now be longer than a single day before sets are refreshed. I will be monitoring this interval, and continue to fine tune things. The goal is still to have sets updated once per day, but there may be a short hiatus whilst the new code settles in.

These ‘behind-the-scenes’ changes are largely invisible to the user, but are the precursor to something much more interesting. There is a whole new Set Manager in development, with a shiny new interface. There’s still some work to do, but I’m hoping we’ll be ready for launch this side of the new year. Here are a few screenshots as a sneak preview: