Logging user's movements

[prev] [thread] [next] [Date index for 2005/02/04]

Hello,

I'm curious how the "pros" would approach an interesting system design 
problem I'm facing. I'm building a system which keeps track of user's 
movements through a collection of information (for the sake of 
argument, a Wiki). For example, if John moves from the "dinosaur" page 
to the "bird" page, the system logs it -- but only once a day per 
connection between nodes per user. That is, if Jane then travels from 
"dinosaur" to "bird," it will log it, but if "John" travels moves back 
to "dinosaur" from "bird," it won't be logged. The result is a log of 
every unique connection made by every user that day.

The question is, how would you do this with the least amount of strain 
on the server?

Currently, I'm using Squid to switch between thttpd (for non-"Wiki" 
files) and mod_perl, with the metadata in MySQL, and the text data in 
flatfiles (don't worry, everything's write-once). The code I'm using to 
generate the "Wiki" pages is fairly fast as I'm testing it, but it's 
not clear (and impossible to test) how well it will scale as more nodes 
and users are added. As a defensive measure, I'm caching the HTML 
output of the mod_perl handler, but the cached files aren't being 
served by thttpd, because the handler still needs to register where 
people are going. So every time a page is requested, the handler looks 
and sees if this user has made this connection in the past 24 hours, if 
not log it, and then either serve the cached file or generate a new one 
(they go out of date sporadically).

My initial thoughts on how to improve the system were to relieve 
mod_perl of having to serve the files, and instead write a perl script 
that would run daily to analyze the day's thttpd log files, and then 
update the database. However, certain factors (including the need to 
store user data in cookies, which have to be checked against MySQL) 
make this impossible.

Am I on the right track with this?

- ben