Last modified: 2010-10-08 17:01:47 UTC
Currently we generate stats on locke to one landingpage or banner impressions file. Then someone has to rsync it for long term archive storage on dataset. After, we have to run the appropriate mining script and ship the fundraiser data. Lets automate this. We'll need to setup a regular job that pulls a specific set of log files from locke to dataset1. Then a cron job can kick off at a specific point in time (say 30min post fundraiser) that will email the fundraisers with a csv version of the data.
The 2step proces is now well automated thanks to Nimish. Documentation to come next.
There is now a cron running on locke which has user file_mover automatically compress the data, send it to dataset1, run analysis on it, and email the CSV to tfinc@wikimedia at 55 min past the hour.