Wikimedia Bugzilla is closed!

Change 139775 had a related patch set uploaded by QChris:
Fix log target for mobile apps data job

https://gerrit.wikimedia.org/r/139775

Change 139775 merged by Springle:
Fix log target for mobile apps data job

https://gerrit.wikimedia.org/r/139775

Change 139785 had a related patch set uploaded by QChris:
Ensure log file for mobile data job exists

https://gerrit.wikimedia.org/r/139785

(In reply to Dan Andreescu from comment #0)
> The cron job that updates the mobile dashboards has not been working since
> June 2nd.  We've enabled logging with
> https://gerrit.wikimedia.org/r/#/c/138884/ but we don't see a log file yet.

I think, those are actually two separate issues:
* Cron job not working, and
* Seeing no log file.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Cron job not working:

For me, the relevant command bails out with:
  [...]
  Generating Uploads cancelled by user (per day)
  Traceback (most recent call last):
    File "/srv/limn-mobile-data/generate.py", line 321, in <module>
      dg.execute()
    File "/srv/limn-mobile-data/generate.py", line 171, in execute
      ok = self.generate_graph_full(key, value)
    File "/srv/limn-mobile-data/generate.py", line 285, in generate_graph_full
      self.save_graph_as_csv(key, headers, rows)
    File "/srv/limn-mobile-data/generate.py", line 303, in save_graph_as_csv
      with open(csv_filename, 'wb') as csv_file:
  IOError: [Errno 13] Permission denied: '/a/limn-public-data/mobile/datafiles/cancelled-uploads.csv'

As that file can only be writted by 610:wikidev [1], but the job being run as
stats [2] (without group "wikidev") the error looks plausible. Owner 610
is bug 64749.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Seeing no log file:

The patch that added logging for this job was broken, and broke puppet
on stat1003 [3]. Broken puppet should have been fixed by

  https://gerrit.wikimedia.org/r/139775

(see comment 1 and comment 2). Thanks springle for the prompt merge.
Puppet is ok again on stat1003.

But it seems that the log generation will fail regardless, due to
log file creation happening on the wrong end. The corresponding patch is

  https://gerrit.wikimedia.org/r/#/c/139785/

(for which I did not yet escalate merging, as it does not seem to affect
this or other services ... just logging. So it can sit in the queue.)



[1]
_________________________________________________________________
qchris@stat1003 // jobs: 0 // time: 06:26:20 // exit code: 0
cwd: ~
ll /a/limn-public-data/mobile/datafiles/cancelled-uploads.csv
-rw-rw-r-- 1 610 wikidev 7956 Jun  2 18:07 /a/limn-public-data/mobile/datafiles/cancelled-uploads.csv



[2] https://git.wikimedia.org/blob/operations%2Fpuppet/79e00984efb6e23cfc59e9fab13be14223baab7a/manifests%2Fmisc%2Fstatistics.pp#L696



[3] For example

  [05:21:18] <icinga-wm>	 PROBLEM - Puppet freshness on stat1003 is CRITICAL: Last successful Puppet run was Fri 13 Jun 2014 20:03:25 UTC

in http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-operations/20140616.txt

Change 139785 merged by Ottomata:
Ensure log file for mobile data job exists

https://gerrit.wikimedia.org/r/139785

Jobs are running again.
Logs get written.
Charts look good [1].
Resolved.



[1] All the graphs of the 'Edits daily' tab [2] look wrong. The graphs
drop for 2014-06-12 and die off on 2014-06-13. But that's not a script
error, but it closely matches EventLogging's upgrade for the MobileWebEditing
schema from revision 7675117 to revision 8599025. To verify that, run

  select timestamp, count(*) from MobileWebEditing_7675117 group by left(timestamp,8) order by timestamp desc limit 20;
  select timestamp, count(*) from MobileWebEditing_8599025 group by left(timestamp,8) order by timestamp asc limit 20;

against the EventLogging database. Switching schemata is offered in

  https://gerrit.wikimedia.org/r/#/c/139889/

but I am defering to mobile to decide, when they want to switch to the new
schema.

[2] http://mobile-reportcard.wmflabs.org/#edits_daily-graphs-tab