Last modified: 2014-04-25 11:14:23 UTC
cp301[34] seem to be getting mobile traffic [1] since 2014-04-18, and are enabled in pybal [2]. But they are not listed in puppets manifests/role/cache.pp, so their requests are not included in udp2log's mobile-sampled-100 stream [3]. They also do not show up in udp2log's zero stream. Kafkatee streams are not affected. We see cp301[34] requests there. [1] ___________________________________________________________ qchris@stat1002 // 0 // 22:29:26 cwd: ~ zgrep '^cp3013' /a/squid/archive/sampled/sampled-1000.tsv.log-20140419.gz | head -n 1 | cut -f 1,3 cp3013.esams.wikimedia.org 2014-04-18T19:03:07 ___________________________________________________________ qchris@stat1002 // 0 // 22:29:49 cwd: ~ zgrep '^cp3014' /a/squid/archive/sampled/sampled-1000.tsv.log-20140419.gz | head -n 1 | cut -f 1,3 cp3014.esams.wikimedia.org 2014-04-18T19:03:04 [2] http://noc.wikimedia.org/pybal/esams/mobile [3] ___________________________________________________________ qchris@stat1002 // 0 // 22:30:07 cwd: ~ zgrep -c '^cp301[34]' /a/squid/archive/mobile/mobile-sampled-100.tsv.log-20140419.gz 0
Prioritization and scheduling of this bug is tracked on Mingle card https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/cards/1559
Change 127456 had a related patch set uploaded by QChris: Add cp301[34] to mobile caches https://gerrit.wikimedia.org/r/127456
(In reply to christian from comment #0) > They also do not show up in udp2log's zero stream. Not sure where I had my eyes ... cp301[34] requests do show up in zero files. They are only missing from mobile-sampled-100 stream.
Thanks Christian -- need anything else on this? -Toby
(In reply to Toby Negrin from comment #4) > Thanks Christian -- need anything else on this? I do not think we need anything else here. Ottomata said that the change looks ok. Waiting for BBlack to chime in on the change.
This bug is a side-effect of bug 64240, and cp301[34] currently acting as frontend caches only. BBlack said that he'll start add cp301[34] as backends as well, but connectivity issues are currently preventing that (see RT #6360). The plan is to add 301[34] separately on different days to lessen the impact on cache hits. Probably, one day per machine. This process “should” start today. BBlack is actively working on it. Plan looks good to me. But that also means that it'll probably take a few more days until the issue is fully resolved.
Ops changed plans, and they removed cp301[34] from the set of frontend varnishes again. They'll redo the machines differently, and will not turn them on until they are ready as backends as well. I checked pybal, an cp301[34] are gone again. I'll check mobile-sampled-100 tomorrow (when the new tsvs are available), if it is again as expected.
Checked mobile-sampled-100 from udp2log against previous weeks, and also against kafkatee files and zero files. cp301[34] are gone again for relevant requests, and number of lines is now within expectations.
Change 127456 abandoned by BBlack: Add cp301[34] to mobile caches Reason: These have been added now (separately, and as backends only for now until lvs config issues are sorted out) and should get analytics properly https://gerrit.wikimedia.org/r/127456
