Last modified: 2013-10-25 15:33:11 UTC
as of Oct 23: on any page in https://test2.wikipedia.org enter "main" in search box search box should show possible results including "Main page" but shows no search suggestions at all
On mw.org too.
Well crap :(
this is also causing a MobileFrontend test failure
Also see bug 56061 which has similar symptoms
(In reply to comment #4) > Also see bug 56061 which has similar symptoms No, it's not the same.
Also on enwikisource, which is also a Cirrus-as-primary wiki. This seems like a good candidate for an associated test with the fix :)
Greg, the test already exists: https://github.com/wikimedia/qa-browsertests/blob/master/features/search.feature#L4-L8 Jenkins runs the test twice a day. The test passes at en.wikipedia.beta.wmflabs.org: https://wmf.ci.cloudbees.com/view/s-e.w-l/job/browsertests-en.wikipedia.beta.wmflabs.org-linux-firefox/405/testReport/(root)/Search/Search_suggestions/ and fails at test2.wikipedia.org: https://wmf.ci.cloudbees.com/view/s-t2.w/job/browsertests-test2.wikipedia.org-linux-firefox/649/testReport/(root)/Search/Search_suggestions/
Elasticsearch state makes the situation unfortunately complex. I test all changes locally with both "new" and "old" state but sometimes things slip through the tracks. I'm learning, I suppose you could say. Any way, we've already updated the elasticsearch state for test2 so I can't guess what is up. I'll be back with my computer in about twelve hours. I understand if this or one of the other issues that came up yesterday needs to be fixed right now. If so, switch back to lucenesearch by removing the wiki from CirrusSearch.dblist and add it to initializesrttings.php as CirrusSearch as alternate.
(In reply to comment #6) > Also on enwikisource, which is also a Cirrus-as-primary wiki. > > This seems like a good candidate for an associated test with the fix :) A little more detail: The test has existed for a long time, it has found bugs in search a number of times. In this particular instance, the bug was reported within hours of it being introduced. It's a robust test. MobileFrontend also has a browser test for search suggestions, and Michelle had notified the mobiletech-l folks of the failure, but I had already reported the bug and Chad had commented just minutes before Michelle sent that email.
The only reason this isn't fixed now is that box chad and I were at conferences yesterday. I'm still there today.
(In reply to comment #9) > (In reply to comment #6) > > Also on enwikisource, which is also a Cirrus-as-primary wiki. > > > > This seems like a good candidate for an associated test with the fix :) > > A little more detail: > > The test has existed for a long time, it has found bugs in search a number of > times. Awesome. Figured that's why you reported it. > In this particular instance, the bug was reported within hours of it being > introduced. It's a robust test. I'd love for it to be found *before* merge, in our grand and glorious future where all tests live in the repo of the code being tested (and where the devs of the code see the tests failing during development (through Gerrit)). I meant to direct my comment towards Chad/Nik in that as the QA team obviously did it's job well here. (In reply to comment #10) > The only reason this isn't fixed now is that box chad and I were at > conferences > yesterday. I'm still there today. Chad: Since you're back today, can you give me a go/no-go decision before 11am pacific (MW Core deploy window) on whether we should switch back to lucene on affected wikis? Don't want this to persist much longer.
> I'd love for it to be found *before* merge, in our grand and glorious future > where all tests live in the repo of the code being tested (and where the devs > of the code see the tests failing during development (through Gerrit)). Well, CirrusSearch is sort of a unique situation, and having test2 answers that situation exactly. > Chad: Since you're back today, can you give me a go/no-go decision before > 11am > pacific (MW Core deploy window) on whether we should switch back to lucene on > affected wikis? Don't want this to persist much longer. I'd vote for keeping CirrusSearch if we can, I don't think the lack of search suggestions in the short term is a blocker.
test2 should be fixed now. I'm going to fix all of them now, might just take me a few hours.
(In reply to comment #12) > > I'd love for it to be found *before* merge, in our grand and glorious future > > where all tests live in the repo of the code being tested (and where the devs > > of the code see the tests failing during development (through Gerrit)). > > Well, CirrusSearch is sort of a unique situation, and having test2 answers > that > situation exactly. Not the pre-commit tests question, though.... which was my point ;) (And yes, I know we don't currently have the infra to run Cirrus tests like this quickly per patchset, but we should strive to get there, and this was a prod to do so.) > > Chad: Since you're back today, can you give me a go/no-go decision before > > 11am > > pacific (MW Core deploy window) on whether we should switch back to lucene on > > affected wikis? Don't want this to persist much longer. > > I'd vote for keeping CirrusSearch if we can, I don't think the lack of search > suggestions in the short term is a blocker. Yep, Chad just confirmed on IRC that after the rebuild he's running right now all should be fine. Everyone: let the bug know if in an hour you still see this bad behavior anywhere.
mw.org rebuilt and working fine.
Following wikis rebuilt: aawiki, aawikibooks, aawiktionary, abwiktionary, advisorywiki, akwikibooks, akwiktionary, alswikibooks, alswikiquote, alswiktionary, amwikiquote, angwikiquote, angwikisource, astwikibooks, astwikiquote, aswikibooks, aswiktionary, avwiktionary, aywikibooks, bawikibooks, bhwiktionary, biwikibooks, biwiktionary, bmwikibooks, bmwikiquote, bmwiktionary, bowikibooks, bowiktionary, chowiki, chwikibooks, chwiktionary, cowikibooks, cowikiquote, crwikiquote, crwiktionary, dzwiktionary, foundationwiki, gawikibooks, gawikiquote, gnwikibooks, gotwikibooks, guwikibooks, howiki, htwikisource, huwikinews, hzwiki, iiwiki, ikwiktionary, itwiktionary, kjwiki, kkwikiquote, knwikibooks, krwiki, krwikiquote, kswikibooks, kswikiquote, kwwikiquote, lbwikibooks, lbwikiquote, lnwikibooks, loginwiki, lvwikibooks, mhwiki, mhwiktionary, miwikibooks, mnwikibooks, mowiki, mowiktionary, muswiki, mywikibooks, nahwikibooks, nawikibooks, nawikiquote, ndswikibooks, ndswikiquote, ngwiki, nlwikinews, nzwikimedia, pa_uswikimedia, piwiktionary, pswikibooks, qualitywiki, quwikibooks, quwikiquote, rmwikibooks, rmwiktionary, rnwiktionary, scwiktionary, sdwikinews, sewikibooks, simplewikibooks, simplewikiquote, snwiktionary, strategywiki, suwikibooks, swwikibooks, tenwiki, testwiki, testwikidatawiki, thwikinews, tkwikibooks, tkwikiquote, towiktionary, ttwikiquote, twwiktionary, ugwikibooks, ugwikiquote, usabilitywiki, uzwikibooks, vewikimedia, vowikibooks, vowikiquote, wawikibooks, wikimania2005wiki, wikimania2006wiki, wikimania2007wiki, wikimania2008wiki, wikimania2009wiki, wikimania2010wiki, wikimania2011wiki, wikimania2012wiki, xhwikibooks, xhwiktionary, yowikibooks, yowiktionary, zawikibooks, zawikiquote, zawiktionary, zh_min_nanwikibooks, zh_min_nanwikiquote, zuwikibooks Only two I haven't rebuilt: enwikisource, cawiki. They're bigger and I wanted to hold off until we saw if there's any breakage.
And rebuilt mediawikiwiki, testwiki, test2wiki, testwikidatawiki and loginwiki one more time for good measure.
success
And enwikisource, because why not ;-)
Verified working on test2wiki. As far as catching this with any automated testing either before merge or after, the state makes it complicated. We'd have to do something like: 1. Check out the newest code. 2. Rebuild the search index with the new code. 3. Run tests and collect results. 4. Check out the code that we used to build all the search indexes in production. 5. Rebuild the search index with that code. 6. Check out the newest code again. 7. Rerun the tests and collect failures. That all assumes that we built all the search indexes in production with the same code. That is true right now because we rebuilt everything but I don't doubt it'll stay true. Not all pushes of CirrusSearch require an index rebuild. And not all pushes require a full index rebuild - some just require a half rebuild.... State sucks.
(In reply to comment #20) ...Not all pushes of CirrusSearch require an index > rebuild. And not all pushes require a full index rebuild - some just > require a > half rebuild.... > > State sucks. Yep. But test2wiki rocks (because we found the bug there), and beta labs too. Maybe we could use the test environments even more effectively?
Added a bug to set regression tests up in gerrit so it can vote properly and linked it here: Bug 56166
