Last modified: 2008-12-21 12:27:09 UTC
I have a list of images gathered on the last month, via the API allimages generator. It should be ordered, and it is, however a few percentage of the images (quite noticeable on the full list) is wrongly order such as Ayutthaya_Elephant_Camp.JPG before Ayun_Halliday_by_David_Shankbone.jpg or Basketball_29082008666_(6).jpg followed by Basiliscus_vittatus-2.jpg The fault could be on three places: my program, the API or the slave. My program doesn't do any reaordering, the list is dumped in the same order as received from the connection. It can produce unordered lists if it is run, stopped and run again from a previous point. But it doesn't explain jumps as going from Auto_Racing_Yellow.svg to Image* (ImageAKMap-doton-Amukta.png, ImageAerienneRafRaf.JPG, ImageAlumbrados_del_Rio-2007_Medellin.jpg, ImageBrute.jpg, ImageBunnygirl_Cosplay2.jpg, ImageCBCdecryptie.jpg, ImageCBCencryptie.jpg, ImageCFBencryptie.jpg, ImageCapendu.PNG) and back to Auto_Racing_Yellow_Chevron.svg (continuing in alphabetical order). As the list is built from many API queries, there can be many servers serving it, and multiple race conditions. However, it happens to images uploaded many months ago, if not years. So it's not a lagged slave nor a new image uploaded between. And I don't know on any locale which would do such misorderings. I can provide the app/files, but they're huge, and not too useful.
(In reply to comment #0) > I have a list of images gathered on the last month, via the API allimages > generator. It should be ordered, and it is, however a few percentage of the > images (quite noticeable on the full list) is wrongly order such as > Ayutthaya_Elephant_Camp.JPG before Ayun_Halliday_by_David_Shankbone.jpg or > Basketball_29082008666_(6).jpg followed by Basiliscus_vittatus-2.jpg > > The fault could be on three places: my program, the API or the slave. > > My program doesn't do any reaordering, the list is dumped in the same order as > received from the connection. It can produce unordered lists if it is run, > stopped and run again from a previous point. But it doesn't explain jumps as > going from Auto_Racing_Yellow.svg to Image* (ImageAKMap-doton-Amukta.png, > ImageAerienneRafRaf.JPG, ImageAlumbrados_del_Rio-2007_Medellin.jpg, > ImageBrute.jpg, ImageBunnygirl_Cosplay2.jpg, ImageCBCdecryptie.jpg, > ImageCBCencryptie.jpg, ImageCFBencryptie.jpg, ImageCapendu.PNG) and back to > Auto_Racing_Yellow_Chevron.svg (continuing in alphabetical order). > If your program somehow messes up aicontinue values, such things could happen. The Auto-Image-Auto bug looks like some weird race condition or improper handling of parallel requests. > As the list is built from many API queries, there can be many servers serving > it, and multiple race conditions. However, it happens to images uploaded many > months ago, if not years. So it's not a lagged slave nor a new image uploaded > between. And I don't know on any locale which would do such misorderings. > Of course image creations/deletions don't explain huge jumps. > I can provide the app/files, but they're huge, and not too useful. > The only useful information you could provide is an example of an API request that returns a wrongly ordered list. If you manage to find one, please REOPEN this bug. A debugging hint: let your program log all its API requests (parameters, response, timestamp) to a separate log file. Then run it again, and if it orders your images wrongly again, examine the log to see what went wrong. Closing as WORKSFORME (because, well, it does).
(In reply to comment #1) > If your program somehow messes up aicontinue values, such things could happen. It's called gaifrom ;) I'm reading it from gaifrom attribute of <allimages> Indeed, if the API somehow returned the namespace on the query-continue, that could explain the first jumping, but not the jump back. Maybe unrelated, I suspect that some results are skipped due to the API providing malformed XML (bug 16106). > The Auto-Image-Auto bug looks like some weird race condition or improper > handling of parallel requests. There're no parallel requests. > Closing as WORKSFORME (because, well, it does). Race conditions, small memory corruptions or load-balancing slaves crazyness are not too reproductible. I'd also like having a failing URL and I am aware it is insufficient for fixing but hopefully someone would have an idea or will find this when facing the same problem.
(In reply to comment #2) > (In reply to comment #1) > > If your program somehow messes up aicontinue values, such things could happen. > > It's called gaifrom ;) > I'm reading it from gaifrom attribute of <allimages> > Indeed, if the API somehow returned the namespace on the query-continue, that > could explain the first jumping, but not the jump back. > It doesn't return the namespace, because all images are in the same namespace (File:). > Maybe unrelated, I suspect that some results are skipped due to the API > providing malformed XML (bug 16106). > If you're using XML, I guess that could be the problem. Have you tried using other formats? Instinctively, I'd say PHP serialized is best shielded from this kind of malformedness. > > The Auto-Image-Auto bug looks like some weird race condition or improper > > handling of parallel requests. > > There're no parallel requests. > > > > Closing as WORKSFORME (because, well, it does). > > Race conditions, small memory corruptions or load-balancing slaves crazyness > are not too reproductible. > I'd also like having a failing URL and I am aware it is insufficient for > fixing but hopefully someone would have an idea or will find this when facing > the same problem. > While having a failing URL would be very nice, even a log excerpt showing that the API is doing *something* wrong (like sorting stuff wrongly, setting the wrong query-continue values or malformed XML messing stuff up) would be good enough for me.
