Wikimedia Bugzilla is closed!

Hi there. You probably want to e-mail wikitech-l about this: <https://lists.wikimedia.org/mailman/listinfo/wikitech-l>. Using Bugzilla as a forum for discussion (or more accurately a forum for a request for determination) doesn't really work well. :-)

(Or maybe you should start an RFC at mediawiki.org: <https://www.mediawiki.org/wiki/Requests_for_comment>. That's probably not necessary here, though.)

Hi MZMcBride, could you forward this to the mailing list? I always get confused with the mail flood so I only joined some -announce lists.

(In reply to comment #2)
> I always get confused
> with the mail flood so I only joined some -announce lists.

You can join lists with mail delivery disabled. Or you could leave delivery on only for a short period. (also, you could request that people CC you on replies)

-Jeremy

(In reply to comment #2)
> Hi MZMcBride, could you forward this to the mailing list? I always get confused
> with the mail flood so I only joined some -announce lists.

Sure, done: <http://lists.wikimedia.org/pipermail/wikitech-l/2012-September/063150.html>.

(In reply to comment #3)
> (In reply to comment #2)
>> I always get confused
>> with the mail flood so I only joined some -announce lists.
> 
> You can join lists with mail delivery disabled. Or you could leave delivery on
> only for a short period. (also, you could request that people CC you on
> replies)

Mailing lists suck. They're a crutch needed primarily because MediaWiki lacks a proper discussion system. If someone doesn't feel comfortable using them, that's perfectly acceptable. Though of course simply listing workarounds to the noise problem is fine.

Currently the additional user prefs are removed the next time the user visits Special:Preferences, see bug 37187.

So the ability to set any option was removed with I98df55f2. Is this just collateral damage from fixing the security bug, or should this be considered binding? I already managed to create a gadget settings manager on pl.wikipedia that used this feature as a fallback to cookies.

I talked to Chris Steipp on IRC. Summary of our discussion (posted with permission):

Bartosz Dziewoński:
Your action=options API security/validation patch, I98df55f2, also removed the ability to set arbitrary options - is this just collateral damage from fixing the security bug, or should this be considered binding?

Chris Steipp:
I think that's something that needs to be determined by the dev community. We needed to fix setting arbitrary content, but if arbitrary options that don't cause an xss are wanted, then we can allow those.

Bartosz Dziewoński:
I already managed to write a gadget on pl.wikipedia that used this to manage settings of other gadgets, as a fallback to cookies - https://pl.wikipedia.org/wiki/MediaWiki:Gadget-gConfig.js (it also has a .css file).

Chris Steipp:
Cool. So if your option names and values are alphanumeric, then it shouldn't be a problem to support them. If they're failing, then that should be a bug we fix. The issue comes in if you want users to set HTML into the values, which it looks like might be the case? The string values look like they would accept HTML in your gadget (from my quick readthrough).

Bartosz Dziewoński:
Yes, they accept anything, but, well, to me it's obvious that you should escape any values before using them. Currently only two gadgets use this manager: one doesn't use the string options at all, only boolean ones; the other uses a string option, but places the value into document.title, so, in this case, HTML-escaping the value could actually break the display, unless the gadget unescapes it again.

The gadgets are:
* gadget A, used to fix disambiguation links, uses boolean options only: https://pl.wikipedia.org/wiki/MediaWiki:Gadget-disFixer.js
* gadget B, hides the orange new message bar and turns it into a small icon used instead of the "My talk" link, has an option to add a user-defined string to the beginning of document.title: https://pl.wikipedia.org/wiki/MediaWiki:Gadget-GolabPocztowy.js
(The code for those two gadgets is not pretty, but the first one I wrote back in 2009 when I was young and stupid, and it received little cleanup since then ("if it ain't broke, don't fix it"), and the other was written primarily by a non-coder. They both work perfectly well, though, and are used by many wikipedians.)

Chris Steipp:
Unfortunately, not everyone thought it was obvious :) Basically, if everyone in the community agrees these strings can contain HTML, and everyone needs to escape them before using, then everyone can work under that assumption. If not, then we'll probably need to work out how to escape them when they're stored, and then when you want to store HTML, we'll have to escape / unescape it.

Bartosz Dziewoński:
We don't know in what format they are going to be output, so how can we escape them? With URL-encoding, SQL-escaping, HTML-escaping? Or, well, CSS-escaping, in this case? You might be right, though, that it might make sense.

To expand on my final words: it might make sense to HTML-escape the contents of unknown preferences by default, to protect the sloppy coders, and document that the preferences *must not* be escaped before saving, and *must* be unescaped after reading if you want to use them outside of HTML.

This wouldn't be pretty (in fact, it would be pretty damn ugly), but it would do the job, allowing us to just turn setting of arbitrary preferences back on again and dealing with most security concerns (as those preferences cannot conceivably be used in SQL commands, and the impact of unsanitized use in URLs seems very low to me). Can gadgets use this anywhere else where it should be sanitised?

Restricting setting of potential MediaWiki options does make sense. The question is whether we want to worry about the case where someone sets an option in version X and then MediaWiki starts using that option for version X+1. There's also the case to consider that we might want to continue having Special:Preferences remove obsolete options. Both of these could be solved by only enabling setting of arbitrary preferences for some prefix (e.g. "ujs-").

As for protecting user script authors from themselves, automatically escaping things would IMO be entirely the wrong choice for the same reason PHP's "magic quotes" was such a bad idea: sometimes things don't get unescaped when they should have, and thanks to that sometimes they wind up being stored double-escaped, and sometimes they do get unescaped when they shouldn't be because people are so used to having to unescape everything. If we do anything, we should just reject entirely whichever characters you might consider escaping. I suppose preventing storage of '<', '>', '"', and "'" in these arbitrary preferences *would* help stop what would probably be the most common kinds of carelessness leading to XSS. But I doubt we can blacklist enough to really prevent everything, and we certainly can't prevent someone from base64-decoding a preference value and then using it in an unsafe manner, so there's a point at which we just have to rely on user script authors to not be stupid.

(In reply to comment #4)
> Sure, done:
> <http://lists.wikimedia.org/pipermail/wikitech-l/2012-September/063150.html>.
Thanks a lot MZMcBride.

(In reply to comment #9)
> As for protecting user script authors from themselves
you/we should define 
1) a prefix system that won't be used in any MW-version in future (I second "ujs-"; MediaWiki could be "configured" to only accept arbitrary values when this prefix was used)
2) a storage quota to avoid illegal use/ abuse
3) a transfer quota to avoid illegal use/ abuse
4) a regexp I can test my keys or values against; if it fails, the key or value can't be used and that's it.

Fancy escaping of input in order to protect me from myself is not required. Thank you.

I'm generally in favor of allowing unused preferences to be set, as long as Chris is on board. Brad makes a good point about MW starting to use a new preference key that was previously used by users, we'd need to make sure validation can't be bypassed that way.

Chris, thoughts?

I like the prefix suggestion from Brad. I think it's a useful tool for user-script authors to have a place to store the prefs, and the prefix makes sure that they don't accidentally start overwriting something important. And that's really where we have to be careful, is when MediaWiki (or a popular gadget) is using a preference with some amount of escaping (or not escaping), and javascript is free to set it to arbitrary values.

I think refusing to save the characters <>&'"/ might be a good compromise. If a script author really wants to save html in a preference, they can base64 encode the value, and decode it when they use it, with the full knowledge that an attacker may have also set that value to a base64 encode version of their hostile html so they need to validate whatever they set when they show it back.

Ugh, to me the idea with removing the characters altogether doesn't sound good. If they are megically escaped, you can at least figure out what happens; if they magically disappear, the only thing you have left is to dig thru a mountain of not-exactly-the-best docs.

JavaScript doesn't even have Base64 encoding functions. (Yes, some browsers provide window.btoa / window.atob, but it's a non-standard extension.) Unless we have them as a RL module (which we don't as far as I know), this is going to encourage some ugly half-assed reimplementations and homemade escaping solutions.

Or, really, we could just allow getting and setting them without obstacles like that.

It's not like you can't do anything to a user anyway if you can inject some JavaScript into his environment; the security implications of non-sanitised preferences which would be used *only* in other JavaScript code seem extremely minuscule to me, compared to the huge vulnerability ;) that user-scripts and gadgets are.

(I do, of course, agree with enforcing prefixed keys - but I'd use 'userjs-' as the prefix, as 'ujs-' isn't too obvious - I couldn't figure out what is it supposed to stand for for a while.)

(In reply to comment #13)
> Ugh, to me the idea with removing the characters altogether doesn't sound good.

You misunderstood. The proposal isn't to remove the characters, it's to return a warning or error and not save the submitted value at all if the characters are included.

> JavaScript doesn't even have Base64 encoding functions. (Yes, some browsers
> provide window.btoa / window.atob, but it's a non-standard extension.) Unless
> we have them as a RL module (which we don't as far as I know), this is going to
> encourage some ugly half-assed reimplementations and homemade escaping
> solutions.

jQuery doesn't have base64 built in? Disappointing.

(In reply to comment #15)
> You misunderstood. The proposal isn't to remove the characters, it's to return
> a warning or error and not save the submitted value at all if the characters
> are included.

Ah, this makes more sense then, thanks. But it's going to make us the butt of jokes all over the Internet, like with these sites that don't allow "truncate", "select" or "drop" as usernames. :P

(In reply to comment #12)
> I think refusing to save the characters <>&'"/ might be a good compromise.

(In reply to comment #15)
> (In reply to comment #13)
> > Ugh, to me the idea with removing the characters altogether doesn't sound good.
> 
> You misunderstood. The proposal isn't to remove the characters, it's to return
> a warning or error and not save the submitted value at all if the characters
> are included.
> 

Why escape them? The values are exported to javascript as an object literal. The values them selves are javascript-escaped by the JSON encoder, which is then embedded in a script tag (which makes it HTML encoded safe). What kind of attack are we preventing by restricting the value? It can be any string, right?

Also:
* Escape them how, when and where?
* Escape them for what? HTML? JavaScript? CSS? URL?


(In reply to comment #12)
> [script] really wants to save html in a preference, they can base64 encode
> the value, and decode it when they use it
> > JavaScript doesn't even have Base64 encoding functions.
> jQuery doesn't have base64 built in? Disappointing.

base64 seems pointless per the previous paragraph. As for jQuery, jQuery is a library for abstracting DOM and Ajax interaction. It isn't an arbitrary collection of utilities (the few utilities it has are mostly undocumented for internal usage, not public APIs). Something like base64 encoding would never be a public API in jQuery. Completely out of its scope.

(In reply to comment #14)
> [I]'d use 'userjs-' as the prefix, as 'ujs-' isn't too obvious.

I'd recommend using a prefix like "gadget" instead since "user scripts" are something else ("user scripts" are non-modularised subpages of a User page).

But not "gadget-" itself since that prefix is already reserved for actually enabling the gadgets themselves.

As for whether we should support allowing arbitrary keys in user preferences storage, I think not. For the following reasons:

* It can easily conflict with core or extension preferences.

* There is no central registry. No rules, no validation, no labels, no documentation. It doesn't have a default-value system or any other system.

* It can easily conflict with other gadgets, site scripts and user scripts.

* This has been the case for years in the Special:Preferences backend, but never on purpose. The recent security fix shows that this was indeed an undocumented anomaly that that has been fixed accordingly. The point here being that not keeping this shouldn't break much since it isn't a widespread yet.

So what about the use case mentioned in comment 0? That is valid indeed and has been filed as bug 21897. In fact implementation has already began in Salvatore Ingala's branch of the Gadgets extension. It implements a small dedicated key/value storage for gadgets that want it. It has value type validation and will provide them in-context. 

Suggesting to:
* Close bug 37187 (special preferences removes unknown keys) as "wont fix", this is an intended lazy sanitation step.
* Close bug 40124 (this bug) as works for me.  The use case presented here is bug 21897, which is pending a dedicated implementation that will offer exactly what gadget authors are looking for and more.

(In reply to comment #18)
> (In reply to comment #14)
> > [I]'d use 'userjs-' as the prefix, as 'ujs-' isn't too obvious.
> 
> I'd recommend using a prefix like "gadget" instead since "user scripts" are
> something else ("user scripts" are non-modularised subpages of a User page).

Yes, and those might want to use this as well. Also, this would theoretically go into core, where there's no notion of 'gadgets'.


> As for whether we should support allowing arbitrary keys in user preferences
> storage, I think not. For the following reasons:
> 
> * It can easily conflict with core or extension preferences.

A documented prefix would prevent that.


> * There is no central registry. No rules, no validation, no labels, no
> documentation. It doesn't have a default-value system or any other system.

The default value is 'undefined'. The rest would be handled by each script itself.


> * It can easily conflict with other gadgets, site scripts and user scripts.

It seems obvious to me that scripts would each have to use a prefix as well.


> So what about the use case mentioned in comment 0? That is valid indeed and has
> been filed as bug 21897. In fact implementation has already began in Salvatore
> Ingala's branch of the Gadgets extension. It implements a small dedicated
> key/value storage for gadgets that want it. It has value type validation and
> will provide them in-context.

It's all very nice, but I was hoping for something that could be reimplemented within my lifetime.

Especially since I already implemented what you're describing myself on pl.wiki, basing it on cookies + the undocumented options API that worked at the time. (I decided to do it due to the glacial pace of changes in MW itself - I know WMF engineers are overworked.) I was hoping that it could be allowed to work again. (Linked above, it's called gConfig, implemented as a gadget[1].)

[1] https://pl.wikipedia.org/wiki/MediaWiki:Gadget-gConfig.js

(In reply to comment #18)
> 
> * This has been the case for years in the Special:Preferences backend, but
> never on purpose. The recent security fix shows that this was indeed an
> undocumented anomaly that that has been fixed accordingly.

That security fix also caused at least two different regressions, so I'd be a little wary of using things it did beyond fixing the editfont injection bug as precedent.


> I'd recommend using a prefix like "gadget" instead since "user scripts" are
> something else ("user scripts" are non-modularised subpages of a User page).
[...]
> So what about the use case mentioned in comment 0? That is valid indeed and has
> been filed as bug 21897. In fact implementation has already began in Salvatore
> Ingala's branch of the Gadgets extension. It implements a small dedicated
> key/value storage for gadgets that want it. It has value type validation and
> will provide them in-context. 

Oh, will we finally be able to make non-centrally-managed "gadgets" now, taking advantage of this feature? Or have some way to not have to wade through a gadgets list 200 pages long if all user scripts are converted to gadgets? If not, this is not particularly helpful.

(In reply to comment #19)
> It's all very nice, but I was hoping for something that could be reimplemented
> within my lifetime.
> 
> Especially since I already implemented what you're describing myself on
> pl.wiki, basing it on cookies + the undocumented options API that worked at the
> time. (I decided to do it due to the glacial pace of changes in MW itself - I
> know WMF engineers are overworked.) I was hoping that it could be allowed to
> work again. (Linked above, it's called gConfig, implemented as a gadget[1].)
> 
> [1] https://pl.wikipedia.org/wiki/MediaWiki:Gadget-gConfig.js


I know the feeling, I write and maintain many community gadgets and user scripts as well! But, with understanding and respect, this is somewhat unfair. You want to avoid implementing anything that is officially reviewed and supported (documented, reliable, maintained, not wiki/language local implement). So, you do it all yourself. Spend a lot of valuable time implementing something locally on your home wiki using undocumented features until it works. Then when it breaks (surprise!), you report the failure and need us to in(directly) fix your feature.

This is somewhat generalised for what I'm seeing elsewhere, the user options API in this case is just a simple example, but I'm sure you've seen it happen before (unfortunately).

Anyhow.. The community contribution workflow has been improved a lot. The perception that we (you, me, Wikimedians) like to share and complain about is getting dated and will only come to your own disadvantage. I recommend you take this opportunity to make something cool with support of the foundation (instead of on your own in a user script, which is a lot harder to maintain and developer). I'm sure you'll like working with us and the community will really appreciate you for it. I'd be happy to make sure you get the support you need from others or myself.

Either way, though this feature is really useful and accepted as something we support (bug 21897), it hasn't been done yet because of lack of maintenance of Gadgets extension (it was implemented in 2007 and been abandoned).

There are currently 2 major phases I want to get Gadgets through:
* Gadgets 2.0: Implement global gadget repository and ResourceLoader/localisation support.
* Gadgets 3.0: User repositories (replacing core "user scripts" with "user gadgets").

Somewhere in there is also room for "gadget options" (which Salvatore started with). However I haven't been able to get back at that because other projects have a higher priority right now.

(In reply to comment #20)
> (In reply to comment #18)
> > So what about the use case mentioned in comment 0? That is valid indeed and has
> > been filed as bug 21897. In fact implementation has already began in Salvatore
> > Ingala's branch of the Gadgets extension. It implements a small dedicated
> > key/value storage for gadgets that want it. It has value type validation and
> > will provide them in-context. 
> 
> Oh, will we finally be able to make non-centrally-managed "gadgets" now, taking
> advantage of this feature? Or have some way to not have to wade through a
> gadgets list 200 pages long if all user scripts are converted to gadgets? If
> not, this is not particularly helpful.

Gadget preferences have little to do with user gadgets.

User gadgets: No, converting a user script to a "User gadget" (with the model I have in mind of Gadgets 3.0) will not add anything to Special:Preferences. It will add it to a user-level repository. You can search trough repositories you have "linked" (e.g. the site-wide repo and any users you have linked. Not a list but a search interface)

Gadget preferences: No, these preferences would be on a per-gadget level. They wouldn't pollute the main Special:Preferences panel (the UI isn't final yet, but afaik Salvatore's proposal was a modal dialog triggered from an icon next to the gadget name on Special:Preferences)

(In reply to comment #22)
> 
> Gadget preferences have little to do with user gadgets.
> 
> User gadgets: No, converting a user script to a "User gadget" (with the model I
> have in mind of Gadgets 3.0) will not add anything to Special:Preferences. It
> will add it to a user-level repository. You can search trough repositories you
> have "linked" (e.g. the site-wide repo and any users you have linked. Not a
> list but a search interface)
> 
> Gadget preferences: No, these preferences would be on a per-gadget level. They
> wouldn't pollute the main Special:Preferences panel (the UI isn't final yet,
> but afaik Salvatore's proposal was a modal dialog triggered from an icon next
> to the gadget name on Special:Preferences)

So in other words, no, your gadget preferences will not really solve this bug.

(In reply to comment #21)
> You want to avoid implementing anything that is officially reviewed and
> supported (documented, reliable, maintained, not wiki/language local
> implement).

My solution is documented in two languages (the ones I speak well enough to do this, Polish and English). I believe it is pretty reliable (it kept working after the options API was "fixed" thanks to the safeguards I built in). It is maintained (but, I admit, currently only by me). I was going to suggest my implementation on other wikis as well, but it was broken before I got to it.

It isn't "officially reviewed", yes. I don't believe it would add much value, to be honest; it would just enforce some of MW's coding conventions which I intentionally didn't use.

> So, you do it all yourself. Spend a lot of valuable time
> implementing something locally on your home wiki using undocumented features
> until it works.

To be honest, I implemented it in one day, then spent an evening doing fixups, and then another evening when I changed some features after pl.wikipedia community discussion. I saved a lot of time by not waiting weeks for somebody to finally review it; I shared it with a few fellow pl.wikipedian coders, and they liked it.

Yes, it wasn't scrutinized as thoroughly as WMF-approved scripts, and it probablt isn't tip-top perfect. But it's good enough, it works, and I have yet to hear a bug report about it. (I used it in one gadget after initial testing, then in a second, more widely-used one after no issues were found.)

> Then when it breaks (surprise!), you report the failure and
> need us to in(directly) fix your feature.

I could probably implement seamlessly saving the data in a user .js file in less time than it took me to write out all these elaborate messages here, as the code I wrote is pretty robust and would support it with little structural changes. I just wanted to solve things the right way before I go offroad again. 

Nor do I need anyone to "fix" it; I am able to implement a fix to the options API myself, but I refrained from doing it until the discussion was over (or at least stabilizing). I will, of course, need someone to approve my patch. (Honestly, that patch would amount to removing one line of code containing a certain "continue" statement, as far as I can see right now, unless there's more issues lurking in the preferences.)


> Anyhow.. The community contribution workflow has been improved a lot. The
> perception that we (you, me, Wikimedians) like to share and complain about is
> getting dated and will only come to your own disadvantage. I recommend you take
> this opportunity to make something cool with support of the foundation (instead
> of on your own in a user script, which is a lot harder to maintain and
> developer). I'm sure you'll like working with us and the community will really
> appreciate you for it. I'd be happy to make sure you get the support you need
> from others or myself.

As I said before, when you want to implement things and see them live in your lifetime, code review the way WMF / MediaWiki is practising it simply gets in the way. It would take months to get someone to review such a large chunk of new code, and years until it got deployed. (I don't think I'm exaggerating.)

I appreciate constructive reviews, and opportunities to improve my skills; I don't appreciate having to wait weeks and months for them. Sorry, but that's the situation as I see it.


> Either way, though this feature is really useful and accepted as something we
> support (bug 21897), it hasn't been done yet because of lack of maintenance of
> Gadgets extension (it was implemented in 2007 and been abandoned).
> 
> There are currently 2 major phases I want to get Gadgets through:
> * Gadgets 2.0: Implement global gadget repository and
> ResourceLoader/localisation support.
> * Gadgets 3.0: User repositories (replacing core "user scripts" with "user
> gadgets").
> 
> Somewhere in there is also room for "gadget options" (which Salvatore started
> with). However I haven't been able to get back at that because other projects
> have a higher priority right now.

Gadgets 2.0 doesn't seem much more real the MediaWiki 2.0 to me.

No, but really: <https://www.mediawiki.org/wiki/Requests_for_comment>. All of you.

In reply to comment #24)
>I could probably implement seamlessly saving the data in a user .js file in
>less time than it took me to write out all these elaborate messages here

Simply do it. jquery.json is a core module ( http://www.mediawiki.org/wiki/RL/DM#jQuery.json ) for stringifying your data; if you want something better than a cookie you can use jquery.jStorage

A cookie is sent to the server with each request within the same domain; it has a very limited storage quota so storing strings can be difficult.

If you can, submit a patch and ask someone you trust to review before the next round of the personal carousel is driven by the WMF or more fancy ideas like disallowing quotation marks (so you can't save JSON strings) pop up here.

(In reply to comment #25)
>RFC
Reading Krinkle's comment, it sounds like WONTFIX or will be fixed when some extension is ready in the future. So an RFC won't change this, I guess.

I submitted changeset I5f9ba5b0, which implements this according to my vision. Reviews welcome.

(In reply to comment #26)
> Reading Krinkle's comment, it sounds like WONTFIX or will be fixed when some
> extension is ready in the future. So an RFC won't change this, I guess.

Indeed. This bug is imho either wontfix of duplicate of bug 21897.

Gadgets 2.0 is being implemented and the preferences work has also seen a lot of progress. It isn't an idea, it is a branch with a lot of work already put into it.

I doesn't make sense to implement this in MediaWiki core because nobody will be maintaining it, and it will not be used by core itself or any extension. It would be a temporary solution that will only solve a very small part of the problem (what about backend validation and sanitisation, front-end user interface to set, read and manage the settings, registry/namespace to avoid conflicts with other gadgets etc.)
 - you can't do this in mediawiki core without facing the fact that we would be expecting that gadget authors do all the above themselves - for every gadget. That isn't impossible, but it will most certainly be inconsistent and with different grades of security and usability. Also, even if they do it all perfectly, it is a waste of effort. They'd be implementing something fairly complex, all in their own way, only for a few months after it would become obsolete.

Instead, keep the current situation: use cookies or localStorage or whatever it is you're using. Native preferences don't exist yet, too bad. They're on their way. What doesn't exist can't be used, but rushing the track with another temporary solution that only solves part of the probelm - side-by-side - does not solve the problem.

You're more than welcome to help out with the implementation in the Gadgets extension that will provide all of this natively, but avoid more temporary solutions that we can't support that we know will lead to unstable security/usability.

(In reply to comment #28)
> Indeed. This bug is imho either wontfix of duplicate of bug 21897.

I have already said it, but I will repeat – this is intended not only for gadgets, but also (or even primarily) for user scripts, as there isn't even a notion of gadgets in core MediaWiki. 


> Gadgets 2.0 is being implemented and the preferences work has also seen a lot
> of progress. It isn't an idea, it is a branch with a lot of work already put
> into it.

Where? There isn't even a mention at https://www.mediawiki.org/wiki/Extension:Gadgets . Is it being tested somewhere? Is it actually runnable? Is it going to be deployed to Wikimedia wikis within, say, a month? A year?

A lot of work was already put into Perl 6. Doesn't mean it will happen anywhere soon.


> I doesn't make sense to implement this in MediaWiki core because nobody will
> be
> maintaining it, and it will not be used by core itself or any extension. It
> would be a temporary solution that will only solve a very small part of the
> problem (what about backend validation and sanitisation, front-end user
> interface to set, read and manage the settings, registry/namespace to avoid
> conflicts with other gadgets etc.)
>  - you can't do this in mediawiki core without facing the fact that we would
> be
> expecting that gadget authors do all the above themselves - for every gadget.

I have already said this, I will repeat for you: there is https://pl.wikipedia.org/wiki/MediaWiki:Gadget-gConfig.js , which implements all of what you've written above that is possible to implement client-side, and has documentation in English (in the code) and in Polish (at [[w:pl:WP:Narzędzia/gConfig]]). If you wish to use it in MW core, I can release it under the GNU GPL (in addition to the other licenses).

We can't do 'backend validation and sanitisation' for user JS. Nor can we do 'registry/namespace to avoid conflicts with other gadgets'. It's just not possible, unless you change the very basic idea here to no longer be user scripts.


> Instead, keep the current situation: use cookies or localStorage or whatever
> it
> is you're using. Native preferences don't exist yet, too bad. They're on
> their
> way. What doesn't exist can't be used, but rushing the track with another
> temporary solution that only solves part of the probelm - side-by-side - does
> not solve the problem.

Cookies and localStorage have the important issue that they're not transferred to other computers, and they can be pretty easily lost, which is unacceptable for user preferences. "Native" preferences don't have any of these flaws.


> You're more than welcome to help out with the implementation in the Gadgets
> extension that will provide all of this natively, but avoid more temporary
> solutions that we can't support that we know will lead to unstable
> security/usability.

The "unstable security" you're talking about here seems to be just fear mongering, unless you can provide examples of what you're talking about that can't be accomplished without the use of the preferences.

With all of this discussion, I'm a little lost on what the arguments are here. I can see the motivation behind wanting to be able to set arbitrary user options. What security vulnerabilities exactly are we worried about that we don't want to implement this? I've seen XSS thrown around, but I'm not sure how user option keys are viable XSS vectors.\

> I've seen XSS thrown around, but I'm not sure how
user option keys are viable XSS vectors.

They aren't. They could be if you could use a GET request to set the preferences, or if the options API didn't require the token, *and* if the scripts themselves didn't sanitize the options' values since they are user input, but you can't and it does and they should.

Apart from this, if you are able to set user prefs, you've already got a script running on somebody's browser, which is what XSS is. The entire user script and gadget system could be considered one huge gaping security hole, but I don't hear about anybody wanting to scrape that.

When I was implementing the change that introduces options module for API my original plan was to use it for storing gadget preferences. It can be seen in unit tests I wrote.

The only way to abuse this module is overloading database with tons of junk.

What I really don't like about the fix is skipping tests.

Gerrit change I5f9ba5b0 thoroughly reviewed and merged.

This means this is now possible, supported, and not a side-effect; I'll write up a quick note to wikitech-l with details. For now please refer to the commit summary and release notes.

(In reply to comment #33)
> Gerrit change I5f9ba5b0 thoroughly reviewed and merged.

\o/

Congrats! And thank you for all of your work on this bug. :-)

> This means this is now possible, supported, and not a side-effect; I'll write
> up a quick note to wikitech-l with details. For now please refer to the
> commit summary and release notes.

Sweet, sounds good. We'll eventually need docs on MediaWiki.org as well, I imagine.

(In reply to comment #0)
> * Is it possible to remove one particular pref without having to reset all?

This is currently not possible; I created bug 43959 for this.

(The wikitech note is coming, I've updated the docs at https://www.mediawiki.org/wiki/API:Options already.)

Also created bug 43960 for the same thing, but in the Special:Preferences interface.

I've sent a note to wikitech: http://lists.wikimedia.org/pipermail/wikitech-l/2013-January/065637.html

(Copying from Bug 43960 comment 4, since this place seems to be more appropriated) 

Just to be sure, does the MW API is only usable through JavaScript? If not,
maybe the "js" in "userjs" is not very appropriate...

True, this could be used by anything, but I'm sure that JavaScript user scripts will the the primary "consumer" of this, so I though that 'userjs' would be most obvous.

Feel free to propose a better name (open a bug possibly?), it can still be changed (the code is not in a release yet, nor even in a wmfN branch).

(In reply to comment #37)
> (Copying from Bug 43960 comment 4, since this place seems to be more
> appropriated) 
> 
> Just to be sure, does the MW API is only usable through JavaScript? If not,
> maybe the "js" in "userjs" is not very appropriate...

Sounds like a bikeshed :P

In practise, the main use case is for user js to store information tied to a specific user. Hence the name is probably appropriate. While other things can use the api (like bots), they probably have better options for storing things, i.e. locally.

"In practise, the main use case is for user js to store information tied to a
specific user."

Yes, but it certainly isn't the only use case. I've used this feature several times in extensions. Indeed, I didn't know about this prefix change until my extensions started breaking on en.wiki. It seems like this feature is useful for any case in which you need to store user options in a way that is more persistent than a cookie, but you don't want to use the prefs interface (either because it would be pref spam or you can't implement the specific interface you need via HTMLForm). While I agree 'userjs-' was a poor choice for a prefix, I wouldn't suggest changing it again.

Also, next time a non-backwards-compatible change is made to the API, please try to advertise it more widely. Thanks!

Actually, I guess it was the fix to bug 42202 that broke the ability to set custom user options, so nevermind about the non-backwards-compatible complaint :)

(In reply to comment #40)
> (either
> because it would be pref spam or you can't implement the specific interface
> you
> need via HTMLForm). While I agree 'userjs-' was a poor choice for a prefix, I
> wouldn't suggest changing it again.

Note that I18a5ffb5 created a way to register preferences that don't show up in Special:Preferences but can still be used via the API.

(In reply to comment #41)
> I guess it was the fix to bug 42202 that broke the ability to set
> custom user options
Indeed. But AFAIK this feature was nowhere documented.

BTW, thank you, Bartosz Dziewoński for fixing this.

(In reply to comment #42)
> Note that I18a5ffb5 created a way to register preferences that don't show up
> in
> Special:Preferences but can still be used via the API.

We already have preferences of type 'hidden'. What's the difference between this and type 'api'? And why is none of this documented anywhere?

The difference between 'hidden' and 'api' is that hidden can still be changed via the Preferences form, whereas api won't even be put on the form so it cannot be changed.

(In reply to comment #40)
> Also, next time a non-backwards-compatible change is made to the API, please
> try to advertise it more widely. Thanks!

I mailed wikitech about this (that is, the final fix, not the security-fix-which-broke-it). Where else should it have gone?

(In reply to comment #46)
> (In reply to comment #40)
> > Also, next time a non-backwards-compatible change is made to the API, please
> > try to advertise it more widely. Thanks!
> 
> I mailed wikitech about this (that is, the final fix, not the
> security-fix-which-broke-it). Where else should it have gone?

IMO you're just fine. This was not a breaking change for the API. Rather it unbroke behavior from a previous bugfix.