Wikimedia Bugzilla is closed!

Which of them do you want exactly (some functions can be implemented by using other functions), and how EXACTLY is this bug different from bug 6455 other than that it doesn't mention $wgPFEnableStringFunctions directly?

Note, currently $wgPFEnableStringFunctions enables the following:
* {{#len:}}
* {{#pos:}}
* {{#rpos:}}
* {{#sub:}}
* {{#count:}}
* {{#replace:}}
* {{#explode:}}
* {{#urldecode:}}

I don't see why this isn't a duplicate of bug 6455.  You're not going to get people turning stuff on when Tim's vetoed it, go ask Tim directly to reconsider if you want it changed.  Or try to persuade other sysops/tech employees/whatever to advocate for your cause.  Opening new bugs on already-decided issues isn't going to change anything.

(In reply to comment #2)
> I don't see why this isn't a duplicate of bug 6455.  You're not going to get
> people turning stuff on when Tim's vetoed it, go ask Tim directly to reconsider
> if you want it changed.  Or try to persuade other sysops/tech
> employees/whatever to advocate for your cause.  Opening new bugs on
> already-decided issues isn't going to change anything.

Bug 6455 is about enabling a specific extension (now a configuration variable inside another extension). This bug is about implementing the functionality, in some form other than the current padleft/padright hacks. Nobody is asking that StringFunctions be re-evaluated. They are asking for this functionality in some form (whether that's Lua, JavaScript, or whatever else) to exist on Wikimedia wikis.

What MzMcbride said. I thought I'd made this perfectly clear in my OP comment, but evidently not.

As I said on bug 6455, if these two bugs are directly related at all, it would be that it blocks this one; this is certainly not a dupe.

This should go under extension requests then. Site requests are for configuration changes, which this is not (since you're so clearly saying this isn't a dupe of bug 6455)

That may be; I'm not familiar with the product/component bug landscape, and just filed this under the categorization that sounded most relevant.

I am going to regret asking this, but *exactly* which features do you want this to have?

I'm really not sure - I was never heavily involved in templates that would have benefited from string parsing myself (though most templates doing advanced parsing would benefit *somehow* from native string parsing), and my involvement with the community has been greatly reduced for months, meaning I'm now rather out-of-touch with the current template landscape on en.WP (and I never participated to any appreciable extent on other projects in the first place, so I have no idea what their needs are).

That being said, I'd imagine that, at the least, functions for determining a string's length and finding the position of, replacing, or returning a substring (or splitting a string into substrings) would be needed. Depending on the implementation, a replacement for pad[left] and padright may also be a good idea.

I'd appreciate it if someone more in-touch with the current needs of (at least) the en.WP community (and, preferably, a stronger background in programming/computer science) could weigh in, too.

(In reply to comment #7)
> I am going to regret asking this, but *exactly* which features do you want this
> to have?

As enwiki has demonstrated admirably with [[Category:String manipulation templates]], the only fundamental requirement is a truncate function: one that returns the first n characters of a string.  This is [[Template:Str left]], and the part of the hierarchy which uses the {{padleft:}} hack.  From that enwiki has built {{str index}} to return the n'th character of a string, by iterating through a (limited) character set and checking whether 
   {{str left|<string>|n-1}} . X == {{str left|<string>|n}}

From that enwiki has built {{str sub}}, exactly analogous to PHP's substr(), by selecting the individual characters with {{str index}} and concatenating them together at the end; and {{str len}} to get the length of a string by checking for 

    {{str left|<string>|n}} == {{str left|<string>|n+1}}

From there, {{str right}}, analogous to substr(<string>|<n>).  From there, {{str find}}, a crude pattern match, and from there {{str sub}}, an even cruder replace-in-string function.

All of these higher functions are built on enormous pyramids of lower functions and come at substantial performance penalties.  But there is very clearly a use for them, and their utilisation is only currently constrained by that poor performance, and specifically that people are afraid of incurring the sysadmins' wrath by overusing them.  People know that it is literally a one-line code change (r46628) to bring the whole house of cards tumbling down.  But fighting against that is very clearly a demand to use these features.

So the order of precedence is as follows:

1) an implementation of PHP's substr(<string>|0|<n>) is what is currently provided by the {{padleft:|<n>|<string>}} hack.

2) a full implementation of string slicing is the next level which avoids an obvious wikitext hack; in PHP this would be the full substr(<string>|<start>|<length>); IIRC in python it would be string[<start>:<length>].  

3) an implementation of PHP's strpos().

4) an implementation of PHP's str_replace().

All of these levels, as enwiki has demonstrated, can be built out of only the levels below.

The real question is, given str_replace() as a baseline rather than {{str left}}, what can you build?  :D

(In reply to comment #7)
> I am going to regret asking this, but *exactly* which features do you want this
> to have?

Well, basically we want StringFunctions, but the developers have said no, so what we really want is something equivalent in power and speed to SF.  Unfortunately, it appears that Lua etc. won't be ready anytime soon (and the end users (like me) hate them), and Tim absolutely refuses to enable SF, so the only option is to re-implement SF from scratch and call it something else and hope Tim doesn't notice it.  Obviously that won't work, so why don't we just WONTFIX this now and save everyone a lot of trouble?

(In reply to comment #10)
> Well, basically we want StringFunctions, but the developers have said no, so
> what we really want is something equivalent in power and speed to SF.

I think you're the first person to call StringFunctions powerful and fast. (And if StringFunctions is really the best you can dream up for a fix to the current wiki templating nightmare, it's quite sad.)

> Unfortunately, it appears that Lua etc. won't be ready anytime soon (and the
> end users (like me) hate them), and Tim absolutely refuses to enable SF, so the
> only option is to re-implement SF from scratch and call it something else and
> hope Tim doesn't notice it.

No. Plenty of bugs are open for years without anyone seriously looking at them, much less working on them and eventually resolving them. This bug is a generic feature request for Wikimedia wikis. Once that functionality has been implemented (in two weeks or two decades), this bug can be closed. There's no rush.

(In reply to comment #10)
> Well, basically we want StringFunctions, but the developers have said no, so
> what we really want is something equivalent in power and speed to SF. 
> Unfortunately, it appears that Lua etc. won't be ready anytime soon (and the
> end users (like me) hate them)

Where was this ever discussed? I don't think I've ever seen any sort of concerted opposition to the use of a wikitext alternative in templates. How can you just dismiss the entire possibility out of hand? Lua is only one of several possible options, one of which would be to implement our own system that could be anything we wanted.

(In reply to comment #12)
> (In reply to comment #10)
> > Well, basically we want StringFunctions, but the developers have said no, so
> > what we really want is something equivalent in power and speed to SF. 
> > Unfortunately, it appears that Lua etc. won't be ready anytime soon (and the
> > end users (like me) hate them)
> 
> Where was this ever discussed? I don't think I've ever seen any sort of
> concerted opposition to the use of a wikitext alternative in templates.

See bug 19298, particularly some of the later comments.

I only see one person there (Robert Rohde, in comment 16) who could really be described as an "end user" who was opposed to Lua. The others are either developers or they're ambivalent about it. I don't think that's really enough evidence to make the generalization that "end users (like me) hate them", especially when no one really knows what "them" is other than Lua.

The objection to Lua was not on the part of MediaWiki users.  The objection was that it would make Wikipedia content impossible to reuse on shared hosting, since they can't install Lua there.  For Wikimedia use, everyone seems to agree that it's ideal.  But last time we discussed it, Brion (then still CTO/lead developer) made it clear that reusability on shared hosts was a blocker issue, so Wikimedia content cannot depend on anything other than a straight PHP install.  Now there's no one in a clear position to make that decision either way, so I assume we'll just see stasis here, with people who have no power to do anything pointlessly rehashing the issue on Bugzilla and/or wikitech-l indefinitely.

I have to agree with Aryeh,  there was a pipe-dram about Lua. Now that's dead this needs escalating. 

We currently have a situation where templates can stop working for mysterious reasons.  We have simple stuff ten-year-old children used to hack on pocket money computers 30 years ago, that professional software engineers are sweating blood to achieve (or not).   

To simply dismiss this crying need because of Lua was bad enough, but because of a throw-away comment on a mailing list - surely not.


Current situation:

o Frustrated template writers
o Inefficient (horrendously inefficient) code
o Missing functionality
o Reduced productivity

Requested situation

o Happy template writers
o Fast code
o buckets of functionality
o Huge leaps in productivity

What's not to like?

(In reply to comment #16)
> I have to agree with Aryeh,  there was a pipe-dram about Lua. Now that's dead
> this needs escalating. 

Escalation is the word, certainly... :P
 
> We currently have a situation where templates can stop working for mysterious
> reasons.  We have simple stuff ten-year-old children used to hack on pocket
> money computers 30 years ago, that professional software engineers are sweating
> blood to achieve (or not).

Allowing this functionality is not going to make templates stop breaking.  If anything, the fear is that it will develop to the point where when they break, they do so in *even more mysterious* ways.

> Current situation:
> 
> o Frustrated template writers
> o Inefficient (horrendously inefficient) code
> o Missing functionality
> o Reduced productivity
> 
> Requested situation
> 
> o Happy template writers
> o Fast code
> o buckets of functionality
> o Huge leaps in productivity
> 
> What's not to like?

The fact that there is no reason to doubt that this utopia will in fact turn into a dystopia where _frustrated template writers_ are using these string functions to simulate the _missing functionality_ of regular expressions with _horrendously inefficient_ code built on str_sub().  All historical precedent suggests that this is what *will* happen, because those template writers are happiest and most productive when they are pushing the envelope of what is possible.  I don't want to look at [[Template:preg_replace]] in three years' time and see something which makes today's string templates look like Hello World functions.  I *especially* don't want to see spaghetti bowls of parser function tags which are rendered even more incomprehensible by mystic {{{{{safesubst|}}}foo:...|safesubst={{{safesubst|}}}}} constructs. 

The objection to StringFunctions is *not*, in general, that the functionality should not be available.  It's that we don't want the next generation of functionality to look like the previous generation of functionality, because that is not a good look.  Cf http://en.wikipedia.org/w/index.php?title=User:Happy-melon/sandbox5&oldid=405905787 for how [[Template:Navbox]] could look in python, for instance.  Programming languages are designed to be extensible, to retain a similar level of *visual* complexity regardless of the *computational* complexity.  ParserFunctions etc, are definitely *not* designed in that fashion.

(mid-air collision with comment 17)

(In reply to comment #16)
> o Happy template writers

And non-template writers? Will they like it when already-complicated templates
get more complicated?

> o Fast code

For how long? How long will it take for these to be used to their limits? It
looks like it took less than 2 years for the string padding functions to be
abused to this point.

> o buckets of functionality
> o Huge leaps in productivity
> What's not to like?

Is it functionality that we necessarily want in wikitext? I've seen people
saying that they could use this to implement a parser for a simple data storage
format (similar to CSV), and there was some allegation that people wanted to do
natural language processing. While most of the added complexity stays in the
templates themselves, it does eventually filter into articles through the
addition of more parameters.

Has it actually been shown that the current hacks really are slow? They're ugly
hacks, but that doesn't necessarily mean that they're actually slow in real
applications. They're almost certainly comparatively slower than native string
functions, but if the difference is only 50ms, then "faster" wouldn't be a good
argument.

(In reply to comment #18)
> Has it actually been shown that the current hacks really are slow? They're ugly
> hacks, but that doesn't necessarily mean that they're actually slow in real
> applications. They're almost certainly comparatively slower than native string
> functions, but if the difference is only 50ms, then "faster" wouldn't be a good
> argument.

Well, I suppose it depends how broadly you define "hacks." But it takes over 30 _seconds_ to render [[Barack Obama]] (<http://en.wikipedia.org/w/index.php?title=Barack_Obama&action=purge>), largely due to the use of overblown and obnoxious templates like [[Template:Cite]], from my understanding. I don't know of anyone who thinks this is acceptably fast.

(In reply to comment #19)
> (In reply to comment #18)
> > Has it actually been shown that the current hacks really are slow? They're ugly
> > hacks, but that doesn't necessarily mean that they're actually slow in real
> > applications. They're almost certainly comparatively slower than native string
> > functions, but if the difference is only 50ms, then "faster" wouldn't be a good
> > argument.
> 
> Well, I suppose it depends how broadly you define "hacks." But it takes over 30
> _seconds_ to render [[Barack Obama]]
> (<http://en.wikipedia.org/w/index.php?title=Barack_Obama&action=purge>),
> largely due to the use of overblown and obnoxious templates like
> [[Template:Cite]], from my understanding. I don't know of anyone who thinks
> this is acceptably fast.

I'm referring to the string hacks. I'm aware that the cite templates are slow. However, while the Obama article uses nearly 200 different templates, none of them appear to to be in [[Category:String manipulation templates]], which would be the ones improved by string parser functions.

(In reply to comment #20)
> I'm referring to the string hacks. I'm aware that the cite templates are slow.
> However, while the Obama article uses nearly 200 different templates, none of
> them appear to to be in [[Category:String manipulation templates]], which would
> be the ones improved by string parser functions.

StringFunctions and any similar implementation are pretty much outright banned and Rich has pivoted the conversation toward taking a more holistic approach to resolving this bug. His point was that a new wiki templating system would be faster. My point is that it's important when discussing a more holistic approach to note that it isn't just string manipulation that makes rendering slow as shit. The entire recursive brace substitution scheme is awful. If something like <http://en.wikipedia.org/w/index.php?title=Template:Citation/core&action=edit> could be rewritten in a sane language, it would undoubtedly improve rendering time of large pages _and_ allow for extra goodies like string manipulation. One can dream.

(In reply to comment #21)
> (In reply to comment #20)
> > I'm referring to the string hacks. I'm aware that the cite templates are slow.
> > However, while the Obama article uses nearly 200 different templates, none of
> > them appear to to be in [[Category:String manipulation templates]], which would
> > be the ones improved by string parser functions.
> 
> StringFunctions and any similar implementation are pretty much outright banned
> and Rich has pivoted the conversation toward taking a more holistic approach to
> resolving this bug. His point was that a new wiki templating system would be
> faster. My point is that it's important when discussing a more holistic
> approach to note that it isn't just string manipulation that makes rendering
> slow as shit. The entire recursive brace substitution scheme is awful. If
> something like
> <http://en.wikipedia.org/w/index.php?title=Template:Citation/core&action=edit>
> could be rewritten in a sane language, it would undoubtedly improve rendering
> time of large pages _and_ allow for extra goodies like string manipulation. One
> can dream.

Oh, yes. On that I totally agree. The current system is not only slow, its ugly and difficult to use. Just a separation between control statements and I/O would go a long way, making crap like [[Template:!]] unnecessary.

Bug 26786 refers to comment 19, comment 21 is also relevant.

Would the installation of http://www.mediawiki.org/wiki/Extension:ParserFunctions fix this?

(In reply to comment #24)
> Would the installation of
> http://www.mediawiki.org/wiki/Extension:ParserFunctions fix this?

Please tell me you're joking...

(In reply to comment #25)
> Please tell me you're joking...

Not joking, but, as MZ pointed out on IRC, I am ignorant.

It has only been dismissively mentioned here and mediawiki.org does say SF is deprecated (which everyone here seems to know).

So it would help developers if it were made clearer here that the thing that is wanted is some features enabled or implemented in ParserFunctions, that is, there are some basic, necessary functions that are requested here for PF.

Msybe everyone here understood that, but an ignorant developer (like me) would miss it.

(In reply to comment #26)
> So it would help developers if it were made clearer here that the thing that is
> wanted is some features enabled or implemented in ParserFunctions, that is,
> there are some basic, necessary functions that are requested here for PF.
> 
> Msybe everyone here understood that, but an ignorant developer (like me) would
> miss it.

It would help if developers read the opening bug comment:

> It is blatantly obvious that *some* form of string parsing functionality is
> needed on WMF wikis - this is most clearly demonstrated by [[Category:String
> manipulation templates]], an entire category dedicated to horrible abuses of
> currently-enabled markup to allow such functionality. However, developers have
> repeatedly stated that StringFunctions is not going to happen (bug 6455), which
> leaves only the option of enabling some actual programming language.

But, MZ, I'm not sure how saying "StringFunctions isn't going to happen" is the same as saying "ParserFunctions can't fill the role".  When you say "oh, if we can't have StringFunctions, then we need some actual programming language" you're ignoring this third solution.

(In reply to comment #28)
> But, MZ, I'm not sure how saying "StringFunctions isn't going to happen" is the
> same as saying "ParserFunctions can't fill the role".  When you say "oh, if we
> can't have StringFunctions, then we need some actual programming language"
> you're ignoring this third solution.

I don't understand how any of this is confusing to you. There were two extensions (StringFunctions and ParserFunctions). In an attempt to get StringFunctions enabled on Wikimedia wikis, the StringFunctions extension was merged into ParserFunctions, making the StringFunctions extension obsolete. Tim added a configuration variable that specifically disables StringFunctions functionality within the ParserFunctions extension.

The ability for string manipulation is already built in to ParserFunctions. If this bug were about getting ParserFunctions to "fill this [string manipulation] role," the bug would already be resolved. ParserFunctions already has the ability to manipulate strings, but it has been clearly stated at bug 6455 that enabling StringFunctions functionality from within the ParserFunctions extension is not going to happen.

This bug is about the generic ability to manipulate strings on a Wikimedia wiki using some kind of markup. This bug is _not_ about adding string manipulation functionality to ParserFunctions (or doing anything with the ParserFunctions extension, for that matter).

(In reply to comment #28)
> But, MZ, I'm not sure how saying "StringFunctions isn't going to happen" is the
> same as saying "ParserFunctions can't fill the role".  When you say "oh, if we
> can't have StringFunctions, then we need some actual programming language"
> you're ignoring this third solution.

In addition, ParserFunctions is already used to help clobber together a precarious patchwork of this functionality, together with padleft/padright. ParserFunctions can't be used to address this bug, simply because it doesn't, and wouldn't, provide a _native_ string manipulation environment.

I find Mark's confusion only too understandable.

The thrust of the argument against providing extended parser functions (and most other string handling possibilities) seems to be "people will use them"  and cause performance problems. This is a form of dynamical system - the more facilities you provide the more are used, until a problem is hit. From the point of view of - say - rendering time, or expanded size or some other metric, the community receives a new, more efficient tool, and after a period of time the rendering times are back to what they were before.  So if you are running the servers you might say "nothing gained".  But in fact, functionality has been gained.  Look if you will at 

[[en:Template:Monthly clean-up category]]

If you want to create a monthly clean-up category you stick this template on the page and that's it. No paramters. No cats. No Defaultsort. No tag, boxes, typing, just "Edit" ctrl V alt-shift-s. 

That is productivity, that means we have more time for other stuff.  And since the pages are behind the scenes, the render time isn't critical (but I'll bet it's pretty quick anyway, and with Wikkd (sp's) speedups could be quicker).

Now there is no reason this sort of functionality shouldn't be used to parse template arguments.  I cannot, for example, reliably remove [[]] from a parameter. This means the only solution to the "link or not" dilemma is to not link, and enforce that by bot, or using string functions to spot the "[", and then use "autolink" which invokes ifexists  which (we are told) is an "expensive parser function" (why it's expensive is a mystery, it should be much cheaper to find if something exists than to transclude it, since transclusion needs to determine if the thing exists anyway).

The recursive {{}} (which are not recursive) are not really a problem, the problem lies with the fact that some of the core or widely used templates are inefficient - I plead guilty to adding a few symbols to {{yesno}} many moons ago, which probably get evaluated several hundreds of millions of times a day.

But  for "deep template magic" I don't really care where the functionality or performance comes from - I could suggest a hashed lookup table might be a great boost, for example - I just want to be able to use the sub-template [[en:Template:Page numbers]] in the cite family without killing pages.

Good software engineering practice is to validate your inputs, we don't have that luxury without simple string handling.

Please, lets not turn this into a re-iteration of the 150 odd comments on bug 6455.

>(why it's expensive is a mystery, it should be much
>cheaper to find if something exists than to transclude it, since transclusion
>needs to determine if the thing exists anyway)

Presumably because it involves a DB hit (in comparison to {{#if which does not). Transclusions aren't parser functions, so can't really fall into the expensive /parser functions/ category. Presumably people also tend to only use transclusions when they want to actually include a page, they don't use it as part of the hidden logic of a template.

Template coders very heavily use transclusion as "part of the hidden logic" of templates - there are things that are far cheaper and easier to do with a transclusion than without (for instance, parameter validation and preprocessing, when the validated/preprocessed parameters are used in more than just one or two places in the template proper). A lot of the stuff transclusions are used for could be simplified, or replaced outright, if we were able to create and store variables, but that's a different bug report (bug 7865, specifically).

People, you generals are talking here, but I'm a simple russian template writer and I see that the situation with string automation is insufficient, and it's an 0_o 0_o that it's keeping insufficient for years.

The main leak is that we can get left side of string by {{padleft:}}, but there is nothig which can give us the right side. The effect is that {{Str right}} works with step by step comparing each symbol with any point of alphabet. 
  * This seems to be less effective than 10 times more used special functions.
  * There is no way to cover all the Unicode in such way, while there are much of very exotic words which might have need to be treatened automatically (e.g. wouldn't one write a brief course of Martian language on Wikiversity and want to automate declesion table creation).
  * You said A, please say B. If we've got the left side of string, it's boring for us to wait for the right side for >5 years.

Nearly the same happens w {{Str len}}. OK, a good Perl regexp replacer can hang the server and even crack it, even {{#rep:}} can slow it down, but only pair of strlen() and substr() will make a great improvement of use without any productivity loss. Can anyone strip them from StringExtension and install on WMF projects separately from other ones? I haven't dealed w this code much but believe it's two good weekends of 2-3 men at much most. I'm even not asking for strpos() since {{Str pos}} at least has nearly similary sane algorithm.

P.S. I don't think builtin script engine is well for Wikipedia. Most templates are written by people who know Martian language, not Lua. It's possible to write as bad program on any language as on MadParserFunctionStringRegexScript, and it will be done. But it's not a reason of doing nothing. If MW won't be developed on WMF, it will then there deprecate and WMF with it, with all my respects to its great awards till nowadays.

With best wishes, [[:ru:User:Ignatus]].

Phillip: Wikimedia wikis now support Scribunto (cf. [[mw:Extension:Scribunto]]) and Lua (cf. [[mw:Extension:Scribunto/Lua reference manual]]). This should provide robust string manipulation capability and resolve this bug as fixed, I believe.

Putting resolved fixed per mzmcbride

(In reply to comment #35)
> Phillip: Wikimedia wikis now support Scribunto (cf.
> [[mw:Extension:Scribunto]])
> and Lua (cf. [[mw:Extension:Scribunto/Lua reference manual]]). This should
> provide robust string manipulation capability and resolve this bug as fixed,
> I
> believe.

I'm acutely aware of that; I've been following Lua development with some interest. I was actually wondering when this bug would be marked resolved fixed as a result. =D