Last modified: 2007-04-25 22:43:00 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T11439, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 9439 - Conditional inclusions should only actually be included if the condition is true
Conditional inclusions should only actually be included if the condition is true
Status: RESOLVED DUPLICATE of bug 8314
Product: MediaWiki
Classification: Unclassified
Parser (Other open bugs)
unspecified
All All
: Normal enhancement (vote)
: ---
Assigned To: Nobody - You can work on this!
http://en.wikipedia.org/wiki/List_of_...
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2007-03-28 19:42 UTC by Paul Robinson
Modified: 2007-04-25 22:43 UTC (History)
0 users

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Paul Robinson 2007-03-28 19:42:09 UTC
When the same macro is transcluded into a page more than once - even where the
uses of each transclusion have different arguments - sometimes the page fails,
inexplicably.  Look at the page for
http://en.wikipedia.org/wiki/List_of_counties_in_Wisconsin (List of counties in
Wisconsin), which uses the County1 template for every entry.  This is done to
make the internal page "cleaner" since the information in each entry in the
table is the same, it makes sense to use a template call to generate each record
in the table.  However, after the same macro is called about 15 times, for some
reason the page renderer partially fails and doesn't reneder all of the macro. 
After it partially fails a couple of times, then it completely "forgets" the
macro exists, and presumes that the macro is missing, it can no longer see it
again. Even changing the macro call to prefix it with :Template: does not fix
the problem, it still thinks the template is in the main name space.
Comment 1 Rob Church 2007-03-28 20:05:27 UTC
You seem to be hitting the inclusion size limit; check the HTML source of the
rendered page for a series of comments indicating this.
Comment 2 Steve Sanbeg 2007-04-05 15:47:10 UTC
It's hitting the pre-expand limit, which could be caused by either large
comments in the template, or by the large #switch templates.
Comment 3 Steve Sanbeg 2007-04-05 15:48:40 UTC
BTW, I think I fixed the problem where the links were going to the wrong
namespace, so when it does hit the limit, it should link the right page now.
Comment 4 Paul Robinson 2007-04-16 02:15:11 UTC
I know what the problem is.  If a macro calls another macro, when it reads the
included macro, it counts every line of the included macro even if it will only
generate one line, or even nothing at all.  So a macro using 50 or 60 lines that
is included but only generates one line, will count as 50 lines.  So if there
are, say, 50 called macros in a {{#switch:}} statement, each of which of the
called macros has 50 lines, and if only one of the macros is activated, it will
generate one line but count as if it is generating 2500 (50*50) lines.
Comment 5 Aryeh Gregor (not reading bugmail, please e-mail directly) 2007-04-16 02:23:17 UTC
Precisely.  This behavior may be changed at some point in the future, but
currently that's how it works.
Comment 6 Steve Sanbeg 2007-04-17 16:28:31 UTC
Yeah, I agree.  But now that I think about it, I don't think the pre-expand
limit should really be cumulative.  Although it should be possible to work
around this with a bunch of ugly meta templates, it's probably better to relax
that rule a bit so parser functions will work here.
Comment 7 Brion Vibber 2007-04-24 20:55:03 UTC
As long as things are processed, limits will apply to them. Closing.
Comment 8 Aryeh Gregor (not reading bugmail, please e-mail directly) 2007-04-24 21:58:38 UTC
The request is for them *not* to be processed, i.e., for
{{#if:0|{{foo}}|{{bar}}}} to never actually look at or parse the contents of
Template:foo, having figured out that it doesn't have to.
Comment 9 Steve Sanbeg 2007-04-25 15:28:49 UTC
They're already not being processed; it was failing the pre-expand size limit.

This is a cumulative limit on template sizes prior to expansion; which
corresponds to how much bigger the article would get if all the templates were
included but not expanded.  Not a very realistic metric I think, but a
reasonable approximations as long as expansions strictly "expand" the templates,
i.e. when parser functions aren't used.

This could be worked around simply by replacing parser functions with a heap of
convoluted meta-templates, or fixed by making pre-expand size non-cumalative,
since you wouldn't have more than one unexpanded template at a time.
Comment 10 Aryeh Gregor (not reading bugmail, please e-mail directly) 2007-04-25 15:55:57 UTC
The reason for the pre-expand limit is it takes time for the parser to crunch
through all the text.  Anything that's actually parsed is counted because
anything that's parsed takes time to parse, whether it's displayed or not.  We
had pages that were taking ten seconds to parse because of all the templates. 
Currently unused inclusion paths must be parsed by the parser, so they're added
to the size.  The idea is to make the parser smart enough to not parse them to
begin with.

If you have an idea for a heap of convoluted meta-templates that will handle
this as desired, please use them.  They'll improve performance, and thereby
evade the parser include size limit.
Comment 11 Steve Sanbeg 2007-04-25 18:19:39 UTC
In this case, I think it was templates like
{{#switch:{{{1}}}|NJ=New Jersey|NY=New York|...}}
That were used repeatedly, creating a lot of pre-expand text.  Changing that to
something like {{statename/{{{1}}}}} and creating 50 sub-templates should
decrease the pre-expand text.

What you're talking about sounds more like bug 8314 that my understanding of
this one.
Comment 12 Paul Robinson 2007-04-25 19:41:15 UTC
Actually, I was doing it in the form of something similar to
((#switch:(((1)))|NJ={{StateName|34}} 
or whatever so it used the FIPS code, or something like that, so that in the
event of a change only the lowest-level underlying template would be needed, and
the changes would "cascade up" to all other templates that used them.  Only
problem with that is that it in effect counted all the templates that were never
referenced, so that if it was CT, then CT would call the one template, it was
still presuming that all 50 subtemplates would be called for this one entry, as
well as presuming that all of the other 49 entries would each call their 50
sub-entries as well, leading to 2500 macro calls and the effective expansions.

Having heard the arguments about inclusion time, I guess I can understand why it
is putting this limit in this way - a macro expansion that takes 10 seconds is
unacceptable, and preventing this is a worthy goal - I never thought about it. 

I do want to thank everyone who looked into this issue and provided such rapid
feedback.  While I was curious about why this was happening, it was more
important that I find a way to fix it, and I have.  Both on allowing the
template to work, and a way to make sure someone else in the future, when new
entries are added (I suspect there will be some new state(s) add to the United
States eventually, maybe Canada will take the original invitation provided in
the Articles of Confederation and accept the offer to have its provinces become
states, maybe Texas decides to take up the automatic provision to allow itself
to divide into more states, or maybe Guam and/or Puerto Rico will be added.)

Well, what I did to fix the problem was simply to put a non-included note on
each transclusion template listing all of the templates related to the
particular function and cross referencing them, reminding any future editor to
change all of them if new information is added.  It's not as foolproof as making
it fully automatic but at least it works, which is primary and fundamental.

But, there is a question about why it counts entries that cannot be included and
will never be included, e.g. when it selects CT or NJ, why is it still expanding
the other entries that will never be used?  I've discovered it did this when I
did a {{subst:county1|parameters}} in order to make the original fix, and what I
noticed is that it broke the one macro into its 50 component macro calls when
the case statement was executed.

If I compile the equivalent of a case statement in a programming language, or
even if it's interpreted, the interpreter takes the case branch that matches and
does not execute the rest of the code in that case statement.  The language
processor skips over the remaining unexecuted case branches until the case end
delimiter is found.  Now, granted there may be some execution time to skip over
the unexecuted branches, it seems like it's excessive to presume that every case
branch has to be expanded or that it's impossible to tell which branch will be
executed.  Unless the decoding of called macros (inclusion) is done before
branch prediction takes place, e.g. the macro is read and decoded into its
component parts then the parameters are examined to see which parts are used.

Would there be an objection if I took a look at the code on how this is done and
offerred suggestions?
Comment 13 Paul Robinson 2007-04-25 19:50:12 UTC
Having read the other bug 8314 I can understand that there are arguments either
way with respect to interpreting the value of a parameter before inclusion or
doing so after inclusion.  What might be asked is, which would be faster or
which would be better as far as usage factors.  For example, if the use of
parameter interpretation in one way or another spoils caching then that's
obviously unacceptable (is caching done at the static page level or at the wiki
code level?) Or perhaps it has to do with whether a parameter is static or
dynamic at rendering time.  Maybe one of these execution forms will only work
with "late binding," e.g. the parameters are translated after the wiki source
has been extracted.  I don't know how that would be possible, but, then I don't
know the code so it's not my place to say.
Comment 14 Steve Sanbeg 2007-04-25 22:42:36 UTC
It is all open source, so you're certainly welcome to look at it and see if
there's a better way.

It's really a matter of precedence; parser functions and templates expect their
arguments to be pre-expanded, but in a case where they through away most of
their arguments, and render the rest a second time, that's not ideal.

What I partially did but never finished was to make {{defer:thing}} expand to
{{thing}}, so you can explicitly defer substitutions when they can be done
later.  That's the best I could come up with.
Comment 15 Steve Sanbeg 2007-04-25 22:43:00 UTC

*** This bug has been marked as a duplicate of 8314 ***

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links