Wikimedia Bugzilla is closed!

marking the more general bug 14045 as dependant on this.

To carify: I wrote about why it does NOT make sense to provide the feed via IRC, and why XMPP would be much better. http://brightbyte.de/page/RecentChanges_via_Jabber

nom nom nom :)

To summarize for those too lazy to go read the awesome blog post:

* Provide for Jabber output embedding XML payload in format of API recentchanges feed
* Some internal interface making it easy for MW to ship those out to a Jabber server to send to the outside world (some handy UDP-packet-stream-to-XMPP-bridge so we can use off-the-shelf Jabber server for that part?)
* in future, consider options for pubsub/filtering fancy stuff

(In reply to comment #1)
> marking the more general bug 14045 as dependant on this.
> 

So the intent is definitely to provide an easy-to-parse/machine-readable feed specifically intended for this use? *drools*

This was discussed at the developer meet-up in Berlin. Notes here: http://www.mediawiki.org/wiki/Project:Developer_meet-up_2009/Notes/XMPP

Catrope and Leafnode said they would work on it.

This would also be wonderful for SUL talkpage changes. I'd love to be able to receive notifications of stuff like that in my Adium client for instance. 

Code for generating the XML notification messages, and some rudiments of XMPP integration, are in SVN at <http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions/XMLRC/>. The code is somewhat incomplete and largely untested though, pending a test setup with a working XMPP server (poke poke brion wink wink).

Is this still being worked on? I've been told not to make changes to the IRC recent changes feed because it's going to be replaced by this, yet all I see is some old discussion. Meanwhile core MediaWiki functionality is completely unsupported in the feed making client applications far more inefficient than they should be.  I don't have the necessary access/knowledge to progress this issue further (apparently some kind of test server needs to be set up) but I am more than capable of modifying the feed code.

@Gurch this has been sitting on my todo-list for some time now, but I plan to make serious progress on this in march and April. The part for emitting XML via UDP is pretty much done, the bit I have to do is a udp-to-xmpp-bridge that stuffs that XML info a MUC or, better, pubsub  channel. 

Poke me again in a month or so.

Just a 2-Months-later-poke....

Is there any possibility of, in addition to summary, user, etc., also distributing the new article contents via XMPP when edits are made? That could take some load off the servers by eliminating the need for bots to query the API for article contents, e.g. when they suspect vandalism and need to check it out.

Best way to include text is probably to have distinct feeds you could subscribe to for metadata-only and metadata-plus-full text.  That'll keep the main feeds lightweight, but let's those who need the text avoid an extra web hit.

I notice that XMLRC_XMPP.class.php has this line: include( "XMPP.php" );

But, there is no XMPP.php in the XMLRC directory in SVN. What PHP XMPP library should be used with this? There seem to be several available on the Internet.

@tisane: that class is work in progress, i never got around to finishing it. i think i was using the xmpphp library, but i'm not sure any more. writing the xmpp connector (direct, and more imprtantly a standalone udp-to-xmpp bridge) is still on my todo list.

*** Bug 23001 has been marked as a duplicate of this bug. ***

Perhaps an alternate method while this is being implemented is to set up an alternate IRC server that sends JSON-encoded messages to make it easier to program bots?

You would hit the size limit of one message quite quickly.

What XMPP size limit are you referring to? I thought one of the advantages of XMPP is that it does not specify a hardcoded limit on the size of XMPP stanzas. http://xmpp.org/extensions/inbox/stanzalimits.html

(In reply to comment #18)
> What XMPP size limit are you referring to? I thought one of the advantages of
> XMPP is that it does not specify a hardcoded limit on the size of XMPP stanzas.
> http://xmpp.org/extensions/inbox/stanzalimits.html

I think comment 17 was referring to the IRC-related idea in comment 16.

I finished XMLRC, an extension implementing this: <http://www.mediawiki.org/wiki/Extension:XMLRC>. I have set up a demo on the toolserver (fed by polling the API). You can join it at enwiki@conference.jabber.toolserver.org using any jabber client. 

The human readable messages are similar to the ones send to the RC channels on IRC, but there's extra XML payload in the messages, like this:

<message xmlns="jabber:client" to="john@jabber.org/mars"
         from="enwiki@conference.jabber.toolserver.org/xmlrc"
         id="38" type="groupchat">
  <body>[[List of Knight's Cross recipients: Z]]  
        http://en.wikipedia.org/w/index.php?diff=378715833&amp;rcid=389791968&amp;oldid=378714319&amp;title=List+of+Knight%27s+Cross+recipients%3A+Z * 
        MisterBee1966 * (+1203) /* Recipients */</body>
  <rc comment="/* Recipients */ " newlen="26554" rcid="389791968"
      pageid="8089657" title="List of Knight's Cross recipients: Z"
      timestamp="2010-08-13T14:08:49Z" wikiid="enwiki" 
      revid="378715833" old_revid="378714319" user="MisterBee1966"
      ns="0" type="edit" oldlen="25351">
    <tags />
  </rc>
</message>

The extension includes a demo client in python.

You may want to set the xml:space="preserve" attribute in the <body> tag, so XML parsers will leave whitespace in the content alone.

(In reply to comment #21)
> You may want to set the xml:space="preserve" attribute in the <body> tag, so
> XML parsers will leave whitespace in the content alone.

Uh, i could, but... there's no text, all information is in attributes. So i see no point...

(In reply to comment #21)
or do you mean in the regular xmpp message body? that'S up the xmpp library, i won't mess with that. anyway, it's all on one line.

All right then, don't bother with the library. I would LOVE to see this, at least on test.wikipedia.org for testing.

Quick query, why use just XML? It would be nice to be able to get an alternate format on request, such as JSON or serialized PHP.

(In reply to comment #25)
> Quick query, why use just XML? It would be nice to be able to get an alternate
> format on request, such as JSON or serialized PHP.

Two reasons: 

1) XMPP supports XML payload - that means I can embed XML natively in XMPP stanzas. No need for wrapping, escaping or encoding. It will also get parsed automatically, and is accessible directly in the xmpp client framework. 

2) there are no requests. Everyone gets the same data, this is basically a multicast setup. Broadcasting everything in all formats would be silly.

That being said: I'm thinking about making an XMPP interface to the API, so people can send queries and receive answers via XMPP. In that case, supporting different formats would be an option, though it still seems silly: with XMPP, XML parsing comes for free, handling any other format adds work and cost.

I have described the prototype setup at http://meta.wikimedia.org/wiki/Recentchanges_via_XMPP

I would appreciate having XMLRC enabled on the Wikimedia wiki family soon as it is *really* helpful in writing RC appliations.  Especially on writing new programs and scripts (as I am doing at the moment) it is suboptimal if you have to implement IRC support knowing that a. it will be outdated soon and b. it is very complicated to extract the real information from the mixture of color codes and special characters.  So I want to confirm the previous wishes for quick activation.

Just realized that this would not be able to be placed in core as it stands (which is what I would like to see eventually), due to the Python prerequisite.

perhaps the bit that emits the udp packets could be split of and put into core - that's the part that is actually a proper mediawiki extension. the rest is utilities, really...

(In reply to comment #24)
> All right then, don't bother with the library. I would LOVE to see this, at
> least on test.wikipedia.org for testing.

This is bug 24836.

(In reply to comment #29)
> Just realized that this would not be able to be placed in core as it stands
> (which is what I would like to see eventually), due to the Python prerequisite.

Well, there's nothing to say that core has to be pure PHP. In fact, parts currently aren't (like math support). It should certainly be discouraged, though.

math is in there for historical reasons. However, I do know that extensions like timeline, etc have been denied merging with core for non-PHP-ness.

This feature would be the long-term solution to problems like 34508 (not marking as blocker since a near-term solution will be pursued perhaps).

Relevant: https://www.mediawiki.org/wiki/Requests_for_comment/Publishing_the_RecentChanges_feed

Change 131040 had a related patch set uploaded by Ori.livneh:
Add 'rcstream' module for broadcasting recent changes over WebSockets

https://gerrit.wikimedia.org/r/131040

Change 131040 merged by Faidon Liambotis:
Add 'rcstream' module for broadcasting recent changes over WebSockets

https://gerrit.wikimedia.org/r/131040

Sometimes it takes more than 5 years, but it's great to see that stuff is happening!

Closing this bug in MediaWiki core.

* MediaWiki now provides the wgRCFeeds and MachineReadableRCFeedFormatter interfaces. These don't set up their own subscribable web service, but that seems outside of the scope for MediaWiki. The interface to broadcast to such a service (set up by the server admin) exists, and that seems enough.



* While this bug specifically requested XMPP, the overall use case for Wikimedia wikis is bug 14045. It focusses on setting up a public service that will make use of this feed for Wikimedia wikis (though JSON-based, not XML like this bug suggested). If we really want XMPP specifically, we should file a new bug for that I think,

Thanks for this work, much appreciated!