XMPP software documentation

In an attempt to dust off my neglected blog, I bring you the following post about something that has been nagging at me for the past couple of days.

Since the recent changes at jabber.org I've been thinking a lot about Jabber from a new user's perspective.

The new web-based registration form may help clear up a lot of confusion that resulted from in-band registration (many users were looking for somewhere on the website to register), but after registration what do they do next?

They no longer need a client to register, and it is now quite likely that after registration they won't yet have one, so obviously most importanly we need to guide them to find a client.

We could simply direct them to the XMPP client list on xmpp.org, however that isn't particularly friendly for someone who might not even be familiar with the term "client".

What we need is a shortlist of some user-friendly clients, with good documentation.

And this set me thinking that perhaps there ought to be a concerted effort to document the basics of XMPP clients (in fact any and all XMPP software) in a unified way.

For each client I would like to see a set of simple single-page tutorials, complete with screenshots, on issues such as:

• How to register an account


• How to add an existing account


• How to add a contact


• How to join a MUC


• How to send/receive files

Where the pages would be hosted (in one place, or on the respective client sites) as well as some other things still need thinking about, but I'd like to know what other people think about the basic idea before setting anything in stone. Thoughts?

irssi notifications over Jabber (or IRC barking)

Thanks to daubers (the brains behind the XMPP doorbell) who suggested it, I now have irssi highlights sent directly to my Jabber client. Quite a handy thing since only yesterday I logged in to find several PMs awaiting me from 4 days ago.

For the base I used an existing irssi script called fnotify, which was designed to write notifications to a file. I found this script from Aaron Toponce's blog post.

mkdir -p ~/.irssi/scripts/autorun
cd ~/.irssi/scripts
wget http://www.leemhuis.info/files/fnotify/fnotify -O fnotify.pl
chmod +x fnotify.pl
ln -s ../fnotify.pl autorun/fnotify.pl
nano fnotify.pl

Change line 56 (which reads something like open(FILE, ...)) to:

open(FILE,"| sendxmpp -i me\@example.com");

Replace the JID with your own, and make sure to put a \backslash before the @ sign as above.

sudo apt-get install sendxmpp

When it is installed, tell sendxmpp what account to use for sending messages:

echo me@example.com verysecret123 > ~/.sendxmpprc
chmod 0600 ~/.sendxmpprc

Replace with your JID and password of course ;)

Now, in irssi:

/win 1
/run fnotify.pl

(switching to window 1 is necessary to see any error messages if they appear)

If it loads successfully, get someone to say your nick or send you a private message. It should appear in your Jabber client within a few seconds.

A consequence of this for me is that IRC now barks too :)

IM Barking

Over the weekend my brother found an old toy of his... a "robotic" dog. It has two functions, walking and "barking" (where bark is more of a squeak). Unfortunately the controller for these said functions had been put beyond practical repair by leaked batteries. I cut the controller off and wondered what uses I could put the thing to. A solution looking for a problem.

People often complain that I don't respond to their Jabber messages, often this is because I simply didn't notice it, or wasn't at my computer. I do have speakers, but I either have music playing, or when not, they are turned down to 0. Problem found.

To replace the dog's original controller I used a USB interface board. It wasn't long before I knocked together a quick Lua script to use the board and make it bark on demand.

With thanks to vArDo (Mateusz Biliński, working to add a plugin system to Gajim for GSoC 2008) for the D-BUS notification handler script, I managed to link up new message arrivals to the barking. The result? See for yourself:

Open Discussion Day 2008

Open Discussion Day (19th May) is very nearly upon us!

However due to various reasons, we did not manage to get the new site online until very late. This means there is still plenty of work to do.

If you have something to suggest, or want to offer a hand at short notice, join opendiscussion@conference.jaim.at, or #opendiscussion on Freenode!

We are keen to know what *you* are planning to do for open protocols this 19th May :)

Jabber Abuse Handling

No blog post since... when? Never mind :)

Something that interests me became a hot topic in the Jabber community (specifically server administrators). We had a meeting at fairly short notice about possible solutions to the problems we know are around the corner.

The following are some *quick* rough notes I made (too early in the morning) of an abuse reporting method, very much along the lines of XEP-0161: SPIM Reporting and XEP-0236: Abuse Reporting. What I have in mind is something of a mixture between the two. Not all abuse is SPIM, so XEP-0161 should probably be a little more generic (as XEP-0236 is).

I haven't thought much about the below, but I believe the general concept is not that far from usable, and doesn't disrupt too much the distributed nature of XMPP.


End-user
It is nearly always going to be the end user who originates abuse reports. Whether they received spam to their JID, or experienced flooding in a MUC room they administrate.

The abuse report should be sent to the abuser's home server.

Abuser's server
Upon receiving an abuse report, the server may choose to act immediately, or wait and gather more evidence (possibly from sources it deems more reliable).

It is important that the server relates reports it receives to IP addresses when weighing the evidence, *not* JIDs, since obviously the same user may have created multiple accounts).

Upon receiving satisfactory evidence that a certain IP is abusing the service, the server may decide what action(s) to take.

Clearly the most obvious step is to ban the IP from the server, and lock associated (reported[1]) accounts.

The other (optional) step is to report the IP of the offender to a central reporting/blacklist service (ie. abuse.xmpp.net), to help aid other servers who wish to prevent similar abuse of their own service.

[1] Only reported accounts should be locked, since legitimate users behind the same NAT router as an abuser could otherwise be (overly) punished. Unfortunately there is no way to avoid that legitimate users are sometimes going to get caught up in the net.

Clearly blocking an IP is not the ideal solution. A new IP is not usually hard to come by. Each reported IP should be investigated, and network administrators notified of the abuse to reduce the chance that it will happen again.

Dates and times are very important. In the case of dynamic IPs, it is the only way to track down the offending user. Perhaps it would be useful to report more than one IP + time and date (ie. the last X IPs that used the account), for this purpose.


Apologies for the unpolished post. Time to sleep on this :)

Summer of Code final update

As of 20th August Google's Summer of Code program has officially ended. Students must upload code produced up until the deadline to Google's hosting, and are only judged on work up to that date.

Continuation of GSoC projects is encouraged however, and I shall be finishing off HTTPS (and have already committed bug fixes) and tidying parts of my code over the next few weeks.

There has been plenty of interest in my summer's work, on the gloox mailing list, and in private emails/IMs to me personally. It is great to know that people are finding my work useful to them.

I would like to thank Jakob my mentor for endless help with figuring things out when they went wrong, and for guiding me around the innards of gloox when I got lost (not to mention for his excellent library existing in the first place!). Thanks to Peter for taking on the hard task of orchestrating the XSF's involvement in GSoC, and for supporting the students throughout. Finally thanks to Google and the team behind the administration of GSoC, for, well, you know what :)

I will update this post with the URL to my uploaded tarball once it is on Google's servers.

However the latest, greatest code will always be found in SVN. Currently BOSH is at svn://svn.camaya.net/gloox/branches/work/mw/bosh but very soon will be moved to the trunk. More information on gloox's download page.

The server-side implementations that I know work with this code are Openfire 3.3.2 (or .3), and Araneo (a standalone CM (connection manager), which allows any BOSH client to connect to any standard Jabber/XMPP server, using the CM as a kind of proxy) from ff.bluendo.com (site down at the moment I write this).

PS. Apologies for the delay in this post... blame my vacation :)

Update: Official tarball uploaded here. Please use the version from SVN (see above) though if you would like to test or use my code.

Summer of Code Update

This week has been a good one for my project, despite my being ill for a few days of it. Firstly, I completed my initial implementation of multiple connections! I say 'initial' because it is currently only tested and working with one server (OpenFire), and I will be testing more, and likely fixing bugs, in the next few days.

I also received a lot of help from people following my previous blog post...

My problems with PunJab are fixed with minor code changes (although it still doesn't work correctly for other reasons at the moment).

badlop let me test against his ejabberd SVN server, and I discovered that it doesn't seem to support persistent HTTP connections, let alone pipelining. Now I have non-persistent connections implemented, I shall be testing ejabberd again.

OpenFire... well, it works (without the previous problems I had with a 30 second delay) with multiple connections, and works well too :)
The only downside is still the lack of XEP-0206 support, so my code won't work with SASL auth to OpenFire.

Which brings me to the news of a new CM, which I have high hopes for. If I can get my code working as well with this CM as it currently works with OpenFire, all servers can be connected to this way, and SASL login will work as well.

A sign of Jabber's increasing popularity?

Some of you reading this are regulars in the Jabber MUC at jabber@conference.jabber.org. This MUC is referenced on www.jabber.org, and is the default room to join in many clients.

Unfortunately the past weeks have included events that I can only say are reminiscent of IRC...

• Flooding


• Abusive language


• Spamming/advertising ("hey, check out this cool site! http://..." and leaving again)

Inevitably as Jabber gains good users it will also gain bad ones. While their activity in private chats is of little concern to us, I believe we need to lay down some guidelines for acceptable behaviour in the public conference.

Currently there are a handful of moderators keeping the MUC friendly. However it becomes hard to draw the line between what is acceptable and what is not. This is a matter of opinion that varies between most people.

My hope is that with some guidelines it will be easier for both the moderator and the user (who can then be directed to them if necessary).

The need for these moves is somewhat disappointing, but necessary it seems, if we are to keep the friendly atmosphere that we currently have in this little corner of the Jabber community :) and to continue providing useful help to Jabber newcomers.

Swearing
I propose the same policy as the official Ubuntu support channels, which works well. Mild swearing is ok, as long as it is not excessive, or directed at a person. Anything else may result in a warning and/or kick at the moderator's discretion.

No deliberate flooding
Though I am not aware of it happening in the Jabber MUC, it has happened in another MUC I am in. A user joined 2 bots into the room, and set them into a loop with each other, and left. Obviously this is deliberate, and I had to ban all 3 JIDs involved (but not until several hundred messages had already been sent to the room).

In IRC it is frequent to get shouted at for pasting more than a couple of lines. In Jabber people seem to be more lenient, and I don't see why this should change. Obviously a warning should be given when needed.

English only
It may seem unfair to prevent people from speaking in their native language. However most languages have their own servers and their own conferences. I don't propose this rule to affect those casual users who come in search of help, yet speak no or little English. Those users are usually directed in a friendly way to a room and server of their native language, where they will hopefully receive better help.

However some users hold long conversations which could easily be taken into private messages, or to other rooms. These conversations are nearly always unrelated to Jabber, and only annoy the other occupants of the room. It also makes it impossible for moderators to moderate when the conversation is in a language they don't know (I have previously been asked to kick people who were being insulting toward other users, although I had no idea because they were not speaking in English).




Obviously these choices are not up to me alone to make, and so I welcome comments and opinions on what I suggest here.


Update: It seems this post caused something of a discussion... :)

Summer of Code Progress

In the past 2 weeks, I have tried every server-side BOSH implementation I can find (although more are always finding me!), and all have their individual problems with my client...

Openfire
No support for XEP-0206, and hence no SASL login is possible. I also have the reproducible (yet still mysterious) problem I blogged about in my last post. I also have had problems with Openfire's overactivity detection. The XEP specifies that the client is always allowed to make a request if it is to send data. The old version of the XEP which Openfire implemented was not clear on this, and as far as I can tell I get disconnected when I should not be.

ejabberd
Well, it seems to work. Yet ejabberd does not support XEP-0206 either, and also refuses non-SASL logins. This makes it impossible to use with my code for now.

JabberHTTPBind (JHB)
A handy little servlet which implements XEP-0124 and a little bit of XEP-0206. Yet I have problems with this also, as it seems to take wait seconds before the session creation response is received. After that I get a reply, which triggers a reply from gloox, to which the server replies instantly (yes, it appears to be polling for some reason). This happens until I begin getting errors from Tomcat (I believe JHB disconnects the client for requesting too frequently). I suspect a bug in the logic for polling, which I will take a look at after implementing multiple connections.

Punjab
The problem with Punjab seems quite simple, yet I have been unable to solve it. All my testing so far is being done locally, and Punjab seems to try to resolve 'localhost' on the external DNS. I don't know Python well, so this is the best I can understand of the problem. I intend to speak with the Punjab developer(s) this week.


All of the above are problems I am having, I would not be surprised to learn that many of them are a problem within my code. Unfortunately not having a working server implementation makes it very hard to know where the source of any particular problem lies. If anyone has suggestions about issues I have listed above, please do contact me :)

Today I managed to talk with Ian Paterson (author of the XEPs), who will kindly allow me to test against his server. Also I spoke with the anonymous commenter on my last post who has been very helpful, and offered me their Python implementation to play with as well. I learnt this week that Tigase has BOSH support in-development. With luck I will be able to test against Tigase in the coming weeks.

This week I also began the biggest change in the code for a while - enabling multiple connections to be made to the connection manager. This will help the client to work through proxies which don't support/allow HTTP 1.1 and/or pipelining.

Summer of Code Update

Time for an update on my progress with my project, now we are halfway through GSoC...

Disconnection (either by error, or as requested by the gloox client) gracefully from the server is working now. I have also made some small improvements in other areas where necessary, too many and too small to list here.

A big problem I am having is that there are regular delays in my test setup. I have a test client for gloox that replies to messages it receives. I have tried both tkabber and Psi as the second client, with the same results.

I checked that my BOSH code is doing everything correctly and it seems the answer is (insert disclaimer here) yes.

Here is the link to a network capture log: Wireshark capture file

The file is so named because the test message I sent was the text "ooo". (No, I don't know why...)

Below is my translation to English of the necessary parts of the log, showing a message going from Psi to gloox (via the server) and the reply going back again:

A: GUI client (Psi 0.10, port 32921->5222)
B: BOSH client (gloox svn, port 45069->5280)
S: Jabber server (Openfire 3.3.2, port 5222)
CM: BOSH connection manager (Openfire 3.3.2, 5280)

# GUI client sends message to server
0 sec: A->S (frame 37 in the capture)
0.000030: TCP ACK for the above

# Server sends message to gloox via BOSH
0.001999: CM->B

# Gloox sends composing event
0.003065: (Composing event) B->CM
0.003090: TCP ACK for the above

# Gloox immediately after this sends reply message
0.003358: B->CM
0.003370: TCP ACK for the above

# Server sends composing event to GUI client
0.004707: (Composing event) S->A
0.004725: TCP ACK for the above

# Server replies with empty response to gloox,due to 30s inactivity (as per spec)
30.007330: (Empty response) CM->B

# Server sends message to GUI client
30.012078: S->A

# All the acks for the last packets
30.012078: TCP ACK for last TCP ACK (RTT to ACK was 30.007 seconds) (??????!)
30.012106: TCP ACK for S->A
30.046637: TCP ACK for (Empty response) CM->B

If anyone could explain why the server appears to hold a message for 30 seconds, please let me know!

The story with ejabberd (which I finally managed to patch, compile and build) isn't good either. Because the new version of the BOSH XEP does not define how to restart a stream, XEP-0206 support is needed, which now defines this function. Unfortunately no server supports this yet, so I disabled SASL until I find a way around this. Unfortunately ejabberd forces clients that advertise XMPP 1.0 to use SASL (which gloox does) so it denies login attempts.

Detecting which version of the BOSH protocol is supported by the server, and using the old method when necessary would work in this case, but not in Openfire's (it reports version 1.6 already).

There are a few server-independent connection managers around, and I will test with those. My intention is to find one that I am able to bring up to date with the current specifications.

This is my focus at the moment, but very soon I will begin implementing multiple connections (a big step) for when pipelining can not be used.

Summer of Code Update

This post is somewhat delayed, for a variety of reasons, and from today I'll hopefully be posting more regularly again.

Despite the lack of posts, I have still been working, and I have made some progress too.

Code for pipelining (a single connection) to the BOSH connection manager is roughly in place. My first message was sent through BOSH last week!

I do have a couple of problems, both of which I believe are down to Openfire's lack of full BOSH support.

SASL authentication will not work, because Openfire does not support the xmpp:restart attribute defined in XEP-0206. I have temporarily overcome this by disabling SASL authentication attempts in the simple client I am testing with. I will see how I can make the code backwards-compatible with the old HTTP binding in this respect*.

The second problem is that Openfire disconnects the client if it sends requests too frequently.

As I understand XEP-0124 however, if no requests are outstanding (and hence no way for the server to push data to the client) the server should allow a request within any amount of time. Although the latest version of the XEP is more clear about this, version 1.5 (on which Openfire's support is based) does not explicitly state it. As a result Openfire is being more restrictive than it should be. I intend to file a bug report for this.

My last problem was a bug I encountered in gloox (resulting in message replies not always getting sent), and it affected the test client I have that is using my BOSH class. It happens that the bug was already fixed, and simply merging with the latest gloox trunk fixed it.

My current and next tasks are to fix an intermittent segmentation fault, and an out-of-range exception (which I believe to be both related), and improve the handling of disconnecting from the server.

* This is not helped by Openfire currently reporting the wrong protocol version...

Vote!

I just found this site, The Webware 100 in which Jabber appears next to many other networks and IM-related websites.

Do a good deed, and if you haven't already, vote for Jabber! :D

(That is, if you think it deserves the vote of course. If you don't, let me know and I'll correct you...)

Update: Whoops, I linked to the wrong category!

Summer of Code - Week 1 summary

As promised I shall give some update on my progress with the project so far.

We held the second weekly meeting, though with Kev temporarily in stpeter's seat - logs here.

I finally have a clear picture of what I am doing and where I am going.

As I said in the meeting, my project is not something I can make fancy screencasts from. On the other hand, I think I will try and produce some diagrams that show what BOSH is, and how it works. I realise that this is not clear to many people who might be reading this yet :)

Coding
Down to the important part. Today I finished my first steps and managed to get my initial code to compile (through the gloox makefile). I haven't finished the simple test client I will be using, so I can't actually test it yet. If I could however, it would hang immediately after connection... but it's surely a start! :)

Once I get some real code I think I will start producing tarballs of my progress. I am currently working in a local SVN repository, so it is not hard for me to do.

Back to work...

First week of Summer of Code

The coding period of Google Summer of Code has officially started (since the 28th of May). A meeting was held on Tuesday for students and mentors working on Jabber projects.

So far...

The laptop

I have set up my new laptop for development, including development tools (subversion, SciTE), and server that I will be using to test my code against.

The server

I have chosen Openfire to be my reference server initially, although obviously I will be testing to ensure compatibility with all servers later on (just in case...). Openfire is easy to install and use, and includes XEP-0124 support by default.

Other implementations

I am interested in finding other clients or their developers that have XEP-0124 implemented already. It would help me make the final decisions this week about the way I will be implementing it. If you know something about it, or know someone who does, please contact me!

The code and protocol

Gloox thankfully already has a nice framework for adding new network-level connection methods. It is this way that HTTP proxy and SOCKS5 support has already been added. Thanks to Jakob's foresight it is also possible to chain multiple classes so that BOSH will be able to work though a SOCKS5 proxy, for example.

I need to create a class descended from ConnectionBase and write implementations of the pure functions from it. I will also have to descend from ConnectionDataHandler to parse the replies from the BOSH connection manager, before gloox is able to parse and handle the XML.

The main difficulty I have right now is deciding which BOSH parameters should be fixed in the implementation, and which should be configurable. I am also looking to see if I can make the connection degrade gracefully from HTTP pipelining to multiple connections, to polling, depending upon what is supported/available in the current environment.

Identifying possible client environments is also one of my tasks. This way I won't end up, for example, only supporting cases where the connection manager is at the same address as the XMPP server (which in the setup I will be testing with, it is).

From now on

I will be posting weekly updates to this blog, and I have decided I will aim for every Tuesday, before or after the weekly meeting. Watch this space!

PS. In the past couple of weeks I have seen BOSH spelt as BISH, NOSH and BOX. Stop it! :D

Tessa IM, the Jabber client

jdev@conference.jabber.org, 29/9/2005:

[10:17:33] I think there is space for 1 more Jabber client
[10:17:36] In C++
[10:17:46] Like Miranda, but stable, and Jabber-only

Finally, a year and a half later, I am writing it myself :)

It's called Tessa IM, and no, I don't really know how the name came to be. But neither do Miranda IM, theirs :)

It is designed in quite the same way as Miranda, but avoiding some weaknesses Miranda has. Some of the most obvious ones are inflexible plugin handling, and the fact that Miranda only runs on Windows.

Although using C++, Tessa employs Lua to bring even more flexibility, and will be cross-platform, by having the ability to be independent of the GUI used. Although we are starting with wxWidgets, it makes possible people writing Qt, Win32, ncurses, telnet interfaces... quite limitless.

Although the design will allow for multiple network plugins, like Miranda, officially (for now) only Jabber will be supported. For this we will use gloox, a great library which supports more XEPs than many clients do now.

If you are interested in joining development (several people have offered help already) then comment here, email (mwild1@gmail.com) me, Jabber me (mwild1@jaim.at), or post to the development group. We also have a MUC at tessa@conference.jabber.org!

Google Summer of Code

Dear Applicant,

Congratulations! This email is being sent to inform you that your
application was accepted to take part in the Summer of Code.

Yay! I applied to work on gloox, a C++ Jabber library, under the hood of the XMPP Software Foundation as the mentoring organisation.

This project is important to me in many ways. Firstly I know I need to get back to C/C++ coding, of which I have not done much for quite some time. Second, my task is all about adding XMPP-over-HTTP support, which is currently my only way around my ISP's restrictions. Gloox is also the library that Tessa (my new Jabber client, more coming later) uses.

All this, plus the (much needed) payment, and being able to add this to my CV, all is good :)

I will be posting development progress updates to this blog, tagged with 'Jabber' and 'GSoC' (view) (rss).

Some other interesting projects I really look forward to seeing:

• Data Form Designer Suite for XMPP (Jabber)
  
• Jingle Audio/Video for Gajim (Jabber)
  
• Revision Controlled Home Directories (Ubuntu)
  
• Mouse Gesture Recognition for the Desktop (Ubuntu)
  
• Proposal for the Development of a Graphical BZW 2.0 Editor (BZFlag)
• Interactive Wiki Maintenance with Jabber/XMPP (MoinMoin Wiki)