Split to separate distribution AnyEvent::MQTT5

[nuclight]

Thanks for the great work!

But, could you please split MQTT client and broker to separate distribution? There are currently no MQTT 5.0 modules on CPAN, except of yours, and for one-machine / no Web/JSONRPC setup - using entire Beekeper for just toy broker seems a significant overhead.

[jmico]

I've always had in mind splitting the distribution into submodules, but it is not a trivial thing to do.

ToyBroker, as its name hints, was never intended to be a full fledged broker. It is part of the test suite of Beekeeper and implements just the bare minimum functionality needed to run Beekeeper applications. Eventually it might be part of a Beekeeper::Test distribution.

I also planned to put the MQTT client into its own distribution, inside the AnyEvent::MQTT namespace. In order to do that I need to create a comprehensive test suite, implement the version 3 of the protocol, and handle the differences in protocol implementations between popular brokers. It is doable, but as i said, it is not a trivial thing to do.

What are you trying to do? Beekeeper is a framework for building distributed applications. If you are only interested in ToyBroker, I strongly suggest to use Mosquitto instead, which is a pretty good broker, well documented and maintained, and easy to install.

If you are interested only in the client you can use it stand-alone, as Beekeeper::MQTT is already a self contained module (see the documentation):

use Beekeeper::MQTT;

my $mqtt = Beekeeper::MQTT->new(
    host     => 'localhost',
    username => 'guest',
    password => 'guest',
);
  
$mqtt->connect( 
    blocking => 1,
);
  
$mqtt->publish(
    topic   => 'foo/bar',
    payload => 'Hello',
);

[nuclight]

I've forgot to pull explanations from our e-mail conversation to this ticket, so I'll do this now (better late than never) for ability of public references, and I'll also add reference to https://github.com/nuclight/tkxgram/blob/master/docs/db.md which don't (yet) explicitly describes this idea of MQTT5 using (but does so implicitly):

Subject: Re: Split to separate distribution AnyEvent::MQTT5
Date: Fri, 24 Dec 2021 03:30:31 +0300
X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; amd64-portbld-freebsd11.4)

At Thu, 23 Dec 2021 05:21:39 -0300
José Micó <jose.mico@gmail.com> wrote:

> I am not sure that Beekeeper is the proper tool to solve your use
> case.  

Yes, I know, 
 
> Let me know what are you trying to do, and I will be glad to help.  

Well, this is a long story, take a cup of tea or coffee...

There is a messenger called Telegram, https://telegram.org (recently
reached 2^31 threshold for their user IDs). And it is internally very,
VERY weird. They formally have open-source clients and even
documentation, but in reality it is hard to even understand, not even
to say "implement", without deep diving to code of their clients, which
is even published with several months after release...

So in fact most alternative clients just use their code (published
lib), making only cosmetic changes (GUI, etc.).

About three years ago, Telegram started to deploy changes which regain
control from users over the received messages. As this is a "cloud"
messenger, where everything, like in Web, stored on their servers, it's
easy. First it was ability to delete messages in peer's inbox for him,
not only for self. From the very last changes - ability to forbid
saving media, copypaste and even screenshots.

While from some points of view this is viewable as censorship-like,
consider following situation: someone threatens to kill you in private
chat, you want to go to police, but then they wipe it, and you have no
evidence anymore. I've heard of real such cases, with other messengers,
though - when lost phone meant lost correspondence.

And as you probably know, that in just decade or two ago Internet was
different - nobody could revoke from you logs (mail, chats, etc.) which
you received! This was absolutely normal, but now they want to take
away our freedom.

So I with my former teamlead wrote a library in Perl for Telegram, from
scratch. It works, I have a logger running on my machine, putting
machine-readable dumps (my fork is at
https://github.com/nuclight/teleperl).

However, it's not very handy to dig by Data::Dumper, also when you have
been banned from some chats, etc. So I want to write a GUI client for
this, which - important! - would have also work on casual user's
machine. No administration, no configuring brokers or database schema's
upgrades... BTW, this means the only DB suitable is SQLite (deployed in
every iron and toaster, very well tested, 100% no DBA needed).

So this is a very challenging and complex task. Apart from DB side,
where there is need to invent something to keep historical data forever
while avoiding schema upgrades/downgrades, challenging by itself (could
see at https://sqlite.org/forum/forumpost/053105c7ae96fb47 if curious,
no one was able to point to ready solution for such tasks) - there are
other problems. For example, our decoder from their binary
serialization format produced classes in Telegram::* namespace. Then,
suddenly, their server started to produce undeserializable data. We
found that these were blobs from different schema version, in violation
of negotiated version. And one can't have different but same named
classes like Telegram::Message in same Perl process - so I developed a
big crutch with several processes (AnyEvent::Fork::RPC) for
unmarshalling - whichever process with it's version who can, then it
responds.

This year this server bug disappeared, but who knows...

"Well", I thought some time later, "if I'm forced to several processes,
I can make benfits from it!". First, it reminded me of Quassel IRC
client which is split to server part (core) and GUI part, and possibly
several GUIs, e.g. from mobile (were connectivity loss is often so it's
not good fro IRC), can connect to same core.

Then I thought about plugins, and may be support of other protocols, or
other code/libraries, which are not always support async or even could
be not very reliable, but crash of separate process will not take down
entire messenger (you know, both major browsers, Chrome and Firefox,
are multiprocess for this reason, too).

Then in multiprocess environment one need to some kind of protocol to
communicate them. I've heard about MQTT earlier, but recently
discovered that version 5.0 ideally fits for my task - it has user
properties and response topics! For example, I could put a user
property like "next-response-topic" in addition of simply
response-topic, receiver will process and shift from this array to
response-topic, thus allowing for plugins to form a chain of processing
like pipes in Unix! And to deal with Telegram brainfarts: for example,
when a GUI will make an "RPC" call, the DB process could subscribe to
subtree of response topics for GUI - thus intercepting answer, too,
processing it for info about users, chats, etc. to save - transparently
for GUI.

Now remember that client should be simple enough for casual user,
running on Windows machine. This means self-contained app, no frightening
Mosquitto downloadin, install, configuration, etc. May be write several
simple lines for plugins of messenger itself (if advanced user), but no
other software.

And then I've found Beekeper. Obviously, it is solution for another
class of tasks. But toy broker perfectly fits for one-machine (say, 3-4
processes) unattended installation, QoS>1 is also not needed inside one
host. Of course, at the end i could just copy paste to modules and patch
for working without Beekeper, but keeping forks is generally not a good
thing in Open Source world.

I've looked to about other parts of Beekeper, but it also seems to be
targeted to Linux only, so again not...

As for version 3/testing. Note that on CPAN there are several other
MQTT (and some are even on AnyEvent) modules, which just put a
disclaimer about unstable API subject to change Work In Progress etc.
Thus, I think, putting just two modules, toy broker and client, with
such disclaimer - are just fine to start with.

So, that's answer - why.

-- 
WBR, tg://@nuclight