Change Proposal: Reorganizing MQTT Topics

[timothybrown]

Okay, so I'm currently working on rewriting the HA Discovery code from the ground up. Part of this requires me to basically redo the MQTT topics. Right now it's kind of a mess and none of the topics are ideal for Home Assistant (or other home automation platforms).

The first step is to change the topic names and their payload. Here is what I've come up with:

• Power Status Topic
  Device Topic/switch => on|off
• Power Command Topic
  Device Topic/switch/set <= on|true|1|off|false|0
• Color Status Topic
  Device Topic/color => R,G,B
• Color Command Topic
  Device Topic/color/set <= #HEX|R,G,B
• Brightness Status Topic
  Device Topic/brightness => 0-255
• Brightness Command Topic
  Device Topic/brightness/set <= 0-255
• Effect Status Topic
  Device Topic/fx => FX Number
• Effect Command Topic
  Device Topic/fx/set <= FX Number|FX Name
• API Response Topic
  Device Topic/api => XML API Response
• API Command Topic
  Device Topic/api/set <= HTTP API Format
• Availability Topic
  Device Topic/status => online|offline

This puts the topic names in-line with other projects (Espurna, ESPHome, etc.) and makes it easier to integrate into custom code and various home automation platforms without resorting to complex templating or XML decoding (however, because we've retained the API topic, this can still be done for advanced users).

Judging by a look through the issue list, it appears most people are using MQTT for HA Discovery support, which means this change will be transparent for those users (i.e., nothing will break for them and it will just work). This will only be a breaking change for those who manually configure HA, use other home automation platforms or use MQTT as a transport for custom software. I feel this is a very small percentage of use cases and this change will ultimately make it easier to use for them.

I'm interested to hear from anyone using the MQTT support in a non-HA setting (i.e., with custom code, other automation platforms and so forth). Would these topics be sufficient for your use? Is there anything else you'd like to see?

Comments from others (contributors, users and @Aircoookie alike) are welcomed as well!

I've currently got the basics of this implemented in my development fork (not yet published) and it's working well, but I wanted to get a discussion going before I go any further with it.

Edit: Renamed the power topic to switch and added availability to the list of topics. (When the device connects to the broker it will set the status topic to online then register what’s called a “last will”, which allows the broker to take care of setting the status topic to offline for us when the device disconnects.)

[timothybrown]

I should also add that, because of the way I've structured my rewrite, when @Aircoookie finishes support for segments, I'll be able to add segment support into the MQTT code as well. What this means is you could have each segment show up as a separate light in Home Assistant, or you could easily control each segment through MQTT using your custom code. The idea is, each segment would append a number onto the base device topic:

Device Topic/0/[switch|color|brightness|fx|api|status]
Device Topic/0/[switch|color|brightness|fx|api]/set
Device Topic/1/[switch|color|brightness|fx|api|status]
Device Topic/1/[switch|color|brightness|fx|api]/set
Device Topic/2/[switch|color|brightness|fx|api|status]
Device Topic/2/[switch|color|brightness|fx|api]/set

This has a lot of possibilities!

[brentbrooks70]

@timothybrown This would ease my implementation of segment control from the backend platform(QIOT Suite) I am using greatly going forward. I think it would actually make the entire system easier to work with via MQTT.

1. In the power. or now switch, command should there be a 2 (toggle) option?
   1a. I guess you could always use the Device Topic/api/set with a T2 so maybe not?
2. I know this `might be a little off-topic (maybe needs its own issue) but is there any possibility of exposing the COOLING setting via MQTT (or HTTP API or both). I know this is only used by one FX but I have 2 lights that use the fire_2012 FX as the norm and I really miss not having that setting in WLED. I don't care if it is exposed on the GUI or not but, I would even prefer it to the speed control for that FX
3. Again, this could be done Device Topic/api/set FP but exposing palette control may not be bad either
4. My only other thought would be (again probably needs an issue of its own) to change responses from XML to JSON, I'm having difficulty in getting the MQTT messages parsed properly in QIOT Suite as it always expects JSON for some stupid reason.

Sorry if it seems like I'm rambling, sort of thinking (out loud) in writing.

Thanks for your work on this and asking for opinions before just making changes!!

[Aircoookie]

Hi,
thank you for ewriting the MQTT integration! I absolutely agree, the topics I chose were quite arbitrary. This is often a problem when implementing something you are not an expert with. I definitely agree with bringing the implementation in line with other projects and easing the integration into HA and similar services, so I am very grateful for your help! Your choice of topics seems to make sense for me, this is a lot more transparent and easy to use than the current system. Maybe we can add speed, intensity and palette, but that could also be done at a later stage.

In fact I haven't been in contact with anyone who used MQTT outside of the home automation use case. HA is by far the most common, but there are users of openHAB and domoticz. Of course I will notify them about the breaking change, there is a high chance that your proposal will benefit the complexity of their implementation as well though!

About segments, the ESP basics are already implemented, just the Javascript UI and a few finishing touches are missing. I like your proposal, structuring the topics like that is very logical!
You can take a look at wled19_json.ino to get an idea how to access them. Basically, you can use the strip.getSegment(index) method to access the Segment data object, which contains all the important variables of each segment:

typedef struct Segment { // 21 bytes
      uint16_t start;
      uint16_t stop; //segment invalid if stop == 0
      uint8_t speed;
      uint8_t intensity;
      uint8_t palette;
      uint8_t mode;
      uint8_t options; //bit pattern: msb first: transitional tbd tbd tbd tbd paused reverse selected
      uint32_t colors[NUM_COLORS];
      //...
      bool isActive()
      {
        return stop > start;
      }
      uint16_t length()
      {
        return stop - start;
      }
    } segment;

strip.getMaxSegments() will be helpful, too. If you want to support setting segment bounds via MQTT (i'm not sure whether that would make sense), use strip.setSegment(index, start, stop). Let me know if there is anything you need, I'm open to changes to the internal Segment api!

A consideration to make with segment support in general is which segment's values to return via the default APIs (XML and MQTT without segment number in topic). I don't think that returning the first segment would make sense for all users, so I'm planning to make this "return segment" user configurable. It is hardcoded to 0 right now, but you can already use strip.getReturnedSegmentId()

I have two small questions:

• What about the group topics (like /wled/all)? HA probably won't use them (because I assume there are better ways to group devices). Do you think there are use cases for them or should we drop them completely? If we keep them, they should of course mirror the device topic names, but only subscribe to the /set topics, not publish anything.

• I'm a little worried that subscribing and publishing to so many topics (especially if we want to support segments) might cause performance issues or even crashing (because of memory overflow by simultaneous requests or similar). We need to make sure that doesn't happen, but for QoS 0 and local MQTT brokers, it could work out.

Thanks once again! Looking forward to hearing from you and let me know if there are any issues!

[brentbrooks70]

What about the group topics (like /wled/all)? HA probably won't use them (because I assume there are better ways to group devices). Do you think there are use cases for them or should we drop them completely? If we keep them, they should of course mirror the device topic names, but only subscribe to the /set topics, not publish anything.

@Aircoookie - IMHO you could drop the ALL, topic, every HA setup (including my own) has grouping abilities or if you really needed groups couldn't you configure a UDP port per group and the UDP sync would handle it for you?

[timothybrown]

@Aircoookie @brentbrooks70 I’ll respond to the other parts of both your posts later tonight when I get more time. I just wanted to respond to the MQTT group stuff right now while I’m thinking about it.

Yes, I kind of feel that MQTT grouping could be dropped without affecting anyone. As @brentbrooks70 mentioned, we already have the built-in UDP sync functionality that does the same thing, plus most home automation platforms have the ability to group multiple devices into a single “virtual” device.

I’ll give you a good example: My bathroom vanity has three cheap Ai-Light RGBW bulbs installed (running Espurna). Now, I could use Espurna’s MQTT Group feature to have them all listen to the same topic and only add one of them to HA, but because they support HA Discovery, it’s easier for me to simply let HA find all three bulbs and create a light group that presents them as a single virtual bulb.

In fact, I’m hard pressed to think of a setup that UDP Sync or Light Groups wouldn’t cover for WLED. (But of course I don’t know every setup, so if anyone can think of one please share!)

[aidbish]

Updating the MQTT topics to work correctly will be a big win especially for those of us running HA. Also agree that the group topic can be removed without any if at all detriment as noted above most automation software allows you to create groups.

Happy to be a willing tester of the implementation

[JAMESBOWLER]

I argree its not too hard for me to change. I new around here and use openhab mqtt for control.

Looking at the HA docs for mqtt the JSON schema alows the most features to be used. https://www.home-assistant.io/components/light.mqtt/#json-schema

[timothybrown]

@brentbrooks70

@timothybrown This would ease my implementation of segment control from the backend platform(QIOT Suite) I am using greatly going forward. I think it would actually make the entire system easier to work with via MQTT.

Yes, that was my thought as well!

1. In the power. or now switch, command should there be a 2 (toggle) option?
   1a. I guess you could always use the Device Topic/api/set with a T2 so maybe not?

Yes, I could easily add a toggle command if it would be useful for you. I had thought about it, but left it out because I couldn’t immediately come up with a use case for it. (Most home automation software will simply track the state by subscribing to the power topic, or keep track of the state itself and work in a so called optimistic mode.)

2. I know this `might be a little off-topic (maybe needs its own issue) but is there any possibility of exposing the COOLING setting via MQTT (or HTTP API or both). I know this is only used by one FX but I have 2 lights that use the fire_2012 FX as the norm and I really miss not having that setting in WLED. I don't care if it is exposed on the GUI or not but, I would even prefer it to the speed control for that FX

Please open an issue for this and I’ll look into adding it to the API! @Aircoookie may have some thoughts on it too.

3. Again, this could be done Device Topic/api/set FP but exposing palette control may not be bad either

See my reply to @Aircoookie below.

4. My only other thought would be (again probably needs an issue of its own) to change responses from XML to JSON, I'm having difficulty in getting the MQTT messages parsed properly in QIOT Suite as it always expects JSON for some stupid reason.

Yes, I planned on opening an issue about this as well! I’m new as a contributor to the project, so I wanted to get @Aircoookie’s thoughts on why he went with XML instead of JSON for responses. XML is a very verbose schema compared to JSON. We’ve already got the dependency baggage of the JSON library, so why not go all the way and have responses in JSON? It would use less memory, bandwidth and flash, plus it’s easier to encode/decode when writing custom software to control your devices.

My thought is that the JSON API is new and maybe his plan is to eventually switch over to it.

Sorry if it seems like I'm rambling, sort of thinking (out loud) in writing.

You’re not rambling at all. Personally, I like people thinking out loud like this in a development environment; it’s one of the things that make open source and communities like GitHub so great!

Thanks for your work on this and asking for opinions before just making changes!!

And thank you for the feedback, keep it coming!

@Aircoookie

Hi,
thank you for ewriting the MQTT integration! I absolutely agree, the topics I chose were quite arbitrary. This is often a problem when implementing something you are not an expert with.

No worries, at the very least you got basic MQTT support working, which is a lot better than nothing!

I definitely agree with bringing the implementation in line with other projects and easing the integration into HA and similar services, so I am very grateful for your help! Your choice of topics seems to make sense for me, this is a lot more transparent and easy to use than the current system. Maybe we can add speed, intensity and palette, but that could also be done at a later stage.

Yes, I was thinking about adding topics for speed, intensity and palette too, but I wasn’t sure how useful they would be. The Home Assistant MQTT Light component doesn’t have corresponding controls for those options (other home automation platforms appear to be similar) so they would only be useful in HA if:

1. You’re using a custom version of the MQTT Light component that you’ve added these controls to.
2. You add input_* control components (like input_select, input_boolean or input_number) and use automation rules or an AppDaemon script to map them to the relevant topics for each light.
3. You could also map speed and intensity to the white_value_ and color_temperature_ controls, since we don’t use them. (You would still have to add another component (like input_select) and write an automation rule to handle the palette for each light.)

However, in all of these cases you can do the same thing by just using the API topic. You’re already implementing custom scripts or automation rules, at which point it’s trivial to just decode and encode for the API topic.

That said, it’s trivial to add them, so if it will make it easier for someone to integrate into their platform or custom code I can add them to the list!

In fact I haven't been in contact with anyone who used MQTT outside of the home automation use case. HA is by far the most common, but there are users of openHAB and domoticz. Of course I will notify them about the breaking change, there is a high chance that your proposal will benefit the complexity of their implementation as well though!

Thanks!

About segments, the ESP basics are already implemented, just the Javascript UI and a few finishing touches are missing. I like your proposal, structuring the topics like that is very logical!
You can take a look at wled19_json.ino to get an idea how to access them. Basically, you can use the strip.getSegment(index) method to access the Segment data object, which contains all the important variables of each segment:
…
strip.getMaxSegments() will be helpful, too. If you want to support setting segment bounds via MQTT (i'm not sure whether that would make sense), use strip.setSegment(index, start, stop). Let me know if there is anything you need, I'm open to changes to the internal Segment api!

Okay, cool. I’ll start diving into this over the weekend and get a first pass implemented. I may be in contact with some questions as I go!

A consideration to make with segment support in general is which segment's values to return via the default APIs (XML and MQTT without segment number in topic). I don't think that returning the first segment would make sense for all users, so I'm planning to make this "return segment" user configurable. It is hardcoded to 0 right now, but you can already use strip.getReturnedSegmentId()

Yes, I was thinking about this earlier. I think the easiest way to handle this on the MQTT side would be to always include the segment number in the topic path. So, even if the user doesn’t use segments their topic would be Device Topic/0/Command because 0 is the default segment consisting of the entire string of LEDs.

I have two small questions:

• What about the group topics (like /wled/all)? HA probably won't use them (because I assume there are better ways to group devices). Do you think there are use cases for them or should we drop them completely? If we keep them, they should of course mirror the device topic names, but only subscribe to the /set topics, not publish anything.

See my earlier reply for more on this.

• I'm a little worried that subscribing and publishing to so many topics (especially if we want to support segments) might cause performance issues or even crashing (because of memory overflow by simultaneous requests or similar). We need to make sure that doesn't happen, but for QoS 0 and local MQTT brokers, it could work out.

I don’t think this will be a big issue, but I planned to do some performance profiling and see how far I can push it. The only place we may run into an issue is if the user has a lot of segments. In those cases they might have to use an ESP32.

The alternative is to only have a single topic per segment which would be a JSON input and output. Home Assistant’s MQTT Light component supports three types of schema: default, JSON and template. Right now I’m designing it for the default schema, where the various controls have their own topics. We could switch to the JSON schema, which basically expects a status topic that contains a JSON dictionary, likewise it sends a JSON dictionary to the command topic. Here’s an example:

{
  "brightness": 255,
  "color_temp": 155,
  "color": {
    "r": 255,
    "g": 180,
    "b": 200,
    "x": 0.406,
    "y": 0.301,
    "h": 344.0,
    "s": 29.412
  },
  "effect": "colorloop",
  "state": "ON",
  "transition": 2,
  "white_value": 150
}

So each segment would subscribe to one topic and publish to one topic:
Device Topic/Segment => Status JSON Dictionary
Device Topic/Segment/set <= Command JSON Dictionary

However, there's a trade off... These JSON payloads can be hundreds of bytes in size, so we’d need large buffers to hold the incoming data before parsing. Additionally, we’d have to up some of the TCP settings to get it to work reliably.

(You have no idea the hoops I had to jump through to get IR-Blaster to fully support MQTT. IR codes can often be 500+ characters long, that’s not even counting the overhead of putting them in a JSON container!)

It’s doable, but it’ll be a little more complex to implement than the default schema. It really all comes down to where our bottleneck is as to which will perform better.

Thanks once again! Looking forward to hearing from you and let me know if there are any issues!

And thank you for this great project! I’m happy to be able to contribute something.

[timothybrown]

I argree its not too hard for me to change. I new around here and use openhab mqtt for control.

Looking at the HA docs for mqtt the JSON schema alows the most features to be used. https://www.home-assistant.io/components/light.mqtt/#json-schema

Yes, compared to the default schema, the JSON schema gives you a flashing/strobing control and a control to set the transition time between colors. The former isn’t really applicable to WLED and I’m not sure how useful the latter would be.

If you guys would prefer the JSON schema (as described at the end my last post) I could absolutely implement it instead. Technically it would be the most flexible, it’ll just be a lot more complex to implement.

[aidbish]

i think the using the json schema would give greater flexibility going forward.

I like the idea of the speed and intensity of an effect being exposed via MQTT as this would for me enable me to set effects with certain speeds in nodered be able to use a zigbee remote to use

[brentbrooks70]

@timothybrown

I don’t think this will be a big issue, but I planned to do some performance profiling and see how far I can push it. The only place we may run into an issue is if the user has a lot of segments. In those cases they might have to use an ESP32.

When you are ready to performance test let me know, I can script some pretty rapid fire calls via QIOT to see how fast it can accept the calls and just how quickly it chews up memory for both platforms.

Very excited about these changes, WLED just keeps getting better!!!

[timothybrown]

i think the using the json schema would give greater flexibility going forward.

Yes, I tend to agree, however...

I like the idea of the speed and intensity of an effect being exposed via MQTT as this would for me enable me to set effects with certain speeds in nodered be able to use a zigbee remote to use

Keep in mind that you’d already be able to do that by just passing the correct command (for speed, intensity or palette) to the API topic. The real question is if it’s worth it to expose those commands as their own unique topics. It seems like in your case either would work, though exposing them as topics would make it simpler for you to implement.

What I’m kind of thinking about right now is maybe going with the topics I outlined in my original post, but changing the API topic to accept a JSON input and provide a JSON output. The JSON schema wouldn’t necessarily conform to the default HA JSON schema, in order to save memory, but would still expose all the available commands. This would be fine as HA would be using the default schema for all the basic functions, the JSON topic would just be there for advanced users or people like you using other platforms.

Hopefully that makes sense, I’m just thinking out loud here.

When you are ready to performance test let me know, I can script some pretty rapid fire calls via QIOT to see how fast it can accept the calls and just how quickly it chews up memory for both platforms.

Very excited about these changes, WLED just keeps getting better!!!

Awesome, this will be very helpful. I’ve only got two custom lamps that have 70 LEDs in them (that I mainly use with the fire effect), plus an additional strip of 100 on a wall as bias lighting (I’m slowly expanding that one to eventually encircle the entire room). So testing with larger arrays or even larger numbers of smaller arrays will be great. (The latter is important because network traffic can seriously slow things down, especially when you’ve got a bunch of WiFi devices all rapidly sending packets).

[Aircoookie]

@brentbrooks70
See #208 !

@timothybrown

Yes, I kind of feel that MQTT grouping could be dropped without affecting anyone.

Seems like we reached a consensus. Let's drop it! If the user only wants to connect some of the lights to the broker for some reason, they can still use UDP sync!

You could also map speed and intensity to the white_value...

That is a possibility, although the custom implementation seems more straighforward. Having white value and color temperature do something completely different would confuse new users and it still doesn't cover palettes. That said, WLED does offer RGBW support for WS2813 strips. The bool useRGBW is true if the user enabled it. The white value is either col[3] or for segments, the 8 most significant bytes of the color uint32_t. Maybe it is possible to add that topic when RGBW is enabled, not sure on that one!

So, even if the user doesn’t use segments their topic would be Device Topic/0/Command because 0 is the default segment consisting of the entire string of LEDs.

Agreed! This is a lot easier both for implementation and for the user.

On the topic of JSON vs. topic-per-property scheme types: I think you should implement this however you see fit! If there are any issues down the line we can still switch to the other type.

My thought is that the JSON API is new and maybe his plan is to eventually switch over to it.

This exactly! MQTT was implemented before the JSON API and I have not yet connected them. Since JSON has become easier to deal with in just about any modern environment, I think this reorganization is a good opportunity to provide a JSON state API topic for MQTT in addition to or instead of the XML one. This shouldn't be too much work because we can just hook to the provided functions serializeState(JsonObject) and deserializeState(JsonObject). Heres the API doc

Okay, cool. I’ll start diving into this over the weekend and get a first pass implemented. I may be in contact with some questions as I go!

Awesome, thanks!

[timothybrown]

What do you guys think about adding some general status topics to the list? These would be published under the root device topic only and not duplicated for each segment. Here's an example of what Espurna publishes:

Bathroom-Vanity-Left/app ESPURNA
Bathroom-Vanity-Left/version 1.13.5
Bathroom-Vanity-Left/board AITHINKER_AI_LIGHT
Bathroom-Vanity-Left/host Bathroom-Vanity-Left
Bathroom-Vanity-Left/ssid ********
Bathroom-Vanity-Left/ip 10.***.***.***
Bathroom-Vanity-Left/Mac **:**:**:**:**:**
Bathroom-Vanity-Left/rssi -52
Bathroom-Vanity-Left/uptime 233119
Bathroom-Vanity-Left/datetime 2019-08-24 21:15:12
Bathroom-Vanity-Left/freeheap 21944
Bathroom-Vanity-Left/vcc 3246
Bathroom-Vanity-Left/status 1
Bathroom-Vanity-Left/loadavg 1

Adding these wouldn't really take up additional resources as we wouldn't be subscribing to them and they could be updated every 60 seconds or during otherwise idle time. While HA and other platforms won't directly use them, I personally find them useful, especially for development or new installations. (I run a custom Python script that tracks this information, which I can review to make sure there's no longterm memory leaks, WiFi issues, stuff like that.)

So, this is what our list looks like now:

• Power Status Topic
  Device Topic/Segment/switch => on|off
• Power Command Topic
  Device Topic/Segment/switch/set <= on|true|1|off|false|0
• Color Status Topic
  Device Topic/Segment/color => R,G,B
• Color Command Topic
  Device Topic/Segment/color/set <= #HEX|R,G,B
• Brightness Status Topic
  Device Topic/Segment/brightness => 0-255
• Brightness Command Topic
  Device Topic/Segment/brightness/set <= 0-255
• White Value Status Topic (*For RGBW LEDs)
  Device Topic/Segment/white => 0-255
• White Value Command Topic (*For RGBW LEDs)
  Device Topic/Segment/white/set <= 0-255
• Effect Status Topic
  Device Topic/Segment/fx => FX Number
• Effect Command Topic
  Device Topic/Segment/fx/set <= FX Number|FX Name
• Palette Status Topic
  Device Topic/Segment/palette => Palette Name
• Palette Command Topic
  Device Topic/Segment/palette/set <= Palette Name
• Intensity Status Topic
  Device Topic/Segment/intensity => 0-255
• Intensity Command Topic
  Device Topic/Segment/intensity/set <= 0-255
• Speed Status Topic
  Device Topic/Segment/speed => 0-255
• Speed Command Topic
  Device Topic/Segment/speed/set <= 0-255
• API Response Topic
  Device Topic/api => JSON API Response
• API Command Topic
  Device Topic/api/set <= JSON API Format
• Status Topics
  Device Topic/fw => WLED xx.yy.zz
  Device Topic/status => online|offline
  Device Topic/vcc => Voltage in mV
  Device Topic/rssi => RSSI
  Device Topic/mac => MAC Address
  Device Topic/host => Hostname
  Device Topic/ip => IPv4 Address
  Device Topic/ssid => SSID
  Device Topic/load => Load Average
  Device Topic/mem => Free Memory
  Device Topic/uptime => Seconds Since Startup
  Device Topic/datetime => Current Date/Time

[aidbish]

Yep i like, you could make sensors in HA for things like, fw version, ip address which saves having to load the gui and could be on a glance card or more info card, someone may even make a custom card for WLED that exposes all options.