Instability of multiple drones

[bjarnemoro]

Hey!

We have raised issues before in the crazyradio git: bitcraze/crazyradio2-firmware#16 regarding latency issues and poor performance when flying several drones. We are aiming to fly at least 8 drones consistently for our experiments but we are encountering very inconsistent behaviour where drones are highly unstable to the point of falling out of the sky. Its not a problem of individual drones as they have very stable behavior. Here we are trying to gather some data when flying with just 4 crazyflies. As of now we can see that there are still issues running carzyflies with crazyswarm, but it is difficult identify the main cause.

Here are the steps we took in details :

Hardware

• 4 crazyflie 2.1 drones (https://www.bitcraze.io/products/old-products/crazyflie-2-1/)
• 2 crazyradio 2.0 (https://www.bitcraze.io/products/crazyradio-2-0/
• qualisys motion capture system

Experiment

We are trying to run a simple takeoff and landing using crazyswarm.

We did 3 different scenario's:

• 1 crazyflie 1 radio
• 4 crazyflies 2 radios
• 4 crazyflies 4 radios

Here is a video for every single scenario
single drone : https://drive.google.com/file/d/1k6e7Pfz4_PJG0ILDVteUVzCd5s6B-Ki6/view?usp=sharing
2 radios https://drive.google.com/file/d/1zmNe8fyUhyCd37PuxBqvplPWrIRQlSjd/view?usp=sharing
4 radios : https://drive.google.com/file/d/1f6G0XbwZ7jcpMMa6BAEFWPa4FAzuCJ2m/view?usp=sharing

Data

Below you will find a zip file with our ros bags containing the status topic for each crazyflie. We plotted the latency for the different scenario's, in the plots the takeoff command is send around 5 seconds.
single drone :

2 radios:

4 radios :

here is the full ros bag for each experiment containing the status topic for each crazyflie: tests.zip

Final Remarks

So here we have been trying many things and we are getting a bit discouraged so I would like to ask you in general if you have any advice in terms of :

1. Which backend should we prefer between cflib and cpp?
2. Any configuration of the radios that could help remove the latency issue or at least reduce it. It seems very severe most of the times.
3. Is the frequency of the mocap a big issue? we are sending messages at 100Hz from qualysis. Is this a problem? should we change it ?

Thanks a lot for all the help and all the support your are giving through the channel both on the crazyflie hardware side and the crazyswarm software side. It is very appreciated.

Best

[whoenig]

We have tested 8 cfs with a single Crazyradio without such issues, so this is really strange.

1. I think for the number of drones you want to use the cpp backend. This also has automatic prioritization of motion capture packets, which would help you. Your screenshots are likely cflib backend, because there are no broadcasts. Also make sure to use either Crazyradio PA, or the firmware 5.0 for Crazyradio 2 (as mentioned in the other issue that you have linked). The latency will go up with the number of drones. You can find a plot for the different backends at https://github.com/IMRCLab/crazyswarm2/blob/joss/docs2/paper/latency.png. So generally anything above 40ms for 4 drones would be "abnormal".
2. It's best if you test with a single radio only as this is what we tested so far. Make sure that all CFs use the same channel, but different addresses. 8 drones should not be a problem for one radio.
3. We typically use 100Hz and the measurements from 1 were obtained with that update rate. Of course increasing it will require more bandwidth and increase latency.

[gregoriomarchesini]

I had the chance to update the repo and the firmware with @bjarnemoro, and now I see the performance is good! Thanks a lot @whoenig for your great help.

I still have one last issue with the CrazyRadios. I receive an error when I try to use multiple radios. Namely, if I launch my config file as

  cf03:
    enabled: true
    # enabled: false
    uri: radio://0/80/2M/E7E7E7E703
    initial_position: [ 0.147, 0.155, 0.11]
    type: cf21

  cf02:
    # enabled: true
    enabled: true
    uri: radio://0/80/2M/E7E7E7E702
    initial_position: [ 0.162, -0.47, 0.0]
    type: cf21

  cf09: # das hatte heute schon funktioniert
    # enabled: true
    enabled: true
    uri: radio://0/80/2M/E7E7E7E709
    initial_position: [ -0.478, -0.47, 0.0]
    type: cf21

  cf01:
    enabled: true
    # enabled: false
    uri: radio://0/80/2M/E7E7E7E701
    initial_position: [ -0.478, 0.175, 0.1]
    type: cf21  # see robot_types

then all is good. But if I do

  cf03:
    enabled: true
    # enabled: false
    uri: radio://0/80/2M/E7E7E7E703
    initial_position: [ 0.147, 0.155, 0.11]
    type: cf21

  cf02:
    # enabled: true
    enabled: true
    uri: radio://1/80/2M/E7E7E7E702
    initial_position: [ 0.162, -0.47, 0.0]
    type: cf21

  cf09: # das hatte heute schon funktioniert
    # enabled: true
    enabled: true
    uri: radio://0/80/2M/E7E7E7E709
    initial_position: [ -0.478, -0.47, 0.0]
    type: cf21

  cf01:
    enabled: true
    # enabled: false
    uri: radio://1/80/2M/E7E7E7E701
    initial_position: [ -0.478, 0.175, 0.1]
    type: cf21  # see robot_types

Then I receive an error that channel 80 is already busy for radio 0, and I can't use it. So my question is, how can I use different radios at the same time? Is there any other parameter I should use?

Thanks a lot.

[whoenig]

We just updated the firmware of Crazyradio2 and Crazyswarm2 again and you should see once more a significant reduction in latency (see https://github.com/bitcraze/crazyradio2-firmware/releases/tag/5.1 for measurements we did). We did operate 8 drones with a single radio without problems (mocap at 100 Hz).

For your question: Since radios send data at the same time, they need to be on different channels. So if you'd like to use multiple radios, you have to change the channel (at least 2 apart, e.g., cf03, cf09 on channel 80 and cf02, cf01 on channel 82). You can change the channel in cfclient, similar to how you changed the address.