ROS2 Topic Re-Publisher segfaults

[doisyg]

Environment:
OS: Ubuntu 22.04
ROS2 Iron lastest binaries as of 02/02/2024
PlotJuggler and ros msg and pluggin lastest source as of 02/02/2024
Cyclone DDS

2024-03-02.11-42-50.mp4

$ ros2 run plotjuggler plotjuggler 
Loading compatible plugins from directory:  "/home/gd/ws_plotjuggler/install/plotjuggler_ros/lib/plotjuggler_ros"
"libDataLoadROS2.so is a DataLoader plugin"
"libDataStreamROS2.so is a DataStreamer plugin"
[WARN] [1709379774.941604065] [rclcpp]: logging was initialized more than once
"libTopicPublisherROS2.so is a StatePublisher plugin"
Number of plugins loaded:  3 

Loading compatible plugins from directory:  "/home/gd/ws_plotjuggler/install/plotjuggler/lib/plotjuggler"
"libDataLoadCSV.so is a DataLoader plugin"
"libDataLoadMCAP.so is a DataLoader plugin"
"libDataLoadULog.so is a DataLoader plugin"
"libDataStreamMQTT_Mosquitto.so is a DataStreamer plugin"
"libDataStreamSample.so is a DataStreamer plugin"  ...disabled, unless option -t is used
"libDataStreamUDP.so is a DataStreamer plugin"
"libDataStreamWebSocket.so is a DataStreamer plugin"
"libDataStreamZMQ.so is a DataStreamer plugin"
"libParserDataTamer.so is a MessageParser plugin"
"libParserROS1.so is a MessageParser plugin"
"libParserROS2.so is a MessageParser plugin"
"libProtobufParser.so is a MessageParser plugin"
"libPublisherCSV.so is a StatePublisher plugin"
QtAV 1.13.0(Mar 17 2022, 15:37:53)
Multimedia framework base on Qt and FFmpeg.
Distributed under the terms of LGPLv2.1 or later.
Shanghai, China Copyright (C) 2012-2019 Wang Bin (aka. Lucas Wang) wbsecg1@gmail.com
Donate: http://qtav.org/donate.html
Source: https://github.com/wang-bin/QtAV
Home page: http://qtav.org
"libPublisherVideoViewer.so is a StatePublisher plugin"
Type conversion already registered from type QString to type QwtText
"libToolboxFFT.so is a Toolbox plugin"
"libToolboxLuaEditor.so is a Toolbox plugin"
Type conversion already registered from type QString to type QwtText
"libToolboxQuaternion.so is a Toolbox plugin"
"libToolboxRemote.so is a Toolbox plugin"
Number of plugins loaded:  17 

Loading compatible plugins from directory:  "/home/gd/.local/share/PlotJuggler"
Number of plugins loaded:  0 

The rosbag loaded the data in  2210  milliseconds
Stack trace (most recent call last):
#31   Source "../sysdeps/nptl/libc_start_call_main.h", line 58, in __libc_start_call_main [0x7f1722029d8f]
#30   Source "/home/gd/ws_plotjuggler/src/PlotJuggler/plotjuggler_app/main.cpp", line 486, in main [0x555f83195110]
        483:   manager_message.get(request_message);
        484: 
        485: 
      > 486:   return app.exec();
        487: }
#29   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.3", at 0x7f1722ac0cf3, in QCoreApplication::exec()
#28   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.3", at 0x7f1722ab875a, in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)
#27   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.3", at 0x7f1722b130b7, in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)
#26   Object "/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.7200.4", at 0x7f172191e3e2, in g_main_context_iteration
#25   Object "/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.7200.4", at 0x7f1721976257, in 
#24   Object "/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.7200.4", at 0x7f1721920d3a, in g_main_context_dispatch
#23   Object "/usr/lib/x86_64-linux-gnu/libQt5XcbQpa.so.5.15.3", at 0x7f171c4bfd6d, in 
#22   Object "/usr/lib/x86_64-linux-gnu/libQt5Gui.so.5.15.3", at 0x7f1722f16a2b, in QWindowSystemInterface::sendWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>)
#21   Object "/usr/lib/x86_64-linux-gnu/libQt5Gui.so.5.15.3", at 0x7f1722f41306, in QGuiApplicationPrivate::processMouseEvent(QWindowSystemInterfacePrivate::MouseEvent*)
#20   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.3", at 0x7f1722ab9e39, in QCoreApplication::notifyInternal2(QObject*, QEvent*)
#19   Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f172376c712, in QApplicationPrivate::notify_helper(QObject*, QEvent*)
#18   Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f17237cbfd4, in 
#17   Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f17237c8d3f, in 
#16   Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f1723772e46, in QApplicationPrivate::sendMouseEvent(QWidget*, QMouseEvent*, QWidget*, QWidget*, QWidget**, QPointer<QWidget>&, bool, bool)
#15   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.3", at 0x7f1722ab9e39, in QCoreApplication::notifyInternal2(QObject*, QEvent*)
#14   Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f1723774363, in QApplication::notify(QObject*, QEvent*)
#13   Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f172376c712, in QApplicationPrivate::notify_helper(QObject*, QEvent*)
#12   Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f17237af4ed, in QWidget::event(QEvent*)
#11   Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f17238641e6, in QAbstractButton::mouseReleaseEvent(QMouseEvent*)
#10   Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f1723864089, in 
#9    Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f1723873954, in QCheckBox::nextCheckState()
#8    Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f1723863ced, in QAbstractButton::setChecked(bool)
#7    Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f1723862675, in 
#6    Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.3", at 0x7f1723862175, in QAbstractButton::toggled(bool)
#5    Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5.15.3", at 0x7f1722af1792, in 
#4    Source "/home/gd/ws_plotjuggler/src/plotjuggler-ros-plugins/src/TopicPublisherROS2/publisher_ros2.cpp", line 112, in setEnabled [0x7f16f4324a85]
        109:       _topics_to_publish.insert( {info.topic_name, true} );
        110:     }
        111: 
      > 112:     updatePublishers();
        113: 
        114:     if (!_tf_broadcaster)
        115:     {
#3    Source "/home/gd/ws_plotjuggler/src/plotjuggler-ros-plugins/src/TopicPublisherROS2/publisher_ros2.cpp", line 66, in updatePublishers [0x7f16f4323b94]
         63:     auto publisher_it = _publishers.find(info.topic_name);
         64:     if (publisher_it == _publishers.end())
         65:     {
      >  66:       _publishers.insert({ info.topic_name, GenericPublisher::create(*_node, info.topic_name, info.type) });
         67:     }
         68:   }
#2  | Source "/home/gd/ws_plotjuggler/src/plotjuggler-ros-plugins/src/TopicPublisherROS2/generic_publisher.h", line 57, in make_shared<GenericPublisher, rclcpp::node_interfaces::NodeBaseInterface*, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const rosidl_message_type_support_t&>
    | >  57:     return std::make_shared<GenericPublisher>(node.get_node_base_interface().get(), topic_name, *type_support);
    |    58:   }
    | Source "/usr/include/c++/11/bits/shared_ptr.h", line 879, in allocate_shared<GenericPublisher, std::allocator<GenericPublisher>, rclcpp::node_interfaces::NodeBaseInterface*, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const rosidl_message_type_support_t&>
    |   877:       typedef typename std::remove_cv<_Tp>::type _Tp_nc;
    |   878:       return std::allocate_shared<_Tp>(std::allocator<_Tp_nc>(),
    | > 879: 				       std::forward<_Args>(__args)...);
    |   880:     }
    | Source "/usr/include/c++/11/bits/shared_ptr.h", line 863, in shared_ptr<std::allocator<GenericPublisher>, rclcpp::node_interfaces::NodeBaseInterface*, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const rosidl_message_type_support_t&>
    |   862:       return shared_ptr<_Tp>(_Sp_alloc_shared_tag<_Alloc>{__a},
    | > 863: 			     std::forward<_Args>(__args)...);
    |   864:     }
    | Source "/usr/include/c++/11/bits/shared_ptr.h", line 409, in __shared_ptr<std::allocator<GenericPublisher>, rclcpp::node_interfaces::NodeBaseInterface*, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const rosidl_message_type_support_t&>
    |   407:       template<typename _Alloc, typename... _Args>
    |   408: 	shared_ptr(_Sp_alloc_shared_tag<_Alloc> __tag, _Args&&... __args)
    | > 409: 	: __shared_ptr<_Tp>(__tag, std::forward<_Args>(__args)...)
    |   410: 	{ }
    | Source "/usr/include/c++/11/bits/shared_ptr_base.h", line 1342, in __shared_count<GenericPublisher, std::allocator<GenericPublisher>, rclcpp::node_interfaces::NodeBaseInterface*, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const rosidl_message_type_support_t&>
    |  1340:       template<typename _Alloc, typename... _Args>
    |  1341: 	__shared_ptr(_Sp_alloc_shared_tag<_Alloc> __tag, _Args&&... __args)
    | >1342: 	: _M_ptr(), _M_refcount(_M_ptr, __tag, std::forward<_Args>(__args)...)
    |  1343: 	{ _M_enable_shared_from_this_with(_M_ptr); }
    | Source "/usr/include/c++/11/bits/shared_ptr_base.h", line 650, in _Sp_counted_ptr_inplace<rclcpp::node_interfaces::NodeBaseInterface*, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const rosidl_message_type_support_t&>
    |   648: 	  auto __guard = std::__allocate_guarded(__a2);
    |   649: 	  _Sp_cp_type* __mem = __guard.get();
    | > 650: 	  auto __pi = ::new (__mem)
    |   651: 	    _Sp_cp_type(__a._M_a, std::forward<_Args>(__args)...);
    |   652: 	  __guard = nullptr;
    | Source "/usr/include/c++/11/bits/shared_ptr_base.h", line 519, in construct<GenericPublisher, rclcpp::node_interfaces::NodeBaseInterface*, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const rosidl_message_type_support_t&>
    |   517: 	  // _GLIBCXX_RESOLVE_LIB_DEFECTS
    |   518: 	  // 2070.  allocate_shared should use allocator_traits<A>::construct
    | > 519: 	  allocator_traits<_Alloc>::construct(__a, _M_ptr(),
    |   520: 	      std::forward<_Args>(__args)...); // might throw
    |   521: 	}
    | Source "/usr/include/c++/11/bits/alloc_traits.h", line 516, in construct<GenericPublisher, rclcpp::node_interfaces::NodeBaseInterface*, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const rosidl_message_type_support_t&>
    |   514: 	{
    |   515: #if __cplusplus <= 201703L
    | > 516: 	  __a.construct(__p, std::forward<_Args>(__args)...);
    |   517: #else
    |   518: 	  std::construct_at(__p, std::forward<_Args>(__args)...);
    | Source "/usr/include/c++/11/ext/new_allocator.h", line 162, in GenericPublisher
    |   160: 	construct(_Up* __p, _Args&&... __args)
    |   161: 	noexcept(std::is_nothrow_constructible<_Up, _Args...>::value)
    | > 162: 	{ ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
    |   163: 
    |   164:       template<typename _Up>
      Source "/home/gd/ws_plotjuggler/src/plotjuggler-ros-plugins/src/TopicPublisherROS2/generic_publisher.h", line 32, in create [0x7f16f4328d71]
         29: #ifdef ROS_HUMBLE
         30:   : rclcpp::PublisherBase(node_base, topic_name, type_support, rcl_publisher_get_default_options())
         31: #else
      >  32:   : rclcpp::PublisherBase(node_base, topic_name, type_support, rcl_publisher_get_default_options(), callbacks_, true)
         33: #endif
         34:   {}
#1    Object "/opt/ros/iron/lib/librclcpp.so", at 0x7f172418dad6, in rclcpp::PublisherBase::PublisherBase(rclcpp::node_interfaces::NodeBaseInterface*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rosidl_message_type_support_t const&, rcl_publisher_options_s const&, rclcpp::PublisherEventCallbacks const&, bool)
#0    Object "/opt/ros/iron/lib/librclcpp.so", at 0x7f17240f0273, in 
Segmentation fault (Signal sent by the kernel [(nil)])
[ros2run]: Segmentation fault

[fbwogn]

Same problem occurs. Did you solve it?

[nachovizzo]

I'm also facing this issue now 👀

[nachovizzo]

So I couldn't find the root cause, but it's unlikely a plotjuggler problem.

What I did to hack this in the meantime to use this functionality is to comment out this line in rclcpp
https://github.com/ros2/rclcpp/blob/8de4b90512469d74802871e6c88db2e3135720e7/rclcpp/src/rclcpp/publisher_base.cpp#L55

event_callbacks_(event_callbacks)

This line invokes a copy construction of each callback object defined within the PublisherEventCallbacks event_callbacks_; member, this is defined as follow (in https://github.com/ros2/rclcpp/blob/8de4b90512469d74802871e6c88db2e3135720e7/rclcpp/include/rclcpp/event_handler.hpp#L15):

using QOSDeadlineOfferedInfo = rmw_offered_deadline_missed_status_t;
using QOSLivelinessLostInfo = rmw_liveliness_lost_status_t;
using QOSOfferedIncompatibleQoSInfo = rmw_offered_qos_incompatible_event_status_t;
using IncompatibleTypeInfo = rmw_incompatible_type_status_t;
using MatchedInfo = rmw_matched_status_t;

using QOSDeadlineOfferedCallbackType = std::function<void (QOSDeadlineOfferedInfo &)>;
using QOSLivelinessLostCallbackType = std::function<void (QOSLivelinessLostInfo &)>;
using QOSOfferedIncompatibleQoSCallbackType = std::function<void (QOSOfferedIncompatibleQoSInfo &)>;
using IncompatibleTypeCallbackType = std::function<void (IncompatibleTypeInfo &)>;
using PublisherMatchedCallbackType = std::function<void (MatchedInfo &)>;
/// Contains callbacks for various types of events a Publisher can receive from the middleware.
struct PublisherEventCallbacks
{
  QOSDeadlineOfferedCallbackType deadline_callback;
  QOSLivelinessLostCallbackType liveliness_callback;
  QOSOfferedIncompatibleQoSCallbackType incompatible_qos_callback;
  IncompatibleTypeCallbackType incompatible_type_callback;
  PublisherMatchedCallbackType matched_callback;
};

When trying to copy the GenericPublisher::callbacks_ into the rclcpp base publisher, the segfault does happen in the copy constructor of the std::function objects, so there is something fishy there

plotjuggler-ros-plugins/src/TopicPublisherROS2/generic_publisher.h

Line 61 in d0692ad

rclcpp::PublisherEventCallbacks callbacks_;

As reference, the implementation of the gcc11 copy constructor of std::function:

/**
 *  @brief %Function copy constructor.
 *  @param __x A %function object with identical call signature.
 *  @post `bool(*this) == bool(__x)`
 *
 *  The newly-created %function contains a copy of the target of
 *  `__x` (if it has one).
 */
function(const function& __x) : _Function_base()
{
  if (static_cast<bool>(__x))
  {
    __x._M_manager(_M_functor, __x._M_functor, __clone_functor); // <---- crashes here!!!
    _M_invoker = __x._M_invoker;
    _M_manager = __x._M_manager;
  }
}

[facontidavide]

I haven't been maintaining the re-publishing in the last... years.

Among all the features I have, it is very low on my priority list. Help appreciated

[ShkarupaDC]

Hi! I have also encountered this issue and came up with one more ugly solution:

1. Do not add the callbacks_ field as an attribute of the GenericPublisher
2. Create the PublisherEventCallbacks from scratch when initializing the PublisherBase by passing {} instead of the callbacks_

@facontidavide, could you please explain why you added the callbacks_ field to the GenericPublisher? And how ugly is my solution?