Effective robots and autonomous systems must be trustworthy. This chapter examines models of trustworthiness from a philosophical and empirical perspective to inform the design and adoption of autonomous systems. Trustworthiness is a property of trusted agents or organisations that engenders trust in other agent or organisations. Trust is a complex phenomena defined differently depending on the discipline. This chapter aims to bring different approaches under a single framework for investigation with three sorts of questions: Who (...) or what is trustworthy?–metaphysics. How do we know who or what is trustworthy?–epistemology. What factors influence what or who should we trust?–normativity. A two-component model of trust is used that incorporates competence (skills, reliability and experience) and integrity (motives, honesty and character). It is supposed that human levels of competence yield the highest trust whereas trust is reduced at sub-human and super-human levels. The threshold for trustworthiness of an agent or organisation in a particular context is a function of their relationship with the truster and potential impacts of decisions. Building trustworthy autonomous systems requires obeying the norms of logic, rationality and ethics under pragmatic constraints–even though there is disagreement on these principles by experts. Autonomous systems may need sophisticated social identities including empathy and reputational concerns to build human-like trust relationships. Ultimately transdisciplinary research drawing on metaphysical, epistemological and normative human and machine theories of trust are needed to design trustworthy autonomous systems for adoption. (shrink)

Prior research has demonstrated that trust in robots and performance of robots are two important factors that influence human–autonomy teaming. However, other factors may influence users’ perceptions and use of autonomous systems, such as perceived intent of robots and decision authority of the robots. The current study experimentally examined participants’ trust in an autonomous security robot (ASR), perceived trustworthiness of the ASR, and desire to use an ASR that varied in levels of decision authority and benevolence. Participants (...) (N = 340) were recruited from Amazon Mechanical Turk. Results revealed the participants had increased trust in the ASR when the robot was described as having benevolent intent compared to self-protective intent. There were several interactions between decision authority and intent when predicting the trust process, showing that intent may matter the most when the robot has discretion on executing that intent. Participants stated a desire to use the ASR in a military context compared to a public context. Implications for this research demonstrate that as robots become more prevalent in jobs paired with humans, factors such as transparency provided for the robot's intent and its decision authority will influence users’ trust and trustworthiness. (shrink)

Robotic and Autonomous Agricultural Technologies (RAAT) are increasingly available yet may fail to be adopted. This paper focusses specifically on cognitive factors that affect adoption including: inability to generate trust, loss of farming knowledge and reduced social cognition. It is recommended that agriculture develops its own framework for the performance and safety of RAAT drawing on human factors research in aerospace engineering including human inputs (individual variance in knowledge, skills, abilities, preferences, needs and traits), trust, situational awareness and cognitive (...) load. The kinds of cognitive impacts depend on the RAATs level of autonomy, ie whether it has automatic, partial autonomy and autonomous functionality and stage of adoption, ie adoption, initial use or post-adoptive use. The more autonomous a system is, the less a human needs to know to operate it and the less the cognitive load, but it also means farmers have less situational awareness about on farm activities that in turn may affect strategic decision-making about their enterprise. Some cognitive factors may be hidden when RAAT is first adopted but play a greater role during prolonged or intense post-adoptive use. Systems with partial autonomy need intuitive user interfaces, engaging system information, and clear signaling to be trusted with low level tasks; and to compliment and augment high order decision-making on farm. (shrink)

Technological developments involving robotics and artificial intelligence devices are being employed evermore in elderly care and the healthcare sector more generally, raising ethical issues and practical questions warranting closer considerations of what we mean by “care” and, subsequently, how to design such software coherently with the chosen definition. This paper starts by critically examining the existing approaches to the ethical design of care robots provided by Aimee van Wynsberghe, who relies on the work on the ethics of care by Joan (...) Tronto. In doing so, it suggests an alternative to their non-principled approach, an alternative suited to tackling some of the issues raised by Tronto and van Wynsberghe, while allowing for the inclusion of two orientative principles. Our proposal centres on the principles of autonomy and vulnerability, whose joint adoption we deem able to constitute an original revision of a bottom-up approach in care ethics. Conclusively, the ethical framework introduced here integrates more traditional approaches in care ethics in view of enhancing the debate regarding the ethical design of care robots under a new lens. (shrink)

Autonomous robots are increasingly interacting with users who have limited knowledge of robotics and are likely to have an erroneous mental model of the robot’s workings, capabilities, and internal structure. The robot’s real capabilities may diverge from this mental model to the extent that one might accuse the robot’s manufacturer of deceiving the user, especially in cases where the user naturally tends to ascribe exaggerated capabilities to the machine (e.g. conversational systems in elder-care contexts, or toy robots in (...) child care). This poses the question, whether misleading or even actively deceiving the user of an autonomous artifact about the capabilities of the machine is morally bad and why. By analyzing trust, autonomy, and the erosion of trust in communicative acts as consequences of deceptive robot behavior, we formulate four criteria that must be fulfilled in order for robot deception to be morally permissible, and in some cases even morally indicated. (shrink)

A historical sketch of Autonomous Systems Research (ASR) is presented to show its roots in cybernetics, AI, Robotics, Cognitive Science, and in theoretical biology. These connections are considered in the light of the epistemology of human observers as a special kind of agents modeling other systems as representing and eventually realising autonomy. It is argued that ASR must primarily be understood as an opposition to traditional AI style ‘disembodied’ robotics, and that contemporary ASR provides a partial shift (...) of focus towards the organismic roots of cognition. Finally, it is suggested that a deeper understanding of situated autonomous systems as being a kind of complex self-organizing semiotic agents with emergent qualitative properties can be gained from the broad field of theoretical biology, including recent perspectives from Artificial Life, Evolutionary Systems, the Developmental Systems Approach, and Biosemiotics. Inspired by theoretical biology we suggest to investigate a new notion of autonomy that includes a qualitative aspect of the organism. (shrink)

Over the past decade, significant progress in artificial intelligence (AI) has spurred the adoption of its algorithms, addressing previously daunting challenges. Alongside these remarkable strides, there has been a simultaneous increase in model complexity and reliance on opaque AI models, lacking transparency. In numerous scenarios, the systems themselves may necessitate making decisions entailing ethical dimensions. Consequently, it has become imperative to devise solutions to integrate ethical considerations into AI system development practices, facilitating broader utilization of AI systems across (...) various domains. Research endeavors should explore innovative approaches to enhance ethical principles in AI systems, fostering greater transparency, accountability, and trustworthiness. Upholding fundamental individual rights, human dignity, autonomy, privacy, equality, and fairness, while mitigating potential harm, remains paramount. Considering ethical values and ensuring compliance with ethical requirements throughout the development lifecycle of autonomous and intelligent systems nurture trust and reliability in their utilization. Ethical considerations should be ingrained within organizational procedures guiding AI research activities, establishing robust frameworks that address ethical concerns and reflect the ethical implications of AI-based systems. This paper presents an overview of ethical approaches and processes aimed at integrating ethical considerations into AI system development practices. It underscores the significance of ethical frameworks in fostering ethical AI implementation and ensuring the ethical integrity of AI technologies. (shrink)

The underlying thesis of the research in ethical autonomy for lethal autonomous unmanned systems is that they will potentially be capable of performing more ethically on the battlefield than are human soldiers. In this article this hypothesis is supported by ongoing and foreseen technological advances and perhaps equally important by an assessment of the fundamental ability of human warfighters in today's battlespace. If this goal of better-than-human performance is achieved, even if still imperfect, it can result in a (...) reduction in noncombatant casualties and property damage consistent with adherence to the Laws of War as prescribed in international treaties and conventions, and is thus worth pursuing vigorously. (shrink)

Artificial life embodies a recent and important conceptual step in modem science: asserting that the core of intelligence and cognitive abilities is the same as the capacity for living. The recent surge of interest in artificial life has pushed a whole range of engineering traditions, such as control theory and robotics, beyond classical notions of goal and planning into biologically inspired notions of viability and adaptation, situatedness and operational closure. These proceedings serve two important functions: they address bottom-up theories of (...) artificial intelligence and explore what can be learned from simple models such as insects about the cognitive processes and characteristic autonomy of living organisms, while also engaging researchers and philosophers in an exciting examination of the epistemological basis of this new trend. Topics Artificial Animals • Genetic Algorithms • Autonomous Systems • Emergent Behaviors • Artificial Ecologies • Immunologic Algorithms • Self-Adapting Systems • Emergent Structures • Emotion And Motivation • Neural Networks • Coevolution • Fitness Landscapes Contributors H. Bersini, Domenico Parisi, Rodney A. Brooks, Christopher G. Langton, S. Kauffman, J.-L. Denenbourg, Pattie Maes, John Holland, T. Smithersm H. Swefel, H. Muhlenbein. (shrink)

Autonomous weapons systems (AWS), sometimes referred to as “killer robots”, are receiving evermore attention, both in public discourse as well as by scholars and policymakers. Much of this interest is connected with emerging ethical and legal problems linked to increasing autonomy in weapons systems, but there is a general underappreciation for the ways in which existing law might impact on these new technologies. In this paper, we argue that as AWS become more sophisticated and increasingly more capable (...) than flesh-and-blood soldiers, it will increasingly be the case that such soldiers are “in the power” of those AWS which fight against them. This implies that such soldiers ought to be considered hors de combat, and not targeted. In arguing for this point, we draw out a broader conclusion regarding hors de combat status, namely that it must be viewed contextually, with close reference to the capabilities of combatants on both sides of any discreet engagement. Given this point, and the fact that AWS may come in many shapes and sizes, and can be made for many different missions, we argue that each particular AWS will likely need its own standard for when enemy soldiers are deemed hors de combat. We conclude by examining how these nuanced views of hors de combat status might impact on meaningful human control of AWS. (shrink)

The concept of “autonomy”, once at the core of the original enactivist proposal in The Embodied Mind, is nowadays ignored or neglected by some of the most prominent contemporary enactivists approaches. Theories of autonomy, however, come to fill a theoretical gap that sensorimotor accounts of cognition cannot ignore: they provide a naturalized account of normativity and the resources to ground the identity of a cognitive subject in its specific mode of organization. There are, however, good reasons for the contemporary neglect (...) of autonomy as a relevant concept for enactivism. On the one hand, the concept of autonomy has too often been assimilated into autopoiesis and the implications are not always clear for a dynamical sensorimotor approach to cognitive science. On the other hand, the foundational enactivist proposal displays a metaphysical tension between the concept of operational closure, deployed as constitutive, and that of structural coupling ; making it hard to reconcile with the claim that experience is sensorimotorly constituted. This tension is particularly apparent when Varela et al. propose Bittorio as a model of the operational closure of the nervous system as it fails to satisfy the required conditions for a sensorimotor constitution of experience. It is, however, possible to solve these problems by re-considering autonomy at the level of sensorimotor neurodynamics. Two recent robotic simulation models are used for this task, illustrating the notion of strong sensorimotor dependency of neurodynamic patterns, and their networked intertwinement. The concept of habit is proposed as an enactivist building block for cognitive theorizing, re-conceptualizing mental life as a habit ecology, tied within an agent’s behaviour generating mechanism in coordination with its environment. Norms can be naturalized in terms of dynamic, interactively self-sustaining, coherentism. This conception of autonomous sensorimotor agency is put in contrast with those enactive approaches that reject autonomy or neglect the theoretical resources it has to offer for the project of naturalizing minds. (shrink)

In October 2011, the “2nd European Network for Cognitive Systems, Robotics and Interaction”, EUCogII, held its meeting in Groningen on “Autonomous activity in real-world environments”, organized by Tjeerd Andringa and myself. This is a brief personal report on why we thought autonomy in real-world environments is central for cognitive systems research and what I think I learned about it. --- The theses that crystallized are that a) autonomy is a relative property and a matter of degree, b) (...) increasing autonomy of an artificial system from its makers and users is a necessary feature of increasingly intelligent systems that can deal with the real-world and c) more such autonomy means less control but at the same time improved interaction with the system. (shrink)

In this article, I shall examine some of the issues and questions involved in the technology of autonomous robots, a technology that has developed greatly and is advancing rapidly. I shall do so with reference to a particularly critical field: autonomous military robotic systems. In recent times, various issues concerning the ethical implications of these systems have been the object of increasing attention from roboticists, philosophers and legal experts. The purpose of this paper is not to (...) deal with these issues, but to show how the autonomy of those robotic systems, by which I mean the full automation of their decision processes, raises difficulties and also paradoxes that are not easy to solve. This is especially so when considering the autonomy of those robotic systems in their decision processes alongside their reliability. Finally, I would like to show how difficult it is to respond to these difficulties and paradoxes by calling into play a strong formulation of the precautionary principle. (shrink)

Cognitive science's basic premises are under attack. In particular, its focus on internal cognitive processes is a target. Intelligence is increasingly interpreted, not as a matter of reclusive thought, but as successful agent-environment interaction. The critics claim that a major reorientation of the field is necessary. However, this will only occur when there is a distinct alternative conceptual framework to replace the old one. Whether or not a serious alternative is provided is not clear. Among the critics there is some (...) consensus, however, that this role could be fulfilled by the concept of a 'behavioral system'. This integrates agent and environment into one encompassing general system. We will discuss two contexts in which the behavioral systems idea is being developed. Autonomous Agents Research is the enterprise of building behavior-based robots. Dynamical Systems Theory provides a mathematical framework well suited for describing the interactions between complex systems. We will conclude that both enterprises provide important contributions to the behavioral systems idea. But neither turns it into a full conceptual alternative which will initiate a major paradigm switch in cognitive science. The concept will need a lot of fleshing out before it can assume that role. (shrink)

The possibility that autonomous weapons will be deployed on the battlefields of the future raises the challenge of determining who can be held responsible for how these weapons act. Robert Sparrow has argued that it would be impossible to attribute responsibility for autonomous robots' actions to their creators, their commanders, or the robots themselves. This essay reaches a much different conclusion. It argues that the problem of determining responsibility for autonomous robots can be solved by addressing it (...) within the context of the military chain of command. The military hierarchy is a system of distributing responsibility between decision makers on different levels and constraining autonomy. If autonomous weapons are employed as agents operating within this system, then responsibility for their actions can be attributed to their creators and their civilian and military superiors. (shrink)

Effective human–robot teaming increasingly requires humans to work with intelligent, autonomous machines. However, novel features of intelligent autonomous systems such as social agency and incomprehensibility may influence the human’s trust in the machine. The human operator’s mental model for machine functioning is critical for trust. People may consider an intelligent machine partner as either an advanced tool or as a human-like teammate. This article reports a study that explored the role of individual differences in the mental model (...) in a simulated environment. Multiple dispositional factors that may influence the dominant mental model were assessed. These included the Robot Threat Assessment, which measures the person’s propensity to apply tool and teammate models in security contexts. Participants were paired with an intelligent robot tasked with making threat assessments in an urban setting. A transparency manipulation was used to influence the dominant mental model. For half of the participants, threat assessment was described as physics-based ; the remainder received transparency information that described psychological cues. We expected that the physics-based transparency messages would guide the participant toward treating the robot as an advanced machine, while psychological messaging would encourage perceptions of the robot as acting like a human partner. We also manipulated situational danger cues present in the simulated environment. Participants rated their trust in the robot’s decision as well as threat and anxiety, for each of 24 urban scenes. They also completed the RoTA and additional individual-difference measures. Findings showed that trust assessments reflected the degree of congruence between the robot’s decision and situational danger cues, consistent with participants acting as Bayesian decision makers. Several scales, including the RoTA, were more predictive of trust when the robot was making psychology-based decisions, implying that trust reflected individual differences in the mental model of the robot as a teammate. These findings suggest scope for designing training that uncovers and mitigates the individual’s biases toward intelligent machines. (shrink)

Autonomous weapons are systems that, once activated, can identify, select and engage targets by themselves. Scharre (2018. Army of None: Autonomous Weapons and the Future of War. New York: Norton) has given a definition of autonomy based on three dimensions: the automatized tasks, the relationship with the human user and the sophistication of the machine’s decision-making process. Based on this definition of autonomy, this article provides an overview of systematic biases that may occur in each of these (...) three dimensions. Before the deployment of autonomous systems in war it is important to know how various biases can affect the functioning of the machines and how these biases can be compensated for. (shrink)

Human autonomy is not technical “autonomy”. That we refer to both phenomena equivocally reveals a label-transfer from human to technical contexts, i.e., metaphorization. Autonomous humans choose to submit to self-given laws (1). Machines, regardless of the metaphor, cannot shed their heteronomy (2). What is literally meant by “technical autonomy” as opposed to or interchangeable with “automation” needs to be explored in detail (3). To differentiate, multi-layered approaches to technical autonomy have been proposed (4) still revealing the unavoidable metaphorization within (...) the description and analysis of autonomous systems. This metaphorization needs to be analyzed in terms of the similarity relation at work in the conceptual transference, which teaches us that an entity (e.g., an autonomous person) is, is not, and is like another (e.g., an autonomous robot) at the same time. (shrink)

It is argued that the notion of Umwelt is relevant for contemporary discussions within theoretical biology, biosemiotics, the study of Artificial Life, Autonomous Systems Research and philosophy of biology. Focus is put on the question of whether an artificial creature can have a phenomenal world in the sense of the Umwelt notion of Jakob von Uexküll, one of the founding figures of biosemiotics. Rather than vitalism, Uexküll's position can be interpreted as a version of qualitative organicism. A historical (...) sketch of Autonomous Systems Research (ASR) is presented to show its theoretical roots and fruitful opposition to traditional AI style robotics. It is argued that these artificial systems are only partly 'situated' because they do not in the full sense of the word experience an Umwelt. A deeper understanding of truly situated autonomous systems as being a kind of complex selforganizing semiotic agents with emergent qualitative properties must be gained, not only from the broad field of theoretical biology, but also from the perspective of biosemiotics in the Uexküll tradition. The paper is thus an investigation of a new notion of autonomy that includes a qualitative aspect of the organism. This indicates that the Umwelt concept is not reducible to purely functional notions. (shrink)

Understanding the impact of robot errors in child-robot-interactions (CRI) is critical, as current technological systems are still limited and may randomly present a variety of mistakes during interactions with children. In this study we manipulate a task-based error of a NAO robot during a semi-autonomous computational thinking task implemented with the Cozmo robot. Data from 72 children aged 7–10 were analysed regarding their attitudes towards NAO (social trust, competency trust, liking, and perceived agency), their behaviour towards the robot (...) (self-disclosure, following recommendations), as well as their task performance. We did not find quantitative effects of the robot’s error on children’s self-reported attitudes, behaviour, or task performance. Age was also not significantly related to either social attitudes or behaviours towards NAO, although there were some age-related differences in task performance. Potential reasons behind the lack of statistical effects and limitations of the study with regards to the manipulation of robot errors are discussed and insights into the design of future CRI studies provided. (shrink)

This article discusses mechanisms and principles for assignment of moral responsibility to intelligent robots, with special focus on military robots. We introduce the concept autonomous power as a new concept, and use it to identify the type of robots that call for moral considerations. It is furthermore argued that autonomous power, and in particular the ability to learn, is decisive for assignment of moral responsibility to robots. As technological development will lead to robots with increasing autonomous power, (...) we should be prepared for a future when people blame robots for their actions. It is important to, already today, investigate the mechanisms that control human behavior in this respect. The results may be used when designing future military robots, to control unwanted tendencies to assign responsibility to the robots. Independent of the responsibility issue, the moral quality of robots’ behavior should be seen as one of many performance measures by which we evaluate robots. How to design ethics based control systems should be carefully investigated already now. From a consequentialist view, it would indeed be highly immoral to develop robots capable of performing acts involving life and death, without including some kind of moral framework. (shrink)

In recent years, autonomous systems have become an important research area and application domain, with a significant impact on modern society. Such systems are characterized by different levels of autonomy and complex communication infrastructures that allow for collective decision-making strategies. There exist several publications that tackle ethical aspects in such systems, but mostly from the perspective of a single agent. In this paper we go one step further and discuss these ethical challenges from the perspective of (...) an aggregate of autonomous systems capable of collective decision-making. In particular, in this paper, we propose the Caesar approach through which we model the collective ethical decision-making process of a group of actors—agents and humans, as well as define the building blocks for the agents participating in such a process, namely Caesar agents. Factors such as trust, security, safety, and privacy, which affect the degree to which a collective decision is ethical, are explicitly captured in Caesar. Finally, we argue that modeling the collective decision-making in Caesar provides support for accountability. (shrink)

Construction of a robot discoverer can be treated as the ultimate success of automated discovery. In order to build such an agent we must understand algorithmic details of the discovery processes and the representation of scientific knowledge needed to support the automation. To understand the discovery process we must build automated systems. This paper investigates the anatomy of a robot-discoverer, examining various components developed and refined to a various degree over two decades. We also clarify the notion of autonomy (...) of an artificial agent, and we discuss the ways in which machine discoverers become more autonomous. Finally we summarize the main principles useful in construction of automated discoverers and we discuss various possible limitations of automation. (shrink)

It is argued in this paper that robots are just quasi-autonomous beings, which must be understood, within an emergent systems view, as intrinsically linked to and presupposing human beings as societal creatures within a technologically mediated world. Biosemiotics is introduced as a perspective on living systems that is based upon contemporary biology but reinterpreted through a qualitative organicist tradition in biology. This allows for emphasizing the differences between an organism as a general semiotic system with vegetative and (...) self-reproductive capacities, an animal body also with sentience and phenomenal states, and higher forms of anthroposemiotic systems such as humans, machines and robots. On all three levels, representations are crucial processes. The “representationalism” invoked by critiques of cognitive science and robotics tends to focus only on simplistic notions of representation, and must be distinguished from a Peircean or biosemiotic notion of representation. Implications for theorizing about the physical, biological, animate, phenomenal and social body and their forms of autonomy are discussed. (shrink)

Among ethicists and engineers within robotics there is an ongoing discussion as to whether ethical robots are possible or even desirable. We answer both of these questions in the positive, based on an extensive literature study of existing arguments. Our contribution consists in bringing together and reinterpreting pieces of information from a variety of sources. One of the conclusions drawn is that artifactual morality must come in degrees and depend on the level of agency, autonomy and intelligence of the machine. (...) Moral concerns for agents such as intelligent search machines are relatively simple, while highly intelligent and autonomous artifacts with significant impact and complex modes of agency must be equipped with more advanced ethical capabilities. Systems like cognitive robots are being developed that are expected to become part of our everyday lives in future decades. Thus, it is necessary to ensure that their behaviour is adequate. In an analogy with artificial intelligence, which is the ability of a machine to perform activities that would require intelligence in humans, artificial morality is considered to be the ability of a machine to perform activities that would require morality in humans. The capacity for artificial (artifactual) morality, such as artifactual agency, artifactual responsibility, artificial intentions, artificial (synthetic) emotions, etc., come in varying degrees and depend on the type of agent. As an illustration, we address the assurance of safety in modern High Reliability Organizations through responsibility distribution. In the same way that the concept of agency is generalized in the case of artificial agents, the concept of moral agency, including responsibility, is generalized too. We propose to look at artificial moral agents as having functional responsibilities within a network of distributed responsibilities in a socio-technological system. This does not take away the responsibilities of the other stakeholders in the system, but facilitates an understanding and regulation of such networks. It should be pointed out that the process of development must assume an evolutionary form with a number of iterations because the emergent properties of artifacts must be tested in real world situations with agents of increasing intelligence and moral competence. We see this paper as a contribution to the macro-level Requirement Engineering through discussion and analysis of general requirements for design of ethical robots. (shrink)

Abstract This article evaluates the ?drive toward greater autonomy? in lethally-armed unmanned systems. Following a summary of the main criticisms and challenges to lethal autonomy, both engineering and ethical, raised by opponents of this effort, the article turns toward solutions or responses that defense industries and military end users might seek to incorporate in design, testing and manufacturing to address these concerns. The way forward encompasses a two-fold testing procedure for reliability incorporating empirical, quantitative benchmarks of performance in compliance (...) with formalized and programmable rules of engagement, and a conception of ?due care? in product liability. This would be designed in analogy with procedures currently followed by well-intentioned governments and militaries with their own (human) military personnel, both to ensure against failure, and to accept responsibility and compensate victims of inadvertent and unintended accidents. The procedure is designed specifically to address objections first posed by Robert Sparrow (2007) and Noel Sharkey (2007), and echoed in P.W. Singer's critically acclaimed Wired for War (2009), that lethal autonomous systems cannot be meaningfully held accountable for commission of war crimes, and thus the development, manufacture, and deployment of such systems would constitute a violation of international law. (shrink)

Recent advances in automation technology have increased the opportunity for collaboration between humans and multiple autonomous systems such as robots and self-driving cars. In research on autonomous system collaboration, the trust users have in autonomous systems is an important topic. Previous research suggests that the trust built by observing a task can be transferred to other tasks. However, such research did not focus on trust in multiple different devices but in one device or several of (...) the same devices. Thus, we do not know how trust changes in an environment involving the operation of multiple different devices such as a construction site. We investigated whether trust can be transferred among multiple different devices, and investigated the effect of two factors: the similarity among multiple devices and the agency attributed to each device, on trust transfer among multiple devices. We found that the trust a user has in a device can be transferred to other devices and that attributing different agencies to each device can clarify the distinction among devices, preventing trust from transferring. (shrink)

Autonomous vehicles, and the larger socio-technical systems that they are a part of are likely to have a deep and lasting impact on our societies. Trust is a key value that will play a role in the development of autonomous driving systems. This paper suggests that trust of autonomous driving systems will impact the ways that these systems are taken up, the norms and laws that guide them and the design of the (...) class='Hi'>systems themselves. Further to this, in order to have autonomous driving systems that are worthy of our trust, we need a superstructure of oversight and a process that designs trust into these systems from the outset. Rather than banning or avoiding all autonomous vehicles should a tragedy occur, despite these systems having some level of risk, we want resilient systems that can survive tragedies, and indeed, improve from them. I will argue that trust plays a role in developing these resilient systems. (shrink)

The UN Convention on Certain Conventional Weapons (CCW) can, on the one hand, be considered vital for the global governance process—in the sense of urging international cooperation on the ethical, developmental, and standards aspects of lethal autonomous weapon systems (LAWS). On the other hand, the CCW may also embody a global trend that does not augur well for international solidarity, namely the lack of credible and comprehensive collaboration to advance global objectives of peace and security. In 2022, a (...) majority of the 125 nations that belong to the CCW requested limits on a specific type of lethal autonomous weapons: “killer robots.” Yet, most of the major global powers—namely the United States, Russia, and China—opposed not only a ban on LAWS but also on any restrictions on the development of these weapons, not least because the United States, Russia, and China are actively developing this weapons technology. While there is currently much focus on the technological evolution of LAWS, less has been written about how ethical values can exert influence on a growing global consciousness around factors such as power, technology, human judgment, accountability, autonomy, dehumanization, and the use of force. This introduction lays the groundwork for dealing with these issues. It does so by showing that all these factors warrant a pluralist approach to the global governance of LAWS, based on multiple grounds, including the military, tech, law, and distinctive theoretical-ethical orientations; the rationale being to combine this expertise into a collection for publication. Reflecting the contributing authors’ firsthand experiences of the ethics surrounding the management of LAWS to address decisive and critical questions at an expert level, it provides a framing for the collection, showing that the use of international legal mechanisms like the CCW are crucial to considering both the potential and the limits of LAWS, as well as what it can contribute to areas such as international law, human rights, and national security. (shrink)

Agents and agent-based systems are becoming essential in the development of various fields, such as artificial intelligence, ubiquitous computing, ambient intelligence, autonomous computing, and intelligent robotics. The concept of autonomous agents, inspired by the observed agency in living systems, is also central to current theories on the origin, development, and evolution of life. Therefore, it is crucial to develop an accurate understanding of agents and the concept of agency. This paper begins by discussing the role of (...) agency in natural systems as an inspiration and motivation for agential technologies and then introduces the idea of artificial agents. A systematic approach is presented for the classification of artificial agents. This classification aids in understanding the existing state of the artificial agents and projects their potential future roles in addressing specific types of problems with dedicated agent types. (shrink)

Teams are a fundamental aspect of life—from sports to business, to defense, to science, to education. While the cognitive sciences tend to focus on information processing within individuals, others have argued that teams are also capable of demonstrating cognitive capacities similar to humans, such as skill acquisition and forgetting (cf., Cooke, Gorman, Myers, & Duran, 2013; Fiore et al., 2010). As artificially intelligent and autonomous systems improve in their ability to learn, reason, interact, and coordinate with human teammates (...) combined with the observation that teams can express cognitive capacities typically seen in individuals, a cognitive science of teams is emerging. Consequently, new questions are being asked about teams regarding teamness, trust, the introduction and effects of autonomous systems on teams, and how best to measure team behavior and phenomena. In this topic, four facets of human-autonomy team cognition are introduced with leaders in the field providing in-depth articles associated with one or more of the facets: (1) defining teams; (2) how trust is established, maintained, and repaired when broken; (3) autonomous systems operating as teammates; and (4) metrics for evaluating team cognition across communication, coordination, and performance. (shrink)

Will future lethal autonomous weapon systems (LAWS), or ‘killer robots’, be a threat to humanity? The European Parliament has called for a moratorium or ban of LAWS; the ‘Contracting Parties to the Geneva Convention at the United Nations’ are presently discussing such a ban, which is supported by the great majority of writers and campaigners on the issue. However, the main arguments in favour of a ban are unsound. LAWS do not support extrajudicial killings, they do not take (...) responsibility away from humans; in fact they increase the abil-ity to hold humans accountable for war crimes. Using LAWS in war would probably reduce human suffering overall. Finally, the availability of LAWS would probably not increase the probability of war or other lethal conflict—especially as compared to extant remote-controlled weapons. The widespread fear of killer robots is unfounded: They are probably good news. (shrink)

Purpose – The purpose of this paper is to explore artificial moral agency by reflecting upon the possibility of a Moral Turing Test and whether its lack of focus on interiority, i.e. its behaviouristic foundation, counts as an obstacle to establishing such a test to judge the performance of an Artificial Moral Agent. Subsequently, to investigate whether an MTT could serve as a useful framework for the understanding, designing and engineering of AMAs, we set out to address fundamental challenges within (...) the field of robot ethics regarding the formal representation of moral theories and standards. Here, typically three design approaches to AMAs are available: top-down theory-driven models and bottom-up approaches which set out to model moral behaviour by means of models for adaptive learning, such as neural networks, and finally, hybrid models, which involve components from both top-down and bottom-up approaches to the modelling of moral agency. With inspiration from Allen and Wallace as well as Prior, we elaborate on theoretically driven approaches to machine ethics by introducing deontic tense logic. Finally, within this framework, we explore the character of human interaction with a robot which has successfully passed an MTT. Design/methodology/approach – The ideas in this paper reflect preliminary theoretical considerations regarding the possibility of establishing a MTT based on the evaluation of moral behaviour, which focusses on moral reasoning regarding possible actions. The thoughts reflected fall within the field of normative ethics and apply deontic tense logic to discuss the possibilities and limitations of artificial moral agency. Findings – The authors stipulate a formalisation of logic of obligation, time and modality, which may serve as a candidate for implementing a system corresponding to an MTT in a restricted sense. Hence, the authors argue that to establish a present moral obligation, we need to be able to make a description of the actual situation and the relevant general moral rules. Such a description can never be complete, as the combination of exhaustive knowledge about both situations and rules would involve a God eye’s view, enabling one to know all there is to know and take everything relevant into consideration before making a perfect moral decision to act upon. Consequently, due to this frame problem, from an engineering point of view, we can only strive for designing a robot supposed to operate within a restricted domain and within a limited space-time region. Given such a setup, the robot has to be able to perform moral reasoning based on a formal description of the situation and any possible future developments. Although a system of this kind may be useful, it is clearly also limited to a particular context. It seems that it will always be possible to find special cases in which a given system does not pass the MTT. This calls for a new design of moral systems with trust-related components which will make it possible for the system to learn from experience. Originality/value – It is without doubt that in the near future we are going to be faced with advanced social robots with increasing autonomy, and our growing engagement with these robots calls for the exploration of ethical issues and stresses the importance of informing the process of engineering ethical robots. Our contribution can be seen as an early step in this direction. (shrink)

ABSTRACTAutonomous weapons systems pose many challenges in complex battlefield environments. Previous discussions of them have largely focused on technological or policy issues. In contrast, we focus here on the challenge of trust in an AWS. One type of human trust depends only on judgments about the predictability or reliability of the trustee, and so are suitable for all manner of artifacts. However, AWSs that are worthy of the descriptor “autonomous” will not exhibit the required strong predictability in the (...) complex, changing contexts of war. Instead, warfighters need to develop deeper, interpersonal trust that is grounded in understanding the values, beliefs, and dispositions of the AWS. Current acquisition, training, and deployment processes preclude the development of such trust, and so there are currently no routes for a warfighter to develop trust in an AWS. We thus survey three possible changes to current practices in order to facilitate the type of deep trust that is required for appropri... (shrink)

Social robots are designed to perform intelligent, emotional, and autonomous behavior in order to establish intimate relationships with humans, for instance, in the context of elderly care. However, the imitation of qualities usually assumed to be necessary for human reciprocal interaction may impact our understanding of social interaction. Against this background, we compare the technical operations based on which social robots imitate human-like behavior with the concepts of emotionality, intelligence, and autonomy as usually attached to humans. In doing so, (...) we aim at making a conceptual contribution to the intuition that the technical imitation of emotional, intelligent, and autonomous behavior differs qualitatively from human forms of emotionality, intelligence, and autonomy. We conclude that the increasing tendency of social robots to imitate human characteristics and the thus increased human tendency of anthropomorphizing technical systems put forth new forms of human–machine interactions (HMI) that call for a critical examination. Thus, the emerging forms of interaction may lead, for instance, to false, reinforced, and perplexing assumptions regarding reciprocity. Therefore, studies are needed that critically examine the potential consequences and psychological effects that may be associated with the use of social robots, especially with regard to vulnerable groups such as children or the elderly. (shrink)

To operate in an unpredictable environment, a vehicle with advanced driving assistance systems, such as a robot or a drone, not only needs to register its surroundings but also to combine data from different sensors into a world model, for which it employs filter algorithms. Such world models, as this article argues with reference to the SLAM problem in robotics, consist of nothing other than probabilities about states and events arising in the environment. The model, thus, contains a virtuality (...) of possible worlds that are the basis for adaptive behavior. The article shows that the current development of these technologies requires new concepts because their complex adaptive behaviors cannot be explained by referring them to mere algorithmic processes. Instead, it proposes the heuristic instrument of microdecisions to designate the temporality of decisions between alternatives that are created by probabilistic procedures of world modeling. Microdecisions are more than the implementation of deterministic processes—they decide between possibilities and, thus, always open up the potential of their otherness. By describing autonomous adaptive technologies with this heuristic, the question of sovereignty inevitably arises. It forces us to re-think what autonomy means when decisions can be automated. (shrink)

This paper examines Aristotle's notion of autonomy and its implication for the mechanicism/autonomy debate. We introduce the basic principles of Aristotle's scientific framework, including his theory of four causes for the explanation of nature. We draw parallels between these notions of autonomy and causation and autopoietic theory, dynamical systems and robotics, suggesting that they may be compatible with Aristotle's framework. We argue that understanding the problem of design of autonomous robots may benefit from the consideration of integration of (...) Aristotle's causes, while robotics, in turn, may contribute to the debate providing a common ground for epistemological and ontological notions of autonomy. (shrink)

There are unusual challenges in ethics for RAS. Perhaps the issue can best be summarised as needing to consider “technically informed ethics”. The technology of RAS raises issues that have an ethical dimension, and perhaps uniquely so due to the possibility of moving human decision-making which is implicitly ethically informed to computer systems. Further, if seeking solutions to these problems – ethically aligned design, to use the IEEE’s terminology – then the solutions must be technically meaningful, capable of realisation, (...) capable of assurance, and suitable as a basis for regulation. Thus, ethics for RAS is a rich, complex multi-disciplinary concern, and perhaps more complex than many other ethical issues facing society today. It is also fast-moving. This paper has endeavoured to give an accessible introduction to some of the key issues, noting that many of them are quite subtle, and it is not possible to do them full justice in such a short document. However, we have sought to counterbalance this by giving an extensive list of initiatives, standards, etc. that focus on ethics of RAS and AI, see Annex A. (shrink)

Focused on city-scale automation, and using self-driving cars (SDCs) as a case study, this article reflects on the role of AI—and in particular, computer vision systems used for mapping and navigation—as a catalyst for urban transformation. Urban research commonly presents AI and cities as having a one-way cause-and-effect relationship, giving undue weight to AI’s impact on cities and overlooking the role of cities in shaping AI. Working at the intersection of data science and social research, this paper aims to (...) counter this trend by exploring the reverse perspective: how do cities affect the development, and expose the present limits, of SDCs? The contribution of this paper is threefold. First, by comparing urban and nonurban environments and thoroughly examining the relationship between computer vision and city-specific sociality and form, it defines machine autonomy/automation as a function of the sociotechnical milieu in which an AI system operates. Second, and related, the paper problematizes the notion of SDCs as autonomous technologies and the role it plays in envisioning contending policy arrangements and technical solutions for achieving full driving automation. Finally, the article offers insight into a materialist and spatialized understanding of AI—namely, not as an abstract quality susceptible to replication within discrete machines, but rather as a distributed property emerging through embodied interactions among a multiplicity of agents (human, non-human, and technological) within/with their environments. (shrink)

A socio-cognitive approach to trust can help us envisage a notion of networked trust for multi-agent systems (MAS) based on different interacting agents. In this framework, the issue is to evaluate whether or not a socio-cognitive analysis of trust can apply to the interactions between human and autonomous agents. Two main arguments support two alternative hypothesis; one suggests that only reliance applies to artificial agents, because predictability of agents’ digital interaction is viewed as an absolute value and human (...) relation is judged to be a necessary requirement for trust. The other suggests that trust may apply to autonomous agents because predictability of agents’ interaction is viewed only as a relative value since the digital normativity that grows out of the communication process between interacting agents in MAS has always deal with some unpredictable outcomes (_reduction of uncertainty_). Furthermore, human touch is not judged to be a necessary requirement for trust. In this perspective, a diverse notion of trust is elaborated, as trust is no longer conceived only as a relation between interacting agents but, rather, as a relation between cognitive states of control and lack of control (_double bind_). (shrink)

The use of artificial intelligence in healthcare contexts is highly controversial for the (bio)ethical conundrums it creates. One of the main problems arising from its implementation is the lack of transparency of machine learning algorithms, which is thought to impede the patient’s autonomous choice regarding their medical decisions. If the patient is unable to clearly understand why and how an AI algorithm reached certain medical decision, their autonomy is being hovered. However, there are alternatives to prevent the negative impact (...) of AI opacity in shared (healthcare professional–patient) decision-making processes, and benefit from the high accuracy of such systems. Despite these ethical issues, this paper advocates for its usage and also defends the positive role that it could have in enhancing patients’ autonomy. I advocate for a system where the implementation of AI is only partial, and humans will remain in control of oversighting algorithm interpretation of medical images and patients’ data. (shrink)

Self-trust is a necessary condition of personal autonomy and self-respect. Self-trust involves a positive sense of the motivations and competence of the trusted person; a willingness to depend on him or her; and an acceptance of vulnerability. It does not preclude trust in others. A person may be rightly said to have too much self-trust; however core self-trust is essential for functioning as an autonomous human being.

Human organizations’ adoption of the paradigm of the Fourth Industrial Revolution is associated with the growth of techno-empowerment, which is the process of transferring autonomy in decision-making to intelligent machines. Particular persuasive strategies have been identified that may coax people to use intelligent devices. However, there is a substantial research gap regarding what antecedents influence human intention to assign decision-making autonomy to artificial agents. In this study, ethological and evolutionary concepts are applied to explain the drivers for autonomous assistants’ (...) techno-empowerment. The method used in the study was a 4 × 2 between-subject experiment made with 278 persons. The research tool used to collect the data was an online survey. The results show that more positive attitudes and higher trust, perceived usefulness, and perceived ease of use are correlated with higher intention to allow the autonomous assistant independence in decision-making. Second, the results suggest that the more human-like a non-human agent is, the higher the intention to empower it—but only if this agent simultaneously provides functional and visual anthropomorphic cues explainable by the mimicry effect. (shrink)

Can autonomous systems replace humans in the performance of their activities? How does the answer to this question inform the design of autonomous systems? The study of technical systems and their features should be preceded by the study of the activities in which they play roles. Each activity can be described by its overall goals, governing norms and the intermediate steps which are taken to achieve the goals and to follow the norms. This paper uses (...) the activity realist approach to conceptualize autonomous systems in the context of human activities. By doing so, it first argues for epistemic and logical conditions that illustrate the limitations of autonomous systems in tasks which they can and cannot perform, and then, it discusses the ramifications of the limitations of system autonomy on the design of autonomous systems. (shrink)

The question of human control is a key concern in autonomous military systems debates. Our research qualitatively and quantitatively investigates values and concerns of the general public, as they relate to autonomous military systems, with particular attention to the value of human control. Using participatory value evaluation (PVE), we consulted 1980 Australians about which values matter in relation to two specific technologies: an autonomous minesweeping submarine and an autonomous drone that can drop bombs. Based (...) on value sensitive design, participants were tasked to enhance the systems with design features that can realize values. A restriction (limited budget) in each design task forced participants to make trade-offs between design options and the values that these options realize. Our results suggest that the ‘general public’ has diverse and nuanced stances on the question of human control over autonomous military systems. A third of participants that is opposed to autonomous military systems when asked directly, selected different combinations of design features realizing varying degrees of human control. Several contextual factors, technology-specific concerns, and certain values seemed to explain these different choices. Our research shows that a focus on human control might overlook other important values that the general public is concerned about, such as system reliability, verifiability, and retrievability. (shrink)

This paper discusses the epistemic status of biology from the standpoint of the systemic approach to living systems based on the notion of biological autonomy. This approach aims to provide an understanding of the distinctive character of biological systems and this paper analyses its theoretical and epistemological dimensions. The paper argues that, considered from this perspective, biological systems are examples of emergent phenomena, that the biological domain exhibits special features with respect to other domains, and that biology (...) as a discipline employs some core concepts, such as teleology, function, regulation among others, that are irreducible to those employed in physics and chemistry. It addresses the claim made by Jacques Monod that biology as a science is marginal. It argues that biology is general insofar as it constitutes a paradigmatic example of complexity science, both in terms of how it defines the theoretical object of study and of the epistemology and heuristics employed. As such, biology may provide lessons that can be applied more widely to develop an epistemology of complex systems. (shrink)