In this paper, we discuss guidelines for a reward design problem that defines when and what amount of reward should be given to the agent/s, within the context of reinforcement learning approach. We would like to take keepaway soccer as a standard task of the multiagent domain which requires skilled teamwork. The difficulties of designing reward for this task are due to its features as follows: i) since it belongs to the continuing task which (...) has no explicit goal to achieve, it is hard to tell when reward should be given to the agent/s. ii) since it is a multiagent cooperative task, it is hard to decide what is a fair share of reward for each agent's contribution to achieve the goal. Through some experiments, we show that the reward design have a major effect on the agent's behavior, and introduce the successful reward function that makes agents perform keepaway better and more interesting than the conventional one does. Finally, we explore the relationship between `reward design' and `acquired behaviors' from the viewpoint of teamwork. (shrink)

Autonomous underwater vehicles are widely used to accomplish various missions in the complex marine environment; the design of a control system for AUVs is particularly difficult due to the high nonlinearity, variations in hydrodynamic coefficients, and external force from ocean currents. In this paper, we propose a controller based on deep reinforcement learning in a simulation environment for studying the control performance of the vectored thruster AUV. RL is an important method of artificial intelligence that can learn behavior (...) through trial-and-error interactions with the environment, so it does not need to provide an accurate AUV control model that is very hard to establish. The proposed RL algorithm only uses the information that can be measured by sensors inside the AUVs as the input parameters, and the outputs of the designed controller are the continuous control actions, which are the commands that are set to the vectored thruster. Moreover, a reward function is developed for deep RL controller considering different factors which actually affect the control accuracy of AUV navigation control. To confirm the algorithm’s effectiveness, a series of simulations are carried out in the designed simulation environment, which is a method to save time and improve efficiency. Simulation results prove the feasibility of the deep RL algorithm applied to the control system for AUV. Furthermore, our work also provides an optional method for robot control problems to deal with improving technology requirements and complicated application environments. (shrink)

As a universal language in the world, English has become a necessary language communication tool under the globalization of trade. Intelligent, efficient, and reasonable English language-assisted learning system helps to further improve the English ability of language learners. English online learning dictionary, as an important query tool for English learners, is an important part of English online learning. This paper will optimize the design of English online learning dictionary system based on multiagent architecture. Based on the hybrid (...) class='Hi'>multiagent cooperative algorithm, this paper will improve the disadvantages of the online English learning dictionary system and propose an appropriate dictionary application evaluation function. At the same time, an improved reinforcement learning algorithm is introduced into the corresponding English online learning dictionary navigation problem so as to improve the efficiency of the online English learning dictionary system. English online learning dictionary is more intelligent and efficient. In this paper, the new online learning dictionary system optimization algorithm is proposed and compared with the traditional system algorithm. The experimental results show that the algorithm proposed in this paper solves the collaborative confusion problem of English learning online dictionary to a certain extent and further solves the corresponding navigation problem so as to improve the efficiency. (shrink)

A tracking control algorithm of nonlinear multiple agents with undirected communication is studied for each multiagent system affected by external interference and input saturation. A control design scheme combining iterative learning and adaptive control is proposed to perform parameter adaptive time-varying adjustment and prove the effectiveness of the control protocol by designing Lyapunov functions. Simulation results show that the high-precision tracking control problem of the nonlinear multiagent system based on adaptive iterative learning control can be well realized (...) even when the input is saturated. Finally, the validity of the proposed algorithm is verified by numerical analysis. (shrink)

A deep reinforcement learning-based computational guidance method is presented, which is used to identify and resolve the problem of collision avoidance for a variable number of fixed-wing UAVs in limited airspace. The cooperative guidance process is first analyzed for multiple aircraft by formulating flight scenarios using multiagent Markov game theory and solving it by machine learning algorithm. Furthermore, a self-learning framework is established by using the actor-critic model, which is proposed to train collision avoidance decision-making neural networks. To (...) achieve higher scalability, the neural network is customized to incorporate long short-term memory networks, and a coordination strategy is given. Additionally, a simulator suitable for multiagent high-density route scene is designed for validation, in which all UAVs run the proposed algorithm onboard. Simulated experiment results from several case studies show that the real-time guidance algorithm can reduce the collision probability of multiple UAVs in flight effectively even with a large number of aircraft. (shrink)

Can humans get arbitrarily capable reinforcement learning agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we study when an RL agent has an instrumental goal to tamper with its reward process, and describe design (...) principles that prevent instrumental goals for two different types of reward tampering. Combined, the design principles can prevent reward tampering from being an instrumental goal. The analysis benefits from causal influence diagrams to provide intuitive yet precise formalizations. (shrink)

The area of smart power grids needs to constantly improve its efficiency and resilience, to provide high quality electrical power in a resilient grid, while managing faults and avoiding failures. Achieving this requires high component reliability, adequate maintenance, and a studied failure occurrence. Correct system operation involves those activities and novel methodologies to detect, classify, and isolate faults and failures and model and simulate processes with predictive algorithms and analytics. In this paper, we showcase the application of a complex-adaptive, self-organizing (...) modeling method, and Probabilistic Boolean Networks, as a way towards the understanding of the dynamics of smart grid devices, and to model and characterize their behavior. This work demonstrates that PBNs are equivalent to the standard Reinforcement Learning Cycle, in which the agent/model has an interaction with its environment and receives feedback from it in the form of a reward signal. Different reward structures were created to characterize preferred behavior. This information can be used to guide the PBN to avoid fault conditions and failures. (shrink)

Sleep helps to protect and renew hippocampus-dependent declarative learning. Less is known about forms of learning that mainly engage the dopaminergic reward system. Animal studies showed that exogenous melatonin modulates the responses of the dopaminergic reward system and acts as a neuroprotectant promoting memory. In humans, melatonin is mainly secreted in darkness during evening hours supporting sleep. In this study, we investigate the effects of a short period of daytime sleep (nap) and endogenous melatonin on reward learning. (...) Twenty-seven healthy, adult students took part in an experiment, either taking a 90-min afternoon nap or watching videos (within-subject design). Before and after the sleep vs. wake interval, saliva melatonin levels and reward learning were measured, and in the nap condition, a polysomnogram was obtained. Reward learning was assessed using a two-alternative probabilistic reinforcement-learning task. Sleep itself and subjective arousal or valence had no significant effects on reward learning. However, this study showed for the first time that an afternoon nap can elicit a small but significant melatonin response in about 41% of the participants and that the magnitude of the melatonin response predicts subsequent reward learning. Only in melatonin responders did a short nap improve reward learning. The difference between melatonin-responders and non-responders occurred very early during learning indicating that melatonin might have improved working memory rather than reward learning. Future studies should use paradigms differentiating working memory and reward learning to clarify which aspect of human feedback learning might profit from melatonin. (shrink)

The large scale, time varying, and diversification of physically coupled networked infrastructures such as power grid and transportation system lead to the complexity of their controller design, implementation, and expansion. For tackling these challenges, we suggest an online distributed reinforcement learning control algorithm with the one-layer neural network for each subsystem or called agents to adapt the variation of the networked infrastructures. Each controller includes a critic network and action network for approximating strategy utility function and desired control (...) law, respectively. For avoiding a large number of trials and improving the stability, the training of action network introduces supervised learning mechanisms into reduction of long-term cost. The stability of the control system with learning algorithm is analyzed; the upper bound of the tracking error and neural network weights are also estimated. The effectiveness of our proposed controller is illustrated in the simulation; the results indicate the stability under communication delay and disturbances as well. (shrink)

Successfully explaining and replicating the complexity and generality of human and animal learning will require the integration of a variety of learning mechanisms. Here, we introduce a computational model which integrates associative learning (AL) and reinforcement learning (RL). We contrast the integrated model with standalone AL and RL models in three simulation studies. First, a synthetic grid‐navigation task is employed to highlight performance advantages for the integrated model in an environment where the reward structure is both diverse and (...) dynamic. The second and third simulations contrast the performances of the three models in behavioral experiments, demonstrating advantages for the integrated model in accounting for behavioral data. (shrink)

The fundamental cognitive problem for active organisms is to decide what to do next in a changing environment. In this article, we analyze motor and action control in computational models that utilize reinforcement learning (RL) algorithms. In reinforcement learning, action control is governed by an action selection policy that maximizes the expected future reward in light of a predictive world model. In this paper we argue that RL provides a way to explicate the so-called action-oriented views of (...) cognitive systems in representational terms. (shrink)

This paper suggests an online solution for the optimal tracking control of robotic systems based on a single critic neural network -based reinforcement learning method. To this end, we rewrite the robotic system model as a state-space form, which will facilitate the realization of optimal tracking control synthesis. To maintain the tracking response, a steady-state control is designed, and then an adaptive optimal tracking control is used to ensure that the tracking error can achieve convergence in an optimal sense. (...) To solve the obtained optimal control via the framework of adaptive dynamic programming, the command trajectory to be tracked and the modified tracking Hamilton-Jacobi-Bellman are all formulated. An online RL algorithm is the developed to address the HJB equation using a critic NN with online learning algorithm. Simulation results are given to verify the effectiveness of the proposed method. (shrink)

Most current online distributed machine learning algorithms have been studied in a data-parallel architecture among agents in networks. We study online distributed machine learning from a different perspective, where the features about the same samples are observed by multiple agents that wish to collaborate but do not exchange the raw data with each other. We propose a distributed feature online gradient descent algorithm and prove that local solution converges to the global minimizer with a sublinear rate O 2 T. Our (...) algorithm does not require exchange of the primal data or even the model parameters between agents. Firstly, we design an auxiliary variable, which implies the information of the global features, and estimate at each agent by dynamic consensus method. Then, local parameters are updated by online gradient descent method based on local data stream. Simulations illustrate the performance of the proposed algorithm. (shrink)

As one of the most effective medical technologies for the infertile patients, in vitro fertilization has been more and more widely developed in recent years. However, prolonged waiting for IVF procedures has become a problem of great concern, since this technology is only mastered by the large general hospitals. To deal with the insufficiency of IVF service capacity, this paper studies an IVF queuing network in an integrated cloud healthcare system, where the two key medical services, that is, egg retrieval (...) and transplantation, are assigned to accomplish in the general hospital, while the routine medical tests are assigned into the community hospital. Based on continuous-time Markov procedure, a dynamic large-scale server scheduling problem in this complicated service network is modeled with consideration of different arrival rates of multiple type of patients and different service capacities of multiple servers that can be defined as doctors of the general hospital. To solve this model, a reinforcement learning algorithm is proposed, where the reward functions are designed for four conflicting subcosts: setup cost, patient waiting cost, penalty cost for unsatisfied patient personal preferences, and medical cost of patient. The experimental results show that the optimal service rule of each server’s queue obtained by the RL method is significantly superior to the traditional service rule. (shrink)

We study a joint pricing and inventory control problem for perishables with positive lead time in a finite horizon periodic-review system. Unlike most studies considering a continuous density function of demand, in our paper the customer demand depends on the price of current period and arrives according to a homogeneous Poisson process. We consider both backlogging and lost-sales cases, and our goal is to find a simultaneously ordering and pricing policy to maximize the expected discounted profit over the planning horizon. (...) When there is no fixed ordering cost involved, we design a deep reinforcement learning algorithm to obtain a near-optimal ordering policy and show that there are some monotonicity properties in the learned policy. We also show that our deep reinforcement learning algorithm achieves a better performance than tabular-based Q-learning algorithms. When a fixed ordering cost is involved, we show that our deep reinforcement learning algorithm is effective and efficient, under which the problem of “curse of dimension” is circumvented. (shrink)

Theories of decision-making and its neural substrates have long assumed the existence of two distinct and competing valuation systems, variously described as goal-directed vs. habitual, or, more recently and based on statistical arguments, as model-free vs. model-based reinforcement-learning. Though both have been shown to control choices, the cognitive abilities associated with these systems are under ongoing investigation. Here we examine the link to cognitive abilities, and find that individual differences in processing speed covary with a shift from model-free to (...) model-based choice control in the presence of above-average working memory function. This suggests shared cognitive and neural processes; provides a bridge between literatures on intelligence and valuation; and may guide the development of process models of different valuation components. Furthermore, it provides a rationale for individual differences in the tendency to deploy valuation systems, which may be important for understanding the manifold neuropsychiatric diseases associated with malfunctions of valuation. (shrink)

The generation of the pitch control signal in a wind turbine (WT) is not straightforward due to the nonlinear dynamics of the system and the coupling of its internal variables; in addition, they are subjected to the uncertainty that comes from the random nature of the wind. Fuzzy logic has proved useful in applications with changing system parameters or where uncertainty is relevant as in this one, but the tuning of the fuzzy logic controller (FLC) parameters is neither straightforward nor (...) an easy task. On the other hand, reinforcement learning (RL) allows systems to automatically learn, and this capability can be exploited to tune the FLC. In this work, a WT pitch control architecture that uses RL to tune the membership functions and scale the output of a fuzzy controller is proposed. The RL strategy calculates the fuzzy controller gains in order to reduce the output power error of the WT according to the wind speed. Different reward mechanisms based on the output power error have been considered. Simulation results with different wind profiles show that this architecture performs better (123.7 W) in terms of power errors than an FLC without RL (133.2 W) or a simpler PID (208.8 W). Even more, it provides a smooth response and outperforms other hybrid controllers such as RL-PID and radial basis function neural network control. (shrink)

Reinforcement learning is a kind of machine learning. It aims to adapt an agent to a given environment with a clue to rewards. In general, the purpose of reinforcement learning system is to acquire an optimum policy that can maximize expected reward per an action. However, it is not always important for any environment. Especially, if we apply reinforcement learning system to engineering, environments, we expect the agent to avoid all penalties. In Markov Decision Processes, a (...) pair of a sensory input and an action is called rule. We call a rule penalty if and only if it has a penalty or it can transit to a penalty state where it does not contribute to get any reward. After suppressing all penalty rules, we aim to make a rational policy whose expected reward per an action is larger than zero. In this paper, we propose a suppressing penalty algorithm that can suppress any penalty and get a reward constantly. By applying the algorithm to the tick-tack-toe, its effectiveness is shown. (shrink)

How can one build AI systems such that they pursue the goals their designers want them to pursue? This is the alignment problem. Numerous authors have raised concerns that, as research advances and systems become more powerful over time, misalignment might lead to catastrophic outcomes, perhaps even to the extinction or permanent disempowerment of humanity. In this paper, I analyze the severity of this risk based on current instances of misalignment. More specifically, I argue that contemporary large language models and (...) game-playing agents are sometimes misaligned. These cases suggest that misalignment tends to have a variety of features: misalignment can be hard to detect, predict and remedy, it does not depend on a specific architecture or training paradigm, it tends to diminish a system’s usefulness and it is the default outcome of creating AI via machine learning. Subsequently, based on these features, I show that the risk of AI alignment magnifies with respect to more capable systems. Not only might more capable systems cause more harm when misaligned, aligning them should be expected to be more difficult than aligning current AI. (shrink)

Reinforcement learning relies on the reward prediction error (RPE) signals conveyed by the midbrain dopamine system. Previous studies showed that dopamine plays an important role in both positive and negative reinforcement. However, whether various reinforcement processes will induce distinct learning signals is still unclear. In a probabilistic learning task, we examined RPE signals in different reinforcement types using an electrophysiology index, namely, the feedback-related negativity (FRN). Ninety-four participants were randomly assigned into four groups: base (no (...) money incentive), positive reinforcement (presentation of money rewards), negative reinforcement (removal of money losses), and combined reinforcement (money rewards and removal of money losses) groups. In addition, in order to evaluate the engagement of emotional activity in the different reinforcement processes, Positive and Negative Affect Schedule-Expanded Form (PANAS-X) scales were applied before and after the experiment to detect the emotional changes. The results showed that there was no difference between groups in the dopamine-related learning bias. However, compared to the other three groups, negative reinforcement elicited smaller FRN (the difference-wave measure) during the learning, stronger positive affect and joviality, and less fatigue after the learning, in which the difference between the negative and positive reinforcement groups was smaller. The results indicated that pure avoidance motivation may induce distinct emotional fluctuations, which influence the feedback processing. (shrink)

GoalThis paper presents an immersive Virtual Reality system to analyze and train Executive Functions of soccer players. EFs are important cognitive functions for athletes. They are a relevant quality that distinguishes amateurs from professionals.MethodThe system is based on immersive technology, hence, the user interacts naturally and experiences a training session in a virtual world. The proposed system has a modular design supporting the extension of various so-called game modes. Game modes combine selected game mechanics with specific simulation content to (...) target particular training aspects. The system architecture decouples selection/parameterization and analysis of training sessions via a coaching app from an Unity3D-based VR simulation core. Monitoring of user performance and progress is recorded by a database that sends the necessary feedback to the coaching app for analysis.ResultsThe system is tested for VR-critical performance criteria to reveal the usefulness of a new interaction paradigm in the cognitive training and analysis of EFs. Subjective ratings for overall usability show that the design as VR application enhances the user experience compared to a traditional desktop app; whereas the new, unfamiliar interaction paradigm does not negatively impact the effort for using the application.ConclusionThe system can provide immersive training of EF in a fully virtual environment, eliminating potential distraction. It further provides an easy-to-use analyzes tool to compare user but also an automatic, adaptive training mode. (shrink)

Recent work in the cognitive sciences has argued that beliefs sometimes acquire signaling functions in virtue of their ability to reveal information that manipulates “mindreaders.” This paper sketches some of the evolutionary and design considerations that could take agents from solipsistic goal pursuit to beliefs that serve as social signals. Such beliefs will be governed by norms besides just the traditional norms of epistemology. As agents become better at detecting the agency of others, either through evolutionary history or individual (...) learning, the candidate pool for signaling expands. This logic holds for natural and artificial agents that find themselves in recurring social situations that reward the sharing of one’s thoughts. (shrink)

This paper presents a human-computer interaction model with a three layers learning mechanism in a pervasive environment. We begin with a discussion around a number of important issues related to human-computer interaction followed by a description of the architecture for a multi-agent cooperative design system for pervasive computing environment. We present our proposed three- layer HCI model and introduce the group formation algorithm, which is predicated on a dynamic sharing niche technology. Finally, we explore the cooperative reinforcement learning (...) and fusion algorithms; the paper closes with concluding observations and a summary of the principal work and contributions of this paper. (shrink)

In this paper, to solve the problem of slow design of arts and crafts and to improve design efficiency and aesthetics, the existing big data and 3D technology are used to conduct an in-depth analysis of the optimization of the rapid design system of arts and crafts machine salt baking. In the system requirement analysis, the functional modules of this system are identified as nine functional modules such as design terminology management system and external information import (...) function according to the actual usage requirements. In the system design, the overall structure design, database design, and functional module design of the system are comprehensively elaborated, and the key issues such as 3D display and home layout generation algorithm based on reinforcement learning are analyzed and designed. In the implementation part of the system, the overall construction of the system and the composition of functional modules are introduced in detail and the main functional modules of the system are presented with interface diagrams. In the system implementation part, the overall system construction and functional module composition are introduced in detail, the main functional modules of the system are shown with interface diagrams, codes, and algorithms, and the specific implementation process of 3D display and soft layout algorithms are also explained in detail. The process of Surface Mount Technology big data processing and analysis is designed, and the design of SMT production line data collection scheme and real-time data processing architecture is completed. Based on the characteristics of SMT production line data, the K-means algorithm is used to detect data outliers and verify the accuracy of the method; also, the Spark-based association rule printing parameter recommendation model is designed, and the efficiency of the Apriori algorithm is significantly improved by parallelization. (shrink)

Reinforcement learning is very effective for robot learning. Because it does not need priori knowledge and has higher capability of reactive and adaptive behaviors. In our previous works, we proposed new reinforcement learning algorithm: “Q-learning with Dynamic Structuring of Exploration Space Based on Genetic Algorithm (QDSEGA)”. It is designed for complicated systems with large action-state space like a robot with many redundant degrees of freedom. And we applied it to 50 link manipulator and effective behavior is acquired. However (...) optimality and fault tolerance of the proposed algorithm were not considered and to demonstrate effectiveness of the proposed algorithm other applications are necessary. Acquiring of locomotion patterns by a multi-legged robot is a very interesting problem. As it has many redundant degrees of freedom, application of usual reinforcement learning is difficult and an optimal locomotion has not been acquired using previous reinforcement learning algorithm. And the redundancy of the robot is effective to the fault tolerance and various locomotion patterns can be acquired for adapting the faults of the legs. In this paper, we applied QDSEGA to acquiring of locomotion pattern by the multi-legged robot and considered the optimality and fault tolerance. Effective behavior has been obtained by using our proposed algorithm. (shrink)

Interactions between an intelligent software agent and a human user are ubiquitous in everyday situations such as access to information, entertainment, and purchases. In such interactions, the ISA mediates the user’s access to the content, or controls some other aspect of the user experience, and is not designed to be neutral about outcomes of user choices. Like human users, ISAs are driven by goals, make autonomous decisions, and can learn from experience. Using ideas from bounded rationality, we frame these interactions (...) as instances of an ISA whose reward depends on actions performed by the user. Such agents benefit by steering the user’s behaviour towards outcomes that maximise the ISA’s utility, which may or may not be aligned with that of the user. Video games, news recommendation aggregation engines, and fitness trackers can all be instances of this general case. Our analysis facilitates distinguishing various subcases of interaction, as well as second-order effects that might include the possibility for adaptive interfaces to induce behavioural addiction, and/or change in user belief. We present these types of interaction within a conceptual framework, and review current examples of persuasive technologies and the issues that arise from their use. We argue that the nature of the feedback commonly used by learning agents to update their models and subsequent decisions could steer the behaviour of human users away from what benefits them, and in a direction that can undermine autonomy and cause further disparity between actions and goals as exemplified by addictive and compulsive behaviour. We discuss some of the ethical, social and legal implications of this technology and argue that it can sometimes exploit and reinforce weaknesses in human beings. (shrink)

There is an apparent connection between reinforcement learning and agency. Artificial entities controlled by reinforcement learning algorithms are standardly referred to as agents, and the mainstream view in the psychology and neuroscience of agency is that humans and other animals are reinforcement learners. This article examines this connection, focusing on artificial reinforcement learning systems and assuming that there are various forms of agency. Artificial reinforcement learning systems satisfy plausible conditions for minimal agency, and those which (...) use models of the environment to perform forward search are capable of a form of agency which may reasonably be called action for reasons. (shrink)

The topics treated in The brain and emotion include the definition, nature, and functions of emotion (Ch. 3); the neural bases of emotion (Ch. 4); reward, punishment, and emotion in brain design (Ch. 10); a theory of consciousness and its application to understanding emotion and pleasure (Ch. 9); and neural networks and emotion-related learning (Appendix). The approach is that emotions can be considered as states elicited by reinforcers (rewards and punishers). This approach helps with understanding the functions of (...) emotion, with classifying different emotions, and in understanding what information-processing systems in the brain are involved in emotion, and how they are involved. The hypothesis is developed that brains are designed around reward-and punishment-evaluation systems, because this is the way that genes can build a complex system that will produce appropriate but flexible behavior to increase fitness (Ch. 10). By specifying goals rather than particular behavioral patterns of responses, genes leave much more open the possible behavioral strategies that might be required to increase fitness. The importance of reward and punishment systems in brain design also provides a basis for understanding the brain mechanisms of motivation, as described in Chapters 2 for appetite and feeding, 5 for brain-stimulation reward, 6 for addiction, 7 for thirst, and 8 for sexual behavior. Key Words: amygdala; brain evolution; consciousness; dopamine; emotion; hunger; orbitofrontal cortex; punishment; reward; taste. (shrink)

In this paper, we consider profit sharing that is one of the reinforcement learning methods. An agent learns a candidate solution of a problem from the reward that is received from the environment if and only if it reaches the destination state. A function that distributes the received reward to each action of the candidate solution is called the reinforcement function. On this learning system, the agent can reinforce the set of selected actions when it gets (...) the reward. And the agent should not reinforce the detour actions. First, we will propose a new constraint equation about reinforcement functions to distribute the reinforcement values on the non-detour actions. If we use the reinforcement function to satisfy the constraint equation, the agent can select the non-detour actions directing to the destination state. Next, it is shown that the reinforcement function can be constant after learning process to suppress the selection of detour actions. Lastly, in computer simulations for maze problems, we show that the learning performance of agents does not depend on the size of environment. (shrink)

This paper investigates a distributed coordination control for a class of high-order uncertain multiagent systems. Under the framework of iterative learning control, a novel fully distributed learning protocol is devised for the coordination problem of MASs including time-varying parameter uncertainties as well as actuator saturations. Meanwhile, the learning updating laws of various parameters are proposed. Utilizing Lyapunov theory and combining with Graph theory, the proposed algorithm can make each follower track a leader completely over a limited time interval even (...) though each follower is without knowing the dynamics of the leader. Moreover, the extension to formation control is made. The validity and feasibility of the algorithm are verified conclusively by two examples. (shrink)

This paper discusses how the association learning principle works for supporting acquisition of basic spelling and reading skills using digital game-based learning environment with the Finland-based GraphoLearn technology. This program has been designed and validated to work with early readers of different alphabetic writing systems using repetition and reinforcing connections between spoken and written units. Initially GL was developed and found effective in training children at risk of reading disorders in Finland. Today GL training has been shown to support learning (...) decoding skills among children independent of whether they face difficulties resulting from educational, social, or biological reasons. (shrink)

Reinforcement Learning is a promising technique for creating agents that can be applied to real world problems. The most important features of RL are trial-and-error search and delayed reward. Thus, agents randomly act in the early learning stage. However, such random actions are impractical for real world problems. This paper presents a novel model of RL agents. A feature of our learning agent model is to integrate the Analytic Hierarchy Process into the standard RL agent model, which consists (...) of three modules: state recognition, learning, and action selecting modules. In our model, the AHP module is designed with {\\it primary knowledge} that humans intrinsically have in a process until a goal state is attained. This integration aims at increasing promising actions instead of completely random actions in the standard RL algorithms. Profit Sharing is adopted as a RL method for our model, since PS is known to be useful even in multi-agent environments. To evaluate our approach in a multi-agent environment, we test a PS RL method with our agent model on a pursuit problem in a grid world. Computational results show that our approach outperforms the standard PS in terms of learning speed in the earlier stages of learning. We also show that the learning performance of our approach is superior at least competitive to that of the standard one in the final stages of learning. (shrink)

Relational database management system is the most popular database system. It is important to maintain data security from information leakage and data corruption. RDBMS can be attacked by an outsider or an insider. It is difficult to detect an insider attack because its patterns are constantly changing and evolving. In this paper, we propose an adaptive database intrusion detection system that can be resistant to potential insider misuse using evolutionary reinforcement learning, which combines reinforcement learning and evolutionary learning. (...) The model consists of two neural networks, an evaluation network and an action network. The action network detects the intrusion, and the evaluation network provides feedback to the detection of the action network. Evolutionary learning is effective for dynamic patterns and atypical patterns, and reinforcement learning enables online learning. Experimental results show that the performance for detecting abnormal queries improves as the proposed model learns the intrusion adaptively using Transaction Processing performance Council-E scenario-based virtual query data. The proposed method achieves the highest performance at 94.86%, and we demonstrate the usefulness of the proposed method by performing 5-fold cross-validation. (shrink)

Learning in natural environments is often characterized by a degree of inconsistency from an input. These inconsistencies occur, for example, when learning from more than one source, or when the presence of environmental noise distorts incoming information; as a result, the task faced by the learner becomes ambiguous. In this study, we investigate how learners handle such situations. We focus on the setting where a learner receives and processes a sequence of utterances to master associations between objects and their labels, (...) where the source is inconsistent by design: It uses both “correct” and “incorrect” object‐label pairings. We hypothesize that depending on the order of presentation, the result of the learning may be different. To this end, we consider two types of symbolic learning procedures: the Object‐Label (OL) and the Label‐Object (LO) process. In the OL process, the learner is first exposed to the object, and then the label. In the LO process, this order is reversed. We perform experiments with human subjects, and also construct a computational model that is based on a nonlinear stochastic reinforcement learning algorithm. It is observed experimentally that OL learners are generally better at processing inconsistent input compared to LO learners. We show that the patterns observed in the learning experiments can be reproduced in the simulations if the model includes (a) an ability to regularize the input (and also to do the opposite, i.e., undermatch) and (b) an ability to take account of implicit negative evidence (i.e., interactions among different objects/labels). The model suggests that while both types of learners utilize implicit negative evidence in a similar way, there is a difference in regularization patterns: OL learners regularize the input, whereas LO learners undermatch. As a result, OL learners are able to form a more consistent system of image‐utterance associations, despite the ambiguous learning task. (shrink)

Reinforcement learning approaches to cognitive modeling represent task acquisition as learning to choose the sequence of steps that accomplishes the task while maximizing a reward. However, an apparently unrecognized problem for modelers is choosing when, what, and how much to reward; that is, when (the moment: end of trial, subtask, or some other interval of task performance), what (the objective function: e.g., performance time or performance accuracy), and how much (the magnitude: with binary, categorical, or continuous values). (...) In this article, we explore the problem space of these three parameters in the context of a task whose completion entails some combination of 36 state–action pairs, where all intermediate states (i.e., after the initial state and prior to the end state) represent progressive but partial completion of the task. Different choices produce profoundly different learning paths and outcomes, with the strongest effect for moment. Unfortunately, there is little discussion in the literature of the effect of such choices. This absence is disappointing, as the choice of when, what, and how much needs to be made by a modeler for every learning model. (shrink)

In this opinionated review, I draw attention to some of the contributions reinforcement learning can make to questions in the philosophy of mind. In particular, I highlight reinforcement learning's foundational emphasis on the role of reward in agent learning, and canvass two ways in which the framework may advance our understanding of perception and motivation.

Security issue against different attacks is the core topic of cyberphysical systems. In this paper, optimal control theory, reinforcement learning, and neural networks are integrated to provide a brief overview of optimal robust control strategies for a benchmark power system. First, the benchmark power system models with actuator and sensor attacks are considered. Second, we investigate the optimal control issue for the nominal system and review the state-of-the-art RL methods along with the NN implementation. Third, we propose several robust (...) control strategies for different types of cyberphysical attacks via the optimal control design, and stability proofs are derived through Lyapunov theory. Furthermore, the stability analysis with the NN approximation error, which is rarely discussed in the previous works, is studied in this paper. Finally, two different simulation examples demonstrate the effectiveness of our proposed methods. (shrink)

Upshot: We acknowledge that our model can be implemented with different reinforcement learning algorithms. Subsystem formation has been successfully demonstrated on the basal level, and in order to show full subsystem formation in the communication system at least both intentional utterances and acceptance/rejection need to be implemented. The comments about intrinsic vs extrinsic rewards made clear that this distinction is not helpful in the context of the constructivist paradigm but rather needs to be replaced by a critical reflection on (...) whether one has truly created autopoietic agents or just an engineering system. (shrink)

The paper presents an approach for developing multi-agent reinforcement learning systems that are made up of a coalition of modular agents. We focus on learning to segment sequences (sequential decision tasks) to create modular structures, through a bidding process that is based on reinforcements received during task execution. The approach segments sequences (and divides them up among agents) to facilitate the learning of the overall task. Notably, our approach does not rely on a priori knowledge or a priori structures. (...) Initial experiments demonstrated the basic promise of the approach. This work shows how bidding and reinforcement learning can be usefully combined, thus pointing to a new research direction. (shrink)

Animals routinely adapt to changes in the environment in order to survive. Though reinforcement learning may play a role in such adaptation, it is not clear that it is the only mechanism involved, as it is not well suited to producing rapid, relatively immediate changes in strategies in response to environmental changes. This research proposes that counterfactual reasoning might be an additional mechanism that facilitates change detection. An experiment is conducted in which a task state changes over time and (...) the participants had to detect the changes in order to perform well and gain monetary rewards. A cognitive model is constructed that incorporates reinforcement learning with counterfactual reasoning to help quickly adjust the utility of task strategies in response to changes. The results show that the model can accurately explain human data and that counterfactual reasoning is key to reproducing the various effects observed in this change detection paradigm. (shrink)

Sequential action makes up the bulk of human daily activity, and yet much remains unknown about how people learn such actions. In one motor learning paradigm, the serial reaction time (SRT) task, people are taught a consistent sequence of button presses by cueing them with the next target response. However, the SRT task only records keypress response times to a cued target, and thus it cannot reveal the full time‐course of motion, including predictive movements. This paper describes a mouse movement (...) trajectory SRT task in which the cursor must be moved to a cued location. We replicated keypress SRT results, but also found that predictive movement—before the next cue appears—increased during the experiment. Moreover, trajectory analyses revealed that people developed a centering strategy under uncertainty. In a second experiment, we made prediction explicit, no longer cueing targets. Thus, participants had to explore the response alternatives and learn via reinforcement, receiving rewards and penalties for correct and incorrect actions, respectively. Participants were not told whether the sequence of stimuli was deterministic, nor if it would repeat, nor how long it was. Given the difficulty of the task, it is unsurprising that some learners performed poorly. However, many learners performed remarkably well, and some acquired the full 10‐item sequence within 10 repetitions. Comparing the high‐ and low‐performers’ detailed results in this reinforcement learning (RL) task with the first experiment's cued trajectory SRT task, we found similarities between the two tasks, suggesting that the effects in Experiment 1 are due to predictive, rather than reactive processes. Finally, we found that two standard model‐free reinforcement learning models fit the high‐performing participants, while the four low‐performing participants provide better fit with a simple negative recency bias model. (shrink)

In reinforcement learning, a decision maker searching for the most rewarding option is often faced with the question: What is the value of an option that has never been tried before? One way to frame this question is as an inductive problem: How can I generalize my previous experience with one set of options to a novel option? We show how hierarchical Bayesian inference can be used to solve this problem, and we describe an equivalence between the Bayesian model (...) and temporal difference learning algorithms that have been proposed as models of RL in humans and animals. According to our view, the search for the best option is guided by abstract knowledge about the relationships between different options in an environment, resulting in greater search efficiency compared to traditional RL algorithms previously applied to human cognition. In two behavioral experiments, we test several predictions of our model, providing evidence that humans learn and exploit structured inductive knowledge to make predictions about novel options. In light of this model, we suggest a new interpretation of dopaminergic responses to novelty. (shrink)

This paper considers the consensus control problem of nonlinear spatial-temporal hyperbolic partial difference multiagent systems and parabolic partial difference multiagent systems with time delay. Based on the system’s own fixed topology and the method of generating the desired trajectory by introducing virtual leader, using the consensus tracking error between the agent and the virtual leader agent and neighbor agents in the last iteration, an iterative learning algorithm is proposed. The sufficient condition for the system consensus error to converge (...) along the iterative axis is given. When the iterative learning number k approaches infinity, the consensus error in the sense of the L 2 norm between all agents in the system will converge to zero. Furthermore, simulation results illustrate the effectiveness of the algorithm. (shrink)

It is often thought that there is one key design principle or at best a small set of design principles, underlying the success of biological organisms. Candidates include neural nets, ‘swarm intelligence’, evolutionary computation, dynamical systems, particular types of architecture or use of a powerful uniform learning mechanism, e.g. reinforcement learning. All of those support types of self-organising, self-modifying behaviours. But we are nowhere near understanding the full variety of powerful information-processing principles ‘discovered’ by evolution. By attending (...) closely to the diver- sity of biological phenomena we may gain key insights into (a) how evolution happens, (b) what sorts of mechanisms, forms of representation, types of learning and development and types of architectures have evolved, (c) how to explain ill-understood aspects of human and animal intelligence, and (d) new useful mechanisms for artificial systems. (shrink)

Recent years have yielded many discussions on how to endow autonomous agents with the ability to make ethical decisions, and the need for explicit ethical reasoning and transparency is a persistent theme in this literature. We present a modular and transparent approach to equip autonomous agents with the ability to comply with ethical prescriptions, while still enacting pre-learned optimal behaviour. Our approach relies on a normative supervisor module, that integrates a theorem prover for defeasible deontic logic within the control loop (...) of a reinforcement learning agent. The supervisor operates as both an event recorder and an on-the-fly compliance checker w.r.t. an external norm base. We successfully evaluated our approach with several tests using variations of the game Pac-Man, subject to a variety of “ethical” constraints. (shrink)