As part of one of my master projects a software environment was developed to examine reinforcement learning algorithms on existing dynamic models from Modelica in order to solve control tasks. Modelica is a non-proprietary, object-oriented, equation based language to conveniently model complex physical systems [1]<\/a>.<\/p>\n The result is the Python library Dymola Reinforcement Learning (dymrl)<\/strong> which allows you to explore reinforcement learning algorithms for dynamical systems.<\/p>\n The code of this project can be found at github<\/a>.<\/p>\n <\/p>\n A good and compact definition of reinforcement learning is given by Csaba Szepesv\u00e1ri:<\/p>\n Reinforcement learning is a learning paradigm concerned with learning to control a system so as to maximize a numerical performance measure that expresses a long-term objective. What distinguishes reinforcement learning from supervised learning is that only partial feedback is given to the learner about the learner\u2019s predictions. Further, the predictions may have long term effects through influencing the future state of the controlled system [2]<\/a>.<\/p><\/blockquote>\n Dymola Reinforcement Learning<\/strong> is a library to examine reinforcement learning algorithms on dynamic models. It consists of a new OpenAI Gym<\/strong> environment with a Python interface to actuate simulations in Dymola<\/a>. Dymola is a simulation tool based on the Modelica open standard.<\/p>\n OpenAI Gym is a toolkit for developing and comparing reinforcement learning algorithms [3]<\/a>. The toolkit has implemented the classic “agent-environment loop”. For each time step, the agent chooses an action, and the environment returns an observation and a reward.<\/p>\n The primary challenge has been to find a fast and stable way to communicate with the simulation tool. This communication has been realized with Functional Mockup Interface for co-simulation (FMI). The FMI co-simulation returns for a given internal state, an input and a step size of a model the output at a time. For the OpenAI Gym environment the advancement of the states and time is completely hidden [4]<\/a>. A component which implements the interface is called Functional Mockup Unit (FMU). A list of FMI supported simulation tools can be found on FMI Support in Tools<\/a>. For loading and interacting with Functional Mock-Up Untis (FMUs) in Python we used PyFMI<\/a>.<\/p>\n The library dymrl<\/strong> has been tested on the classical control task problem Cart Pole<\/strong>. The configuration (action, observation and rewards) was taken from the example of OpenAI Gym (Example<\/a>).<\/p>\n The objective of this tasks is to apply forces to a cart moving along a frictionless track so as to keep a pole hinged to the cart from falling over [5]<\/a>. The system is controlled by applying a force of +1 or -1 to the cart. For every time step that the pole is not more than 12 degree from the vertical or the cart moves not more than 2.4 units from the center, a reward of +1 is provided.<\/p><\/blockquote>\n Basis for this example is an existing dynamical model in Modelica and a FMU co-simulation. Several simulation tools offer an export function for FMI co-simulation.<\/p>\n In our example we used the build-in function The library dymrl provides a basic implementation of an environment in OpenAI. This means that this environment manages the communication with the simulation tool and returns the observation for a given action.<\/p>\n For a specific problem you have to derive the So our next step is to derive a new class, called By now we just created the environment and define some action and observation space. Solving our control task problem requires the implementation of an agent.<\/p>\n For our task, we used the simple table-based Q-Learning algorithm. This algorithms is suitable for such a small action and observation space. A good explanation of the Q-Learning algorithm can be found on Demystifying Deep Reinforcement Learning<\/a>.<\/p>\n To create a new agent, you have to load the new environment and implement an algorithm. In the Folder A understandable implemenation of Q-Learning is given by Carlos Aguayo<\/a>.<\/p>\n In this project we developed an environment to explore reinforcement learning for complex control tasks, in which a model is given. The verification of our implementation was done by solving a classical control task.<\/p>\n We hope to encourage people to explore reinforcement learning in the topic of optimal control tasks with dynamic models.<\/p>\n [1] Modelica, Modelica And The Modelica Association, accessed 5 Sept. 2016, https:\/\/www.modelica.org\/<\/a>.<\/p>\n [2] Szepesv\u00e1ri, C. (2010). Algorithms for reinforcement learning. Synthesis lectures on artificial intelligence and machine learning, 4(1), 1-103.<\/p>\n [3] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016). OpenAI Gym. arXiv preprint arXiv:1606.01540.<\/p>\n [4] Andersson, C. (2016). Methods and Tools for Co-Simulation of Dynamic Systems with the Functional Mock-up Interface (Doctoral dissertation, Lund University).<\/p>\n [5] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. Vol. 1. No. 1. Cambridge: MIT press, 1998.<\/p>\n By Gabriel Eyyi (ZHAW) In this blog post I will show how to combine dynamic models from Modelica with reinforcement learning. As part of one of my master projects a software environment was developed to examine reinforcement learning algorithms on existing dynamic models from Modelica in order to solve control tasks. Modelica is a non-proprietary, […]<\/p>\n","protected":false},"author":265,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[1,7,9],"tags":[41,40,38,39],"features":[],"class_list":["post-842","post","type-post","status-publish","format-standard","hentry","category-allgemein","category-blog","category-research","tag-modelica","tag-openai-gym","tag-python","tag-reinforcement-learning"],"yoast_head":"\nWhat is reinforcement learning?<\/h2>\n
<\/h1>\n
<\/a>Dymola Reinforcement Learning (dymrl)<\/h2>\n
\n
![\"agent-environment-loop\"](\"https:\/\/github.com\/eyyi\/dymrl\/raw\/master\/docs\/img\/rl_loop_dymrl.png\")
<\/a><\/p>\n<\/a>Tutorial: Solving Cart Pole problem<\/h2>\n
<\/a>Cart pole problem<\/h5>\n
![\"cart](\"https:\/\/github.com\/eyyi\/dymrl\/raw\/master\/docs\/img\/cart_pole.png\")
<\/a><\/p>\n<\/a>Prerequisite<\/h3>\n
translateModelFMU()<\/code> in Dymola to export the FMU and we moved the generated files to a folder in .\/dymrl\/envs\/assets\/<\/code>.<\/p>\n\u251c\u2500\u2500 docs\r\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 img\r\n\u251c\u2500\u2500 dymrl\r\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 envs\r\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 assets\r\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u00a0\u00a0 \u2514\u2500\u2500 inverted_pendulum\r\n\u2514\u2500\u2500 examples\r\n \u00a0\u00a0 \u251c\u2500\u2500 agents\r\n \u00a0\u00a0 \u2514\u2500\u2500 scripts\r\n<\/code><\/pre>\n<\/a>Create a specific OpenAI environment<\/h2>\n
DymolaEnv<\/code> class and define your observation and action space. Fortunately, OpenAI gym offers two convenient space objects. First a Discrete<\/code> space, which allows to represent a fixed range of numbers. A Box<\/code>space represents a n-dimensional box.<\/p>\nDymolaInvertedPendulumEnv<\/code>, from the DymolaEnv<\/code> class, and define an action and observation space. Consider that we only want two actions (-1, +1), so we choose a Diskrete<\/code> action space with two possible values. Furthermore, we only want to observe positions between [-2.4, 2.4] and angles between [-12\u00b0, 12\u00b0], so we choose a Box<\/code> space for our observation space.<\/p>\n\n
class<\/span> DymolaInvertedPendulumEnv<\/span>(dymola_env<\/span>.DymolaEnv<\/span>):\r\n NINETY_DEGREE_IN_RAD<\/span> =<\/span> (90<\/span>\/<\/span>180<\/span>)*<\/span>math.pi\r\n TWELVE_DEGREE_IN_RAD<\/span> =<\/span> (12<\/span>\/<\/span>180.0<\/span>)*<\/span>math.pi\r\n\r\n def<\/span> __init__<\/span>(self<\/span>):\r\n self<\/span>.theta_threshold_radians =<\/span> self<\/span>.TWELVE_DEGREE_IN_RAD<\/span>\r\n self<\/span>.x_threshold =<\/span> 2.4<\/span>\r\n dymola_env.DymolaEnv.__init__<\/span>(self<\/span>, '<\/span>inverted_pendulum\/Pendel_Komponenten_Pendulum.fmu'<\/span><\/span>)\r\n\r\n self<\/span>.force_magnitude =<\/span> 10.0<\/span>\r\n\r\n self<\/span>.config =<\/span> {\r\n '<\/span>action'<\/span><\/span>: {'<\/span>u'<\/span><\/span>: 10<\/span>},\r\n '<\/span>state'<\/span><\/span>: ['<\/span>s'<\/span><\/span>, '<\/span>v'<\/span><\/span>, '<\/span>phi1'<\/span><\/span>, '<\/span>w'<\/span><\/span>],\r\n '<\/span>initial_parameters'<\/span><\/span>: {'<\/span>m_trolley'<\/span><\/span>: 1<\/span>, '<\/span>m_load'<\/span><\/span>: 0.1<\/span>, '<\/span>phi1'<\/span><\/span>: self<\/span>.NINETY_DEGREE_IN_RAD<\/span>}\r\n }\r\n\r\n def<\/span> _get_action_space<\/span>(self<\/span>):\r\n return<\/span> spaces.Discrete(2<\/span>)\r\n\r\n def<\/span> _get_observation_space<\/span>(self<\/span>):\r\n high =<\/span> np.array([self<\/span>.x_threshold, np.inf, self<\/span>.theta_threshold_radians, np.inf])\r\n return<\/span> spaces.Box(-<\/span>hi<\/pre>\n<\/div>\n<\/a>Agent<\/h2>\n
.\/examples\/agents\/<\/code> you can find an example agent.<\/p>\n<\/a>Conclusion<\/h2>\n
<\/a>References:<\/h2>\n
<\/div>","protected":false},"excerpt":{"rendered":"