(Teaching kites to fly using) Reinforcement Learning

Engineering Flight Control
1

An earlier topic: Teaching a glider to soar using reinforcement learning in the field

Links to more info and related topics here.

1 Like
2

…

3

Original topic: Swiss researchers invent drone-flying AI that tops champions

Champion-level Drone Racing using Deep Reinforcement Learning (Nature, 2023) - YouTube

Autonomous Drone Racing with Deep Reinforcement Learning - YouTube

4
5

Off-topic here, but still on kite control:

7

NVIDIA’s New AI Just Made Real Physics Look Slow – Two Minute Papers

8

A few thoughts on this use for Reinforcement Learning(RL):

  • RL is normally done in simulation (as opposed to an actual .. kite in this case), because
    1. it needs a lot of trials to learn an useful behavior. If a simulation can be run sufficiently fast the learning system has “more time” to learn.
    2. failing a physical machine can be expensive.
  • While simulations can be almost arbitrarily accurate to the physical real “thing”, there are some important caveats:
    1. they can predict only what we foresee. e.g. if we do not (or don’t know how to) model kite line failure conditions and mode, the RL model will never learn how to handle a line break.
    2. we cannot tell in advance which properties of the kite are most relevant to its dynamic behavior (e.g. humidity influence on fabric weight and elasticity?) therefore one feels compelled to fill in all property details without knowing whether they-re needed or not. Which takes significant work.
    3. We also need to account for inaccuracies/differences between the physical sensors vs. simulated ones.
    4. In case of wind-dependent machines it is hard to both simulate wind changes (turbulence, gusts) in presence of ground features (trees, hills, buildings) and to predict all possible ground features of a future location of the wind power station.

An alternative to reinforcement learning is transfer learning, or supervised learning, by which both a human’s “correct” control actions and the physical state of the controlled system (= kite’s speed, acceleration, position, orientation, etc..) are measured for an amount of time to collect sufficient data to feed a learning algorithm that directly learns from real-world data.

The advantages here are

  • the learning algorithm is fed accurate actual sensor data, we don’t need to worry about the expertise and details on how to create an accurate simulation.
  • we can skip the lengthy initial trial-and-error stages of a totally “dumb” RL agent before it begins to grasp the right actions, which translates in less training experience required.

Some disadvantages still persist, e.g. not having recorded data on unexpected circumstances and possibly a long amount of human work spent on handling a real kite. But IMO it might be easier to handle a kite for days or weeks to collect sufficient data instead of attempting to develop a simulation of same kite.
If only because it is more fun and a more available domain of expertise.

1 Like
9
Slow Chat IV
10

What do you want to show about reservoir computing?

11

I hadn’t heard of it before. It seems like another way to do flight control, but more efficient; thousands to millions of times less data and computer time needed for training compared to deep learning, Daniel Gauthier: https://youtu.be/wbH4En-k5Gs?t=848

The time-stamped link in my previous comment showed a practical application.

2 Likes
12

Ah, ok here it is in a different topic. I’m quite fond of reservoirs, they make sense for dynamic awareness and control in robots and drones. And likely kites too.

1 Like
13

Just a couple notes

  • Gauthier’s “next generation” reservoirs are just one unorthodox approach to this kind of computation. They may be better on certain problems. What I got from other papers is it tends to be more expensive computationally, but I don’t know what exactly that means.

  • I mentioned above collecting human-controlled flight data for transfer learning - Compared to classical deep-learning, reservoirs tend to be very sample efficient that means comparatively they need very little data to learn a dynamic pattern.

  • The same real-kite collected data might also be useful to imitate the dynamic response of the kite, that means a trained reservoir might replace a simulation using a physics engine. A most challenging part in robotic development via reinforcement learning is transferring the trained behavior from simulation to the real system.

1 Like