Typical exploration strategies in Model-free Policy Search

Thanks J. Peters et al for their great work of A Survey for Policy Search in robotics.

The exploration strategy is used to generate new trajectory samples τ[i] \tau^{[i]}. All exploration strategies in Model-free policy search is local and use stochastic policies to Implemen T exploration. Typically, Gaussian polices is used to model such stochastic policies.

Many model-free policy search approaches update the exploration distribution and, hence, the covariance of the G Aussian policy. Typically, a large exploration rate are used in the beginning of learning which are then gradually decreased to fine tune th E policy parameters. Action space vs Parameter Space

In action space. We can simply add an exploration noise ϵu \epsilon_{u} to the executed actions, i.e. ut=μ (x,t) +ϵu U_{t}=\mu (x,t) + \epsilon_{u} The Exploratino nose is all sampled independently for each time step from a zero-mean Gaussia N Distribution with Covarianceσu \sigma_{u}. The POLICYΠ\PI is given as: πθ (u|x) =n (U|μu (x,t), Σu) \pi_{\theta} (U|x) =\mathcal{n} (U|\mu_{u} (x,t), \sigma_{u}) Applications of exploration in action space can is found in reinforce algorithm or ENAC algorithm.

Exploration in parameter space perturb the Paramter Vectorθ\theta. In contrast to exploration in action space, which in paramter space can use more structured nose and adapt the variance of the exploration noise in dependence of the state featuresϕt (x) \phi_{t} (x).

Many approaches can be formulized and the concept of an upper-level policyπw (θ) \pi_{w} (\theta) which selects T He parameters of the actual Control policyπθ (u|x) \pi_{\theta} (u|x), i.e. the lower-level policy. The Upper-level policy is typically modeled as a Gaussian distribution πw (θ) =n (θ|μθ,σθ) \pi_{w} (\theta) =\mathcal N (\theta|\mu_{\theta},\sigma_{\theta}). The Lower-level Control Policy u=πθ (x,t) U=\pi_{\theta} (X,t) is typically modeled as deterministic policy since Exploratio N only takes place in the parameter space.

Now we use the paramter vector w W defining a distribution Overθ\theta. Then we can use the this distribution to directly explore in parameter space. The optimization problem for learning upper-level polices goes as maximizing: