Last year, OpenAI and DeepMind teamed up to do the coolest experiments of the time, without the classical reward signals to train the agents, but rather a new approach to reinforcement learning based on human feedback. There is a blog dedicated to this experiment learning from Human Preferences, the original paper is "deep reinforcement learning from Human Preferences" (based on the human preferences of depth enhancement learning).
Link:https://arxiv.org/pdf/1706.03741.pdf
With some deep intensive learning, you can also train sticks for a somersault.
I've seen some suggestions: reproducing papers is a great way to improve machine learning, which is an interesting attempt on my own. Learning from Human Preferences really is a very interesting project, I am very happy to reproduce it, but in retrospect this experience, but with the expected discrepancy.
If you also want to reproduce your paper, here are some tips for deep reinforcement learning:
· · ·
First of all, reinforcement learning is usually more complicated than you expect.
A large part of the reason is that intensive learning is very sensitive. There are a lot of details that need to be dealt with correctly, and if not, it's hard to tell what's wrong.
Scenario 1: When the basic implementation is complete, the training is not successful. I have a variety of ideas about this, but it turns out to be because of the regularization of incentives and the 1 of pixel data at the critical stage. Even though I knew what was wrong, I couldn't find the path to follow: The network accuracy of the excitation predictor based on the pixel data is really good, and it took me a long time to examine the excitation predictor carefully, only to notice that the excitation regularization error was noticed. Find out the cause of the problem is accidental, because notice a small mistake, only to find the right path.
Scenario 2: When I did the final code cleanup, I realized I had dropout wrong. The excitation Predictor network takes a pair of video clips as inputs, and each video fragment is treated identically by two networks with shared weights. If you add dropout to each network and accidentally forget to provide the same random seed for each network, the dropout network is different for each network, so the video clips won't be processed the same way. Although predicting the accuracy of the network looks exactly the same, it actually completely destroys the original network training.
Which one is bad? Well, I don't see it either.
I think this is something that often happens (for example: deep reinforcement learning doesn's t work yet). My harvest is that when you start an intensive learning project, you will theoretically encounter a dilemma like being trapped in a math problem. It's not like my usual programming experience: in a place where you're trapped, there's usually a clear clue to follow, and you can get out of trouble in a matter of days. It's more like when you're trying to solve a problem where there's no obvious progression, the only way is to try it out until you find the key evidence or find the important inspiration to find the answer.
So the conclusion is that you should be aware of the details whenever you are confused.
There are a lot of points in this project, the only clue is the attention to the trivial small things. For example, it is sometimes useful to have differences between frames as features. It was tempting to take advantage of the new features, but I realized that I didn't know why it had such a big impact on the simple environment I was using. Only under such confusion can we find that the difference between the frame and the background zero will make the regularization problem appear.
I'm not sure how to make people aware of this, but my best guess at the moment is:
Also, it's best to be prepared to get into trouble every few weeks. If you keep going, pay attention to the small details and be confident, you can reach the other side.
· · ·
Speaking of the different experiences of programming in the past, the second major learning experience is the difference in the way of thinking needed to work in the long iteration time.
Debugging seems to involve four basic steps:
In most of the programming I've done before, I've gotten used to quick feedback. If something doesn't work, you can make a change and see what difference it will produce in a few seconds or minutes. It's easy to collect the evidence.
In fact, in the case of quick feedback, it is much easier to collect evidence than to make assumptions. When you can validate the first idea in a short time, why take 15 minutes to think about everything, even if they may be the cause of the phenomenon? In other words: If you can get quick feedback, you don't have to think about it, just keep trying.
If you're using an attempt strategy, it's a waste of time to spend 10 hours each time you try. What if the last time was unsuccessful? Well, I admit it's sometimes the case. Then we'll check it again. Coming back the next morning: still useless? Well, maybe another one, so let's run it again. After one weeks, you haven't solved the problem yet.
Running multiple times at the same time, trying different things in some way will help. But a) unless you can use clustering, you may end up paying a lot of cost to the cloud (see below); b) Because of the difficulties of reinforcement learning mentioned above, if you try to iterate too quickly, you may never realize what kind of evidence you need.
From a large number of experiments and a small amount of thinking, to a small number of attempts and a lot of thinking, is a key shift in productivity. When debugging over long iterations, you really need to devote a lot of time to building assumptions--forming steps--thinking about what all the possibilities are, how likely they seem to be, and how much they seem to be possible, depending on what you've seen so far. Spend as much time as you can on it, even if it takes 30 minutes or one hours. Once you've enriched the hypothetical space as much as possible, you know what evidence allows you to best differentiate between different possibilities before you start experimenting.
(If you take the project as an amateur project, it is particularly important to consider the issue carefully.) If you work only one hours per day on this project and each iteration takes a day, the number of runs per week is more like a rare commodity that you have to make full use of. Squeezing out working time every day to think about how to improve the results of the operation can make people feel very stressed. So change your mind and take a few days to think, rather than start anything, until I'm confident that the "What's the problem" hypothesis. ）
To think more, sticking to a more detailed work log is a very important link. When it takes less than a few hours to progress, it doesn't matter if you don't have a weekday log, but if you grow a bit longer, you'll easily forget what you've already tried, and you'll just have to go in circles. The log format I summarized is:
Log 1: What is the specific output I'm doing now?
Log 2: Think boldly, such as the assumptions about the current problem, what to do next?
Log 3: Make a record of the current run and briefly remind you which questions to answer each time you run.
Log 4: Results of the operation (Tensorboard, any other important observations), categorized by run type (e.g., in the context of the agent's training)
At first, my log was relatively small, but at the end of the project, my attitude was more inclined to "record everything I think". While the cost of time is high, I think it is worthwhile, in part because some debugs require cross-referencing results and ideas, often in intervals of days or weeks, partly because (at least this is my impression) from a large-scale upgrade to an overall improvement in effective thinking.
A typical log
· · ·
In order to maximize the effectiveness of the experimental results, I did two things in the course of the experiment, which may play a role in the future.
First, record all metrics so you can maximize the amount of evidence collected at each run. There are some obvious indicators, such as training/validation accuracy. Of course, with a lot of time to brainstorm, it's also important to study other indicators to diagnose potential problems.
The reason I'm suggesting this is partly because of the back-view bias, because I know which indicators should start recording earlier. It is difficult to predict which indicators will be useful at the advanced stage. However, the strategies that might be useful are:
For every important component in the system, consider what you can measure. If you have a database, measure how fast it grows in size. If there is a queue, measure the speed of the processing item.
For each complex process, it took a long time to measure the different parts of it. If you have a training cycle, measure the time required for each batch to run. If you have a complex reasoning process, measure the time spent on each sub-inference. These times can be very useful for future performance tuning, and sometimes some other bugs that are hard to spot. (For example, if you see some results that are getting longer, it could be because of a memory leak.) )
Similarly, consider analyzing memory usage for different components. Small memory leaks can point to a variety of things.
Another strategy is to observe what other people are measuring. In the context of intensive learning, John Schulman in his research nuts and bolts of deep RL talk (https://www. Youtube.com/watch? V=8ecdack9kaq ) has some good suggestions. For the strategy gradient approach, I found that the policy entropy is a good indicator, which can well reflect whether the training is done again, more sensitive than the reward of each training.
Unhealthy and healthy strategies for entropy maps. Failure Mode 1 (left): converges to Chang (randomly selects a subset of behaviors); failure Mode 2 (middle): converges to zero entropy (the same action is selected each time). Right: The strategy entropy of successful table tennis training running
When you see something suspicious in the measurement of the record, remember to be aware of the confusion, rather than assume that it is an important thing, rather than just a inefficient implementation of some data structures. (I ignored a tiny but inexplicable decay in the frames per second, resulting in several months of multithreaded errors.) )
Debugging is much easier if you can see all the metrics in one place. I like to use Tensorboard as much as possible. Using TensorFlow to record metrics is difficult, so consider using view Easy-tf-log (https://github.com/mrahtz/easy-tf-log), It provides a simple interface without any additional setup interface Tflog (key, value).
The second thing that seems to make sense is to spend time trying and predicting failure in advance.
The reason for the failure is often obvious, thanks to the back-view bias. But what's really frustrating is that the failure pattern is obvious until you see what it is. When you start to train a model, and then you come back the next day to see it fail, even before you study the cause of failure, you realize "Oh, that must be because I forgot to set Frobulator"?
The simple fact is that sometimes you can trigger this semi-hindsight (half-hindsight-realisation) in advance. It requires a conscious effort to stop and think for five minutes before starting to run. I think the most useful script is: 2
1. Ask yourself, "How surprised if you fail to run?" ”
2. If the answer is "not very surprised", then imagine yourself in a future scenario-this run has failed, and then ask yourself, "if it fails, what is wrong?" ”
3, correction of the thought of any place
4, repeat the above steps to know the answer to question 1 is "very surprised" (or at least "how surprised to be surprised")
There will always be some errors you can't predict, and sometimes you'll still ignore some obviously avoidable errors, but this approach at least seems to reduce some of the very foolish mistakes you might have made without thinking ahead.
· · ·
Finally, the most surprising thing about the project is the time spent, as well as the computational resources required.
I initially estimated that as an amateur project, it would take 3 months. But it actually took about 8 months. (The initial estimate is actually very pessimistic!) This is partly due to an underestimation of the amount of time that may be spent in each phase, but the biggest underestimation is that there are no projections of other things outside the project. It's hard to say how rigorous this inference is, but for an amateur project, doubling your initial estimated time (already pessimistic) may be a good experience.
Even more surprising is the amount of time that is actually spent in each phase. My initial project plan in the main phase of the timetable is basically as follows:
This is the actual time spent at each stage
Instead of writing code, it took a long time to debug the code. In fact, running up to a so-called simple environment took 4 times times the initial expected implementation time. (This is the first amateur project I've spent hours in a row, but the experience has been similar to a past machine learning project.) ）
(note: Carefully designed from the outset, you can imagine the "simple" environment of reinforcement learning.) In particular, consider carefully: a) Whether your reward can really convey the correct information about the task, B) whether the reward depends only on the previous observations or on the current action. In fact, if you are making arbitrary reward predictions, the latter may also be relevant, for example, using a critic)
The other is the total amount of compute resources required. I am fortunate to be able to use the school cluster, although the machine is only CPU, but for some work is already very good. I experimented with two cloud services for work that required GPUs (such as fast iterations on some small parts) or when the cluster was too busy: the Google Cloud Engine virtual machine (https://console.cloud.google.com/ Projectselector/compute/instances?supportedpurview=project&pli=1), Floydhub (https://www. floydhub.com/).
If you only want to access the GPU machine through the shell, Google Cloud computing engine is good, but I am more on the floydhub to try. Floydhub is basically a cloud computing service specifically for machine learning. Run the Floyd run Python awesomecode.py command, Floydhub will initialize a container, upload your code, and run your code. Floydhub is so powerful there are two key factors:
The Web interface for Floydhub. Above: An overview of the history run index, and a single run. Below: The code used for each run and any data that runs the output are automatically archived.
The 2nd point of importance is beyond my word. For any project, this long-term, detailed recording of operations and the ability to reproduce prior experiments is absolutely necessary. Although version control software can help, a) managing large volumes of output is difficult; b) needs to be very diligent. (For example, if you start to run some, then make a little change and then run another, can you clearly use which code when you submit the results for the first run?) You can take notes carefully or check your own system, but on Floydhub it's done automatically without the effort you spend.
I like other aspects of Floydhub:
One of the troubles I encountered with Floydhub was that it could not customize the container. If your code has a lot of dependent packages, you need to install them before each run. This limits the iteration rate on the short run. However, you can solve this problem (for example, create_floyd_base.sh) by creating a "DataSet" that contains file system changes after installing these dependent packages, and then copying them from the dataset every time you start running. It's awkward, but it's still better than solving a GPU-driven problem.
Floydhub is a little more expensive than Google Cloud: Floydhub a K80 GPU machine for $1.2 per hour, and a similar configuration on Google Cloud would only cost $0.85 per hour (lower if you don't need a machine up to 61G). Unless your budget is really limited, I think the extra convenience Floydhub brings is worth the price. Google Cloud is even more cost-effective when running large numbers of computations in parallel, because you can run multiple on a single large virtual machine.
(the third option is Google's new colaboratory service, which is equivalent to providing you with a jupyter notebook that can be accessed for free on the K80 GPU machine.) But it's not delayed because of Jupyter: You can execute arbitrary commands, and you can set up shell access if you really want to. The biggest drawback is that if you close the browser window, your code won't stay running, and there's a limit to how long it will run before the container that hosts the notebook is reset. So this is not suitable for long-term operation, but it is helpful to run fast prototypes on the GPU. ）
The project cost a total of:
I was surprised to find that it cost about $850 ($ 200 on Floydhub and $650 on Google's cloud computing engine) in the 8 months that it took to achieve the project.
Some of the reasons are I'm clumsy (see the Slow Iterations section above). Some of the reasons are that intensive learning is still so inefficient that it takes a long time to run (it takes up to 10 hours each time to train a pong agent).
But a big part of this is that I had an accident at the end of the project: Intensive learning may be less stable and we need to run multiple times to determine performance using different random seeds.
For example, once I think that basically everything is done, I do end-to-end testing on this environment. But even though I have been using the simplest environment, when training a point to move to the center of the square, I still have a very big problem. So I went back to Floydhub to make adjustments and run three copies, and it turns out I thought the excellent super-parameters were only successful once in three Tests.
2/3 of random seeds fail (red/blue) not uncommon
For the amount of computation needed, I give you an intuitive number:
Cost Aspects:
It's equivalent to spending 3 sandwiches each time you try to verify your idea.
Again, from the deep reinforcement learning doesn's t work yet this article can be seen, this instability seems normal also acceptable. In fact, even if "five random seeds (a common standard) may not suffice to show that the results are meaningful, by carefully selecting you will get a non-overlapping confidence interval." ”
(All of a sudden, the $25,000 AWS credits that OpenAI scholars plan to offer don't look so crazy, which may be related to the number you give to someone, so don't worry about computing at all.) ）
I mean, if you want to solve an in-depth reinforcement learning project, make sure you know what you're doing. Make sure you're ready to spend as much time and money as you need.
· · ·
In general, it is interesting to reproduce an intensive study paper as an amateur project. In turn, you can think about what skills have been learned, and I'd like to know if it's worthwhile to spend a few months reproducing a paper.
On the other hand, I feel that my research skills in machine learning have not improved significantly (in retrospect, this is actually my goal), but the ability to apply has been improved, and the considerable difficulties in the study seem to produce many interesting and concrete ideas that will make you feel that the time spent is worthwhile. Producing an interesting idea seems to be a problem a) requires a lot of available concepts, B) have a keen sense of smell on good ideas or ideas (for example, what kind of work is useful to the community). In order to achieve these goals, I think it is a good practice to read influential papers, summarize and critically analyze these papers.
So I think the main conclusion I get from this project is that whether you want to improve your engineering skills or your research skills, it's worth thinking carefully. It is not that there is no such thing as both, but if something is your weakness, you can look for a project that is specific to this item to improve your level.
If you want to improve both of these skills, it may be better to read a large number of papers, find a paper that interests you and have a clear code, and try to implement or extend it.
· · ·
If you want to do a deep intensive learning project, here are some details to note.
Find some research papers to reproduce.
Find some relatively single paper, avoid the need for multiple knowledge points to work together in the paper;
Intensive Learning
General Machine Learning
TensorFlow
When you cannot use the GPU, be careful to switch to the CPU using Allow_soft_placement. If you occasionally write code that does not run on the GPU, it can smoothly switch to the CPU. For example:
I don't know how many things like this can't run on the GPU, but for security reasons, manually switch to the CPU, for example:
A Healthy Mind
· · ·
If you don't hesitate to read this article, it would be great!
If you also want to enter the field of deep reinforcement learning, here are some resources for you to refer to when you get started:
To find out what's going on in the field of deep reinforcement learning, take a look at these things:
Read this article before you reproduce a deep-strengthening study paper!
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
1 on 1 presale consultation
24/7 Technical Support 6 Free Tickets per Quarter Faster Response
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.
A comprehensive suite of global cloud computing services to power your business
Payment Methods We Support
