Gato, the ＂generalist＂ AI agent of DeepMind, is coming, with multi-modal and multi-task, inspired by the big language model.

Machine heart report

Machine zhixin editorial department

After writing articles and drawing pictures, the AI big model now has the ability to play games at the same time. I can’t help thinking, what kind of flowers can DeepMind’s agent Gato play in the future?

If a single sequence model can solve all tasks, it is a great thing, because this model reduces unnecessary troubles. However, it needs to increase the number and diversity of training data. In addition, the performance of this general model will be improved with the expansion of data and model. Historically, general models that are better at using computing will eventually surpass models that are specific to specific fields.

Today, inspired by large-scale language modeling, Deepmind applied similar methods to construct a single "generalist" agent Gato, which has the characteristics of multi-mode, multi-task and multi-embodiment.

Paper address: https://storage.googleAPIs.com/deepmind-media/a% 20generalist% 20agent/genericist% 20agent.pdf.

Gato can play Atari games, output subtitles to pictures, chat with others, stack building blocks with robotic arms and so on. In addition, Gato can also decide whether to output text, joint torque, button press or other token according to the context.

Unlike most agents playing games, Gato can play many games using the same training model without training for each game separately.

The training data set of Gato should be as extensive as possible, including different modes, such as images, texts, proprioception, joint torque, button pressing and other discrete and continuous observations and actions.

In order to be able to handle this multimodal data, Deepmind serializes all the data into a flat token sequence. In this representation, Gato can be trained and sampled from a large-scale language model similar to the standard. During deployment, the sampled token are combined into dialog responses, subtitles, button presses or other actions according to the context.

Wang Jun, a professor of computer science at UCL, told Machine Heart that DeepMind’s latest work combines reinforcement learning, computer vision and natural language processing. Although it follows the previous methods in technical thinking, it is not easy to map CV, NLP and RL into the same space and express them with a set of parameters.

Its positive significance lies in that it proves that the combination of CV, NLP and RL is feasible, and some problems of decision intelligence can be solved through sequence prediction. Considering that the current parameters of Gato model are only moderate, it will be of great significance to continue to explore in this direction and build a larger model.

However, the RL part of the Gato model only adopts the supervised learning method, which does not touch the real core of reinforcement learning-Reward design mechanism, and there is no multi-agent decision-making problem in the current task. Professor Wang Jun said that his team has done a lot of exploration on the decision-making model recently, including proving that multi-agent decision-making can also be a sequential model, and relevant results will be announced in the near future, so you are welcome to pay attention.

Gato agent details

In the training stage of Gato, data from different tasks and modes are serialized into flat token sequences, which are processed by a transformer neural network similar to a large language model for batch and other processing. Because the loss is masked, Gato only predicts action and text targets.

The following picture shows the training process of Gato.

When deploying Gato, prompts (such as demonstrations) are tokenised, forming an initial sequence. Then, the environment produced the first observation, which was tokenised and added to the sequence. Gato samples the motion vectors by autoregressive method, and only samples one token at a time.

Once all the token containing the action vectors are sampled (determined by the action specification of the environment), the actions are decoded and sent to the environment, and then new observations are gradually generated. Repeat this process. Gato model always looks at all previous observations and actions in the context window containing 1024 token.

The following figure shows the process of deploying Gato as a control policy.

In addition to various natural language and image data sets, Gato has also been trained on a large number of data sets containing the experience of agents in simulated and real environments.

Table 1 below shows the control data set for training Gato on the left and the visual and linguistic data set on the right. Sample weight indicates the average proportion of each data set in the training sequence batch.

Gato agent capability

The researchers summarized the performance of Gato when training on the above data. In other words, all the results of all tasks come from a single pre-training model with a set of weights. The fine-tuning results will be shown in the chapter "Experimental Analysis".

Analog control task

Figure 5 below shows the number of different control tasks performed by Gato above a given score threshold relative to the expert performance in Gato training data. Among them, the value on the x axis represents a specific percentage of the expert score, and 0 corresponds to the random agent performance. The y-axis represents the number of tasks when the average performance of the pre-training model is equal to or higher than a certain percentage.

The researchers report the performance as a percentage, where 100% corresponds to the experts of each task and 0% corresponds to the random strategy. For each simulation control task of the training model, they roll out Gato strategy 50 times in the corresponding environment and average the defined scores. As shown in the figure below, Gato performed more than 450 of the 604 tasks with an expert score threshold of over 50%.

In ALE Atari, Gato scored an average (or higher) score for human beings in 23 Atari games and twice the score for human beings in 11 games. Although the single-task online RL agent that generates data is still superior to Gato, it can be overcome by increasing capacity or using offline RL training instead of pure supervision. The researcher also introduced a professional single-domain ALE Atari agent, which scored better than humans in 44 competitions.

In BabyAI, Gato got more than 80% expert scores at almost all levels. For the most difficult task, BossLevel, Gato scored 75%. In contrast, the other two published benchmarks, BabyAI 1.0 and BabyAI 1.1, used 1 million demonstrations to train this single task, but their scores were only 77% and 90%.

In Meta-World, Gato scored more than 50% in 44 of the 45 tasks trained, more than 80% in 35 tasks and more than 90% in 3 tasks. On the standardized DM Control Suite, Gato scored more than 50% in 21 of 30 tasks and more than 80% in 18 tasks.

Robot benchmark evaluation

Remote operation from the first perspective can collect expert demonstrations. However, such demonstrations are slow and costly to collect. Therefore, the data-efficient behavior cloning method is desirable for training universal robot manipulators, and off-line pre-training has become a very dynamic research field. The researchers also evaluated Gato on the established RGB Stacking robot benchmark.

The skill generalization challenge of RGB Stacking robot benchmark tests the ability of agents to stack objects with shapes that have never been seen before. Agents are trained on a dataset containing episodes of robot stacked objects of various shapes. However, the triplets of five object shapes are not included in the training data, but are used as test triplets. The researchers evaluated the trained Gato for 200 rounds for each test triplet on the real robot.

The results in Table 2 below show that the success rate of Gato on each test triplet is equivalent to the benchmark of single task BC-IMP(filtered BC) proposed by Lee et al. (2021).

Text example

Gato agents can also generate basic dialogues and add subtitles (or descriptions) to images. Figure 6 below shows a representative example of Gato subtitling an image.

Figure 7 below shows some selected examples of plain text dialogue and communication.

experimental analysis

In Figure 8 below, DeepMind evaluated three different model sizes (measured by parameter count): 79M model, 364M model and 1.18B model (Gato). It can be concluded that under the same number of token, with the expansion of the model, the performance of the model will be improved.

Figure 10 below compares the success rate of Gato in different fine-tuning data mechanisms with that of sim-to-real experts and Critical-Regulated Regression (CRR) agents. The results are as follows: In reality and simulation (the red curves on the left and right respectively), Gato recovered the performance of experts with only 10 episodes, and reached the peak when fine-tuning data with 100 or 1000 episodes, surpassing the experts. After this point (at 5000), the performance will drop slightly, but it will not be far lower than that of experts.

Table 3 below shows the comparison results of Gato and BC-IMP.

Original link:

Please contact Ben WeChat official account for authorization.

Contribute or seek reports: content@jiqizhixin.com.

Original title: "DeepMind" generalist AI agent Gato is coming, multi-modal, multi-tasking, inspired by the big language model "

Read the original text