DeepSeek R-1 Model Overview and how it Ranks against OpenAI’s O1

DeepSeek is a Chinese AI business “dedicated to making AGI a truth” and open-sourcing all its models. They began in 2023, but have been making waves over the past month approximately, and especially this past week with the release of their two most current thinking models: DeepSeek-R1-Zero and the more advanced DeepSeek-R1, likewise known as DeepSeek Reasoner.

They have actually released not only the designs however likewise the code and evaluation prompts for public usage, along with a detailed paper detailing their technique.

Aside from producing 2 extremely performant models that are on par with OpenAI’s o1 model, the paper has a great deal of valuable info around support learning, chain of idea reasoning, timely engineering with thinking models, and more.

We’ll start by concentrating on the training process of DeepSeek-R1-Zero, which uniquely relied solely on reinforcement knowing, rather of standard supervised learning. We’ll then move on to DeepSeek-R1, how it’s reasoning works, and some prompt engineering finest practices for reasoning models.

Hey everyone, Dan here, co-founder of PromptHub. Today, we’re diving into DeepSeek’s most current design release and comparing it with OpenAI’s thinking designs, specifically the A1 and A1 Mini models. We’ll explore their training process, thinking abilities, and some essential insights into prompt engineering for thinking models.

DeepSeek is a Chinese-based AI business devoted to open-source development. Their current release, the R1 thinking design, is groundbreaking due to its open-source nature and ingenious training techniques. This consists of open access to the models, triggers, and research documents.

Released on January 20th, DeepSeek’s R1 attained outstanding performance on various criteria, equaling OpenAI’s A1 designs. Notably, they also released a precursor design, R10, which acts as the foundation for R1.

Training Process: R10 to R1

R10: This model was trained specifically utilizing reinforcement learning without supervised fine-tuning, making it the first open-source model to accomplish high efficiency through this technique. Training included:

– Rewarding proper responses in deterministic jobs (e.g., mathematics issues).
– Encouraging structured reasoning outputs utilizing templates with “” and “” tags

Through thousands of models, R10 developed longer thinking chains, self-verification, and even reflective habits. For example, throughout training, the model showed “aha” moments and self-correction habits, which are rare in standard LLMs.

R1: Building on R10, R1 included numerous enhancements:

– Curated datasets with long Chain of Thought examples.
– Incorporation of R10-generated reasoning chains.
– Human choice alignment for refined responses.
– Distillation into smaller sized designs (LLaMA 3.1 and 3.3 at numerous sizes).

Performance Benchmarks

DeepSeek’s R1 model performs on par with OpenAI’s A1 models across numerous thinking standards:

Reasoning and Math Tasks: R1 competitors or surpasses A1 designs in accuracy and depth of thinking.
Coding Tasks: A1 designs generally carry out much better in LiveCode Bench and CodeForces tasks.
Simple QA: R1 typically outmatches A1 in structured QA tasks (e.g., 47% accuracy vs. 30%).

One significant finding is that longer reasoning chains typically enhance efficiency. This lines up with insights from Microsoft’s Med-Prompt structure and OpenAI’s observations on test-time calculate and reasoning depth.

Challenges and Observations

Despite its strengths, R1 has some restrictions:

– Mixing English and Chinese responses due to an absence of supervised fine-tuning.
– Less polished responses compared to chat models like OpenAI’s GPT.

These problems were addressed during R1’s refinement process, including supervised fine-tuning and human feedback.

Prompt Engineering Insights

An interesting takeaway from DeepSeek’s research is how few-shot prompting abject R1’s efficiency compared to zero-shot or concise customized prompts. This lines up with findings from the Med-Prompt paper and OpenAI’s recommendations to restrict context in thinking models. Overcomplicating the input can overwhelm the model and minimize accuracy.

DeepSeek’s R1 is a considerable step forward for open-source reasoning models, showing capabilities that rival OpenAI’s A1. It’s an amazing time to try out these designs and their chat user interface, which is complimentary to use.

If you have questions or wish to discover more, check out the resources linked listed below. See you next time!

Training DeepSeek-R1-Zero: A support learning-only method

DeepSeek-R1-Zero sticks out from a lot of other cutting edge models due to the fact that it was trained using just support knowing (RL), no supervised fine-tuning (SFT). This challenges the current traditional method and opens up brand-new chances to train reasoning models with less human intervention and effort.

DeepSeek-R1-Zero is the first open-source model to verify that advanced thinking capabilities can be developed simply through RL.

Without pre-labeled datasets, the model learns through trial and mistake, improving its habits, parameters, and weights based entirely on feedback from the options it creates.

DeepSeek-R1-Zero is the base design for DeepSeek-R1.

The RL process for DeepSeek-R1-Zero

The training process for DeepSeek-R1-Zero included providing the design with different thinking jobs, varying from math problems to abstract logic challenges. The model created outputs and was examined based upon its performance.

DeepSeek-R1-Zero got feedback through a reward system that assisted assist its knowing process:

Accuracy benefits: Evaluates whether the output is proper. Used for when there are deterministic outcomes (mathematics problems).

Format benefits: Encouraged the design to structure its thinking within and tags.

Training prompt design template

To train DeepSeek-R1-Zero to generate structured chain of idea sequences, the researchers used the following timely training template, changing timely with the reasoning question. You can access it in PromptHub here.

This template prompted the design to clearly describe its thought procedure within tags before delivering the final response in tags.

The power of RL in reasoning

With this training process DeepSeek-R1-Zero started to produce advanced reasoning chains.

Through countless training actions, DeepSeek-R1-Zero evolved to solve significantly complex issues. It found out to:

– Generate long reasoning chains that enabled much deeper and more structured analytical

– Perform self-verification to cross-check its own responses (more on this later).

– Correct its own errors, showcasing emergent self-reflective behaviors.

DeepSeek R1-Zero performance

While DeepSeek-R1-Zero is mostly a precursor to DeepSeek-R1, it still achieved high efficiency on several standards. Let’s dive into some of the experiments ran.

Accuracy enhancements throughout training

– Pass@1 accuracy started at 15.6% and by the end of the training it enhanced to 71.0%, comparable to OpenAI’s o1-0912 design.

– The red solid line represents performance with majority voting (similar to ensembling and self-consistency strategies), which increased accuracy even more to 86.7%, exceeding o1-0912.

Next we’ll look at a table comparing DeepSeek-R1-Zero’s efficiency throughout multiple thinking datasets against OpenAI’s reasoning models.

AIME 2024: 71.0% Pass@1, slightly listed below o1-0912 however above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a rating of 73.3%.

– Performed much even worse on coding tasks (CodeForces and LiveCode Bench).

Next we’ll take a look at how the action length increased throughout the RL training procedure.

This chart reveals the length of responses from the model as the training process advances. Each “action” represents one cycle of the model’s knowing procedure, where feedback is provided based upon the output’s efficiency, evaluated utilizing the prompt template talked about earlier.

For each concern (corresponding to one step), 16 actions were sampled, and the typical accuracy was determined to make sure steady examination.

As training progresses, the design generates longer thinking chains, enabling it to solve increasingly complicated thinking tasks by leveraging more test-time compute.

While longer chains do not always ensure much better outcomes, they generally associate with enhanced performance-a trend also observed in the MEDPROMPT paper (find out more about it here) and in the original o1 paper from OpenAI.

Aha moment and self-verification

Among the coolest elements of DeepSeek-R1-Zero’s development (which likewise applies to the flagship R-1 model) is simply how good the design ended up being at reasoning. There were sophisticated reasoning behaviors that were not explicitly configured however developed through its reinforcement learning process.

Over countless training actions, the design started to self-correct, reassess flawed logic, and verify its own solutions-all within its chain of idea

An example of this kept in mind in the paper, referred to as a the “Aha minute” is below in red text.

In this circumstances, the model literally stated, “That’s an aha moment.” Through DeepSeek’s chat feature (their variation of ChatGPT) this type of reasoning generally emerges with phrases like “Wait a minute” or “Wait, however … ,”

Limitations and obstacles in DeepSeek-R1-Zero

While DeepSeek-R1-Zero had the ability to perform at a high level, there were some drawbacks with the model.

Language blending and coherence problems: The design sometimes produced responses that blended languages (Chinese and English).

Reinforcement learning trade-offs: The lack of monitored fine-tuning (SFT) suggested that the model lacked the improvement required for fully polished, human-aligned outputs.

DeepSeek-R1 was established to deal with these concerns!

What is DeepSeek R1

DeepSeek-R1 is an open-source reasoning design from the Chinese AI laboratory DeepSeek. It constructs on DeepSeek-R1-Zero, which was trained entirely with support knowing. Unlike its predecessor, DeepSeek-R1 includes monitored fine-tuning, making it more fine-tuned. Notably, it exceeds OpenAI’s o1 model on a number of benchmarks-more on that later.

What are the primary differences in between DeepSeek-R1 and DeepSeek-R1-Zero?

DeepSeek-R1 builds on the structure of DeepSeek-R1-Zero, which serves as the base design. The 2 differ in their training methods and total efficiency.

1. Training method

DeepSeek-R1-Zero: Trained entirely with support knowing (RL) and no supervised fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that consists of supervised fine-tuning (SFT) first, followed by the exact same support learning process that DeepSeek-R1-Zero wet through. SFT helps improve coherence and readability.

2. Readability & Coherence

DeepSeek-R1-Zero: Struggled with language mixing (English and Chinese) and readability problems. Its reasoning was strong, but its outputs were less polished.

DeepSeek-R1: Addressed these concerns with cold-start fine-tuning, making responses clearer and more structured.

3. Performance

DeepSeek-R1-Zero: Still an extremely strong reasoning design, often beating OpenAI’s o1, but fell the language mixing concerns minimized usability significantly.

DeepSeek-R1: Outperforms R1-Zero and OpenAI’s o1 on a lot of reasoning standards, and the reactions are far more polished.

Simply put, DeepSeek-R1-Zero was an evidence of principle, while DeepSeek-R1 is the totally enhanced variation.

How DeepSeek-R1 was trained

To deal with the readability and coherence concerns of R1-Zero, the researchers incorporated a cold-start fine-tuning phase and a multi-stage training pipeline when constructing DeepSeek-R1:

Cold-Start Fine-Tuning:

– Researchers prepared a top quality dataset of long chains of idea examples for initial supervised fine-tuning (SFT). This information was collected using:- Few-shot triggering with in-depth CoT examples.

– Post-processed outputs from DeepSeek-R1-Zero, improved by human annotators.

Reinforcement Learning:

DeepSeek-R1 underwent the very same RL procedure as DeepSeek-R1-Zero to refine its reasoning abilities even more.

Human Preference Alignment:

– A secondary RL phase improved the model’s helpfulness and harmlessness, guaranteeing better positioning with user requirements.

Distillation to Smaller Models:

– DeepSeek-R1’s thinking capabilities were distilled into smaller sized, efficient designs like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.

DeepSeek R-1 benchmark performance

The researchers evaluated DeepSeek R-1 throughout a range of standards and designs: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.

The criteria were broken down into a number of categories, shown below in the table: English, Code, Math, and Chinese.

Setup

The following parameters were used throughout all models:

Maximum generation length: 32,768 tokens.

Sampling setup:- Temperature: 0.6.

– Top-p value: 0.95.

– DeepSeek R1 exceeded o1, Claude 3.5 Sonnet and other models in the bulk of thinking criteria.

o1 was the best-performing model in 4 out of the five coding-related criteria.

– DeepSeek performed well on innovative and long-context task task, like AlpacaEval 2.0 and ArenaHard, outshining all other models.

Prompt Engineering with reasoning models

My favorite part of the post was the scientists’ observation about DeepSeek-R1’s level of sensitivity to prompts:

This is another datapoint that aligns with insights from our Prompt Engineering with Reasoning Models Guide, which referrals Microsoft’s research study on their MedPrompt structure. In their research study with OpenAI’s o1-preview model, they discovered that overwhelming thinking models with few-shot context broken down performance-a sharp contrast to non-reasoning models.

The key takeaway? Zero-shot triggering with clear and concise directions appear to be best when utilizing thinking models.