Ocean Finance

Open-R1: a Completely Open Reproduction Of DeepSeek-R1

Hey there! This article is an intro to the task, not a claim that we’ve recreated R1 yet. We’re integrating in the open, so as quickly as we have evaluation numbers, we’ll share them. You can follow our progress on Hugging Face and GitHub.

True, however it appears like there’s absolutely nothing to be examined as of today. I assume the supreme goal is to train a brand-new reasoning design and then utilize the exact same evaluation metrics as o1 and the DeepSeek-R1.

Well, there ought to be at least some peace of mind check and recognition to guarantee the design was trained correctly.

Oh yes, if you are talking about the assessment number of deepseek’s design it’s coming very quickly!

As discussed in the post there is no design called Open-R1 to check at all … not yet anyhow. This is a blog describing that Hugging face will take the R1 Deepseek design, exercise how it was constructed as described in the paper and from what they released, and then duplicate that process.

in reality this is basically how science works … A comes up with a plan, discovery or development and it is evaluated by B, C and D to see if it is reproduceable. Thats been the cornerstone of research now for a few centuries.

This blog is not stating they have currently done so … Its a blog describing an intent to start training a design like R1 and calling it Open-R1.

Also DeepSeek-R1 was only released recently, and even in their paper they outlined the compute hours required. While those are low compute hours for a SOTA model this does not suggest you can train stated design in a week. I ‘d personally like to be able to train a transformer design in a week, but we might need to wait a while for that level of compute innovation.

So there are no standards for a design that has not been built yet right? As laid out in the blog, and once again in reply to your question.

However fear not, there is a GitHub Repo currently and contributors (hell I might join myself), some prelim work done, and a strategy of attack. A good starting position.

n
@edbeeching
has examined the released models currently

( src: https://x.com/edwardbeeching/status/1884273209136275742)

R1 simply trained on o1 outputs, so jointly …/ s. This is what the new AI czars are stating

Hi! This post is an intro to the job, not a claim that we have actually recreated R1 yet. We will completely share the missing piece when we have them, you can anticipate the models and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

That’s nice and essential to understand this remarkable buzz that lacks technical comprehension and explanation. Science has to do with recreation, and if they claim to be open, let them fullfill the open part.

Please do publish the training expense.

We will!

Excalidraw Hi n
@bojan2501
thanks, we will indeed be striving to make certain this training dish can work for little language models on customer hardware since not everyone has a cluster of H100s at home:-RRB- The tool we used for the images was Excalidraw! https://excalidraw.com

anticipating it! WTF are your discussing?

should be a joke

It’s actually cool to see how the entire open source community comes together!

Ops …

5.5 M is number press reporter in the deepseekv3 tech report (just the training, not the experiment afaik), for R1 tough to estimate tbh but much less than 5.5 M imo

Historically, they have never launched code or datasets of their LLM training, so I would not anticipate this time to be different. If they would launch it that would be remarkable of course!

Yes obviously!

So essentially you’re asking to change existing censorship with another flavour of censorship?

The code for the designs are inside the design repositories, e.g. for V3: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py

Hello Team, I’m Ray Bernard, the author and developer of EQUATOR. My research group will be dealing with a paper focused on replicating particular parts of DeepSeek R1. Our aim is to replicate the cold start and supply your group with a dataset that consists of COT and other methods to support these efforts. We like to contribute our work to assist. Please let me understand if you find this beneficial. Best, Ray Bernard https://www.facebook.com/groups/1186310571520299/

Where is the assessment numbers? without it you can’t call it reproduction.

8 replies

True, however it seems like there’s nothing to be assessed since right now. I assume the is to train a new reasoning model and after that use the exact same evaluation metrics as o1 and the DeepSeek-R1.

That’s rather interesting, I was asking myself why the questions the author exposed here are not being asked by others? I believe the work they have done is unforgettable but at the exact same time I wonder why they would not put these missing pieces on if they are expected to be fully open.
Why even without recreation and understanding of the innovation they could affect a lot the market in this method?

4 replies

Hi! This article is an intro to the job, not a claim that we have actually recreated R1 yet. We will completely share the missing piece when we have them, you can expect the models and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

Interesting read, and it is good that we see more effort into this direction: more optimization and less strength.
Also wonder what tool did the author use for creating action diagram.

2 replies

Excalidraw I’m so delighted that initiative like this currently exist, I’m gon na try to contribute:-RRB- 1 reply

looking forward to it! So racist articel

2 replies

WTF are your speaking about?

Awesome to have this open recreation started!

For Step # 1 check out https://github.com/open-thoughts/open-thoughts!

https://x.com/ryanmart3n/status/1884284101265612856

Let’s do this thing!

1 reply

It’s actually cool to see how the entire open source community comes together!

Does anybody understand the real training cost of r1? I can’t discover it in the paper or the announcement post. Is the 6M expense reported by media simply the number drawn from v3’s training cost?

2 replies

Ops …

Has anybody asked the DeepSeek group to publish their training data and code, or a minimum of share them privately with an independent replication project like this? Have they rejected such a request?

A devoted duplication depends on utilizing the same dataset and hyperparameters. Otherwise, any major discrepancies with the released benchmarks would be difficult to pin down-whether due to training data differences or the duplication technique itself.

1 reply

Historically, they have actually never launched code or datasets of their LLM training, so I wouldn’t anticipate this time to be various. If they would launch it that would be incredible obviously!

In the meantime we have to make finest guess price quotes and see if we can arrive ourselves.

You supply excellent replication procedure of Deepseek reasoning training. I will try something comparable to it.

This is really great details, can we fine tune with specific use case when code is released?

1 reply

Yes obviously!

Please consider eliminating biased, polluted or unaligned training data and make an effort to eliminate copyrighted works from the crawl from intake. This will make the design more usable. If you reused anthropic curation checks, this might likewise help, get rid of obviouslybiased information will likely add a great deal of value. We do not want another tainted, unaligned open source model, right? And no corporate would ever utilize deepseek or a model that reuses it, right?
We value your work for the benefit of humanity, we hope.
Miike C from NJ

1 reply

So generally you’re asking to replace existing censorship with another flavour of censorship?

Can’t wait! Hopefully the design will be uncensored however whatever you can do is alright! Love seeing open source structure itself up. I’m not smart enough to actually assist however I can contribute moral assistance lol

Hello guys, I am even simply looking for code for DeepSeek-V2, in order to fully comprehend multi-head hidden attention. You do not appear to have code in Hugging Face even for that. Or am I missing out on something? Don’t see anything in src/transformers/models. MLA is not properly explained in their paper, so it would be necessary to have code for this.