Nvidia’s new coding LLM will make you a better programmer and can run on a CPU

Key Takeaways

  • ServiceNow, Hugging Face, and Nvidia collaborated to create StarCoder2, a versatile LLM for generating code.
  • StarCoder2 sets the bar high with smaller parameters, yet impressive performance when compared to larger models.
  • The algorithm, with a vast range of programming languages, aims to boost transparency and maximize user trust.

If you’re a programmer looking to make your life a little bit easier, ServiceNow, Hugging Face, and Nvidia have released StarCoder2, an LLM that you can run locally to generate code. Trained on 619 languages, the 15 billion parameter version of StarCoder2 was built by Nvidia and is the largest of the lot. ServiceNow built the three billion parameter version, and Hugging Face built the seven billion parameter version.

StarCoder2 was built by the BigCode community, and the inspiration behind the platform is that it takes into account transparency and cost-effectiveness, while respecting the wishes of developers on GitHub. It only uses code from repositories with a permissive license attached, and it only uses public code. The collection of source code that it’s trained on is called “The Stack v2” and contains 67.5TB of code. The de-duplicated version comes in at 32.1TB of code, which is still a lot.


Nvidia’s Chat with RTX will connect an LLM with YouTube videos and documents locally on your PC

Nvidia is making it even easier to run a local LLM with Chat with RTX, and it’s pretty powerful, too.

StarCoder2 is one of the best-performing free code generation models

StarCoder2 in humaneval

The benefit of an LLM trained specifically for coding is that it can have significantly smaller parameters, making it more portable. In StarCoder2’s research paper, it’s noted that the 15 billion parameter model consistently matches or even outperforms CodeLlama-34B, a model twice its size. Even StarCoder2’s 3 billion parameter model outperforms the original StarCoder’s 15 billion parameter model, an incredible feat.

As for the differences between the small, medium, and large models, those primarily come down to programming languages and training data. While the largest model has 619 programming languages, the seven billion and three billion parameter models cuts this down to just 17. Those 17 languages are:

C, C++
















StarCoder2 has a context window of 16,000 tokens, making it perfectly apt for small to medium size codebases. The 15 billion parameter model also manages to achieve a 46.3% score in the HumanEval benchmark, where the original StarCoder only achieved 29.3%. Maintaining competitivity, while StarCoder2’s 15 billion parameter doesn’t outperform DeepSeekCoder-33B (widely regarded as the best coding model out there), it comes pretty close for a model half the size.

In the research paper, the teams behind StarCoder2 say that “By not only releasing the model weights but also ensuring complete transparency regarding the training data, we hope to increase trust in the developed models and empower other engineering teams and scientists to build upon our efforts.”

How to use StarCoder2

StarCoder2 can be found on Hugging Face, and Nvidia has also shared instructions on how to customize and deploy the model yourself. You can run it on a CPU or on an Nvidia graphics card, and the smaller variants will be more forgiving to run on any platform with less RAM. They can all be deployed using Python, and Hugging Face has instructions for each model and how to use it on your own computer at home.


Best GPUs in 2024: Our top graphics card picks

Picking the right graphics card can be difficult given the sheer number of options on the market. Here are the best graphics cards to consider.

(Visited 12 times, 1 visits today)

Related posts