When Mark Zuckerberg isn't wake surfing in a tuxedo and a puka shell necklace at his Lake Tahoe mansion, he's battling Google and OpenAI for artificial intelligence supremacy. Yesterday, Meta released its biggest and baddest large language model ever, which also happens to be free and arguably open source. It took months to train on 16,000 Nvidia H100 GPUs, likely costing hundreds of millions of dollars and using enough electricity to power a small country. The end result is a massive 405 billion parameter model with a 128,000 token context length, which according to benchmarks is mostly superior to OpenAI's GPT-4 and even beats Claude 3.5 on some key benchmarks. But benchmarks lie, and the only way to find out if a new model is any good is to vibe with it.

Llama 3.1 is a model that cannot be ignored. It comes in three sizes: 8B, 70B, and 405B, where B refers to billions of parameters, or the variables that the model can use to make predictions. In general, more parameters can capture more complex patterns, but more parameters don't always mean that the model is better. GPT-4 has been rumored to have over 1 trillion parameters, but we don't really know the true numbers from companies like OpenAI and Anthropic.

The cool thing about Llama is that it's open source, well, kind of. You can make money off of it as long as your app doesn't have 700 million monthly active users, in which case you need to request a license from Meta. What's not open source is the training data, which might include your blog, your GitHub repos, all your Facebook posts from 2006, and maybe even your WhatsApp messages.

Interestingly, we can take a look at the actual code used to train this model, which is only 300 lines of Python and PyTorch along with a library called FairScale to distribute training across multiple GPUs. It's a relatively simple decoder-only transformer, as opposed to the mixture of experts approach used in a lot of other big models like its biggest open-source rival, Mistral. Most importantly, the model weights are open, and that's a huge win for developers building AI-powered apps. Now you don't have to pay a bunch of money to use the GPT-4 API; instead, you can self-host your own model and pay a cloud provider to rent some GPUs.

The big model would not be cheap to self-host. I used OLama to download it and use it locally, but the weights weigh 230GB, and even with an RTX 4090, I wasn't able to ride this llama. The good news is that you can try it for free on platforms like Meta or Nvidia's Playground. The initial feedback from random users on the internet is that big Llama is somewhat disappointing, while the smaller Llamas are quite impressive.

The real power of Llama is that it can be fine-tuned with custom data. In the near future, we will have some amazing uncensored fine-tuned models like Dolphin. My favorite test for new LLMs is to ask it to build a Svelte 5 web application with Runes, a new yet-to-be-released feature. The only model I've seen do this correctly in a single shot is Claude 3.5, and Llama 405B failed pretty miserably and seems to have no awareness of this feature.

Overall, Llama 3.1 is decent at coding but still clearly behind Claude. It performed well in creative writing and poetry, but it wasn't the best I've seen. If we take a minute to reflect, what's crazy is that we have multiple different companies that have trained massive models with massive computers, and they're all plateauing at the same level of capability. OpenAI was the first to make a huge leap from GPT-3 to GPT-4, but since then, it's only been small incremental gains.

Last year, Sam Altman practically begged the government to regulate AI to protect humanity, but a year later, we still haven't seen the apocalyptic Skynet human extinction event that they promised us. AI still hasn't even replaced programmers. It's like that time airplanes went from propellers to jet engines, but the advancement to light speed engines never happened. When talking about LLMs, artificial superintelligence is still nowhere in sight except in the imagination of the Silicon Valley mafia.

It feels wrong to say this, but Meta is really the only big tech company keeping it real in the AI space. I'm sure there's an evil ulterior motive hidden in there somewhere, but Llama is one small step for man, one giant leap for Zuckerberg's redemption arc. This has been The Code Report. Thanks for watching, and I will see you in the next one.