Meta Platforms has unveiled its next-generation LLMs under the Llama 4 banner. There are two key models available for open source download:
Llama 4 Scout: Designed as a highly efficient, multimodal model optimized for personalized applications.
Llama 4 Maverick: Geared toward more demanding tasks in multimodal processing and chat, featuring a larger set of experts.
In addition, Meta previewed Llama 4 Behemoth — a teacher model currently still in training — which is expected to set new benchmarks in reasoning, STEM tasks, and advanced coding capabilities. These models mark a significant upgrade over previous iterations, both in architecture and performance.
Llama 4 Scout: Key Features and Technical Specifications
Model Size and Efficiency
Active Parameters & Experts: Llama 4 Scout is built with 17 billion active parameters complemented by 16 experts. This design allows the model to be highly specialized while keeping the inference cost manageable.
Hardware Efficiency: One of the most notable achievements is that Llama 4 Scout is small enough to fit on a single NVIDIA H100 GPU when using Int4 quantization. This makes it highly accessible for developers and enterprises without large-scale hardware.
Context Window: The model supports an industry-leading context window of 10 million tokens. This extended context capacity is crucial for tasks that require long document processing and maintaining coherence over extended interactions.
Multimodal Capabilities
Multimodal Integration: Llama 4 Scout isn’t limited to text. It’s designed to process and integrate data across multiple formats — text, images, video, and audio — through a process known as early fusion. This allows the model to jointly learn from diverse datasets, offering improved performance in tasks that blend modalities.
Llama 4 Family: Maverick and Behemoth
Llama 4 Maverick
Enhanced Expert Layers: While Maverick shares the 17 billion active parameter count, it distinguishes itself by incorporating 128 experts. This design enhances its ability to manage more complex tasks, especially in contexts that involve intricate reasoning and precise image understanding.
Performance & Cost Efficiency: Llama 4 Maverick is optimized for a lower cost-to-performance ratio. It achieves competitive results when benchmarked against models such as GPT-4o and Gemini 2.0 Flash, particularly in reasoning, coding, and multilingual tasks.
Deployment: Similar to Scout, Maverick is designed to be deployable on single high-performance hardware setups (e.g., a single NVIDIA H100 DGX host), making it appealing for both research and commercial applications.
Llama 4 Behemoth (Teacher Model)
Scale and Capabilities: With 288 billion total parameters (though only a fraction is activated per token due to the mixture-of-experts design), Behemoth is positioned as one of the smartest LLMs globally. It is specifically crafted to serve as a teacher for the other models.
Benchmark Performance: Early evaluations suggest that Llama 4 Behemoth outperforms competitive models like GPT-4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro on STEM benchmarks, which include advanced math and reasoning challenges.
Ongoing Development: Although Behemoth is still undergoing training, Meta’s preview indicates that its innovations in large-scale model training will be key to future improvements in the entire Llama ecosystem.
Architectural Innovations and Training Process
Mixture-of-Experts (MoE) Architecture
Efficient Parameter Utilization: The Llama 4 series introduces a mixture-of-experts architecture, which activates only a subset of the total parameters for each token. For instance, while Llama 4 Maverick has 400 billion total parameters, only a fraction (17 billion active parameters) is used per inference. This selective activation reduces computational cost and improves latency.
Alternating Layers: The architecture employs alternating dense and MoE layers to balance efficiency and performance. MoE layers route tokens to one of many specialized experts along with a shared expert, ensuring that each token benefits from both general and specialized processing.
Pre-training Techniques
Multimodal Pre-training: Early fusion of text and vision tokens allows the model to be jointly pre-trained on a mix of text, image, and video data. An improved vision encoder, based on MetaCLIP but fine-tuned for the LLM, enhances image understanding.
MetaP Training Technique: A novel training method called MetaP has been adopted to reliably set and transfer critical hyper-parameters such as per-layer learning rates and initialization scales. This technique is key to maintaining consistency across varying model sizes, batch sizes, and data scales.
Large-Scale Data: The training data mixture for Llama 4 includes over 30 trillion tokens from diverse sources, covering more than 200 languages (with over 100 languages having more than 1 billion tokens each). This extensive dataset is more than double the pre-training mixture of Llama 3, ensuring robust multilingual performance.
Post-training and Fine-Tuning Strategies
Curriculum and Data Filtering: To balance multimodal input, reasoning, and conversational ability, Meta implemented a curriculum strategy that emphasizes harder prompts. More than 50% of the easier examples were filtered out using Llama-based evaluations, ensuring that training focused on challenging tasks.
Reinforcement Learning (RL) and Direct Preference Optimization (DPO): The post-training pipeline consists of lightweight supervised fine-tuning (SFT), followed by an online reinforcement learning stage, and concludes with a lightweight DPO phase. This multi-step approach helps balance the model’s exploration capabilities with response quality, particularly in complex reasoning, coding, and math domains.
Continuous Online RL: By alternating between training and using the model itself to filter and retain medium-to-hard difficulty prompts, Meta achieved a step change in performance. This dynamic strategy is key to refining the model’s intelligence and conversational abilities while maintaining high efficiency.
Availability
Meta has made Llama 4 Scout and Llama 4 Maverick available for download via its website and Hugging Face. In addition, these models are being integrated into popular Meta platforms like WhatsApp, Messenger, and Instagram Direct, as well as on the Meta.AI website.
The open-weight nature of these models encourages developers to experiment and build personalized, multimodal experiences. This democratization of high-performance LLMs is expected to spur a wave of new applications and research.
With aggressive investments in AI — reportedly up to $65 billion for this year — Meta is positioning itself to remain competitive. The improvements in reasoning, multimodal integration, and cost efficiency reflect a strategic focus on not only advancing AI research but also enabling practical, scalable solutions.
The Llama 4 series not only sets new benchmarks in the industry but also reaffirms Meta’s commitment to openness and collaboration in AI innovation.
Llama 4 Maverick demonstrates strong all-around performance compared to its rivals Gemini 2.0 Flash, DeepSeek v3.1, and GPT-4o, especially when factoring in its low inference cost. It leads or ties in many of the image reasoning and understanding benchmarks, scoring 73.4 on MMMU, 73.7 on MathVista, 90.0 on ChartQA, and an impressive 94.4 on DocVQA — making it a standout choice for multimodal tasks. These results consistently outperform Gemini and GPT-4o, and significantly surpass DeepSeek, which does not currently support multimodal inputs.
In reasoning and knowledge benchmarks, Llama 4 Maverick holds its own with an 80.5 score on MMLU Pro, just behind DeepSeek’s 81.2, and takes the lead on the GPQA Diamond benchmark with 69.8. It also shows excellent multilingual capabilities, scoring 84.6 on the Multilingual MMLU — slightly above GPT-4o and with no available score from other rivals.
Llama 4 Maverick handles long-context tasks exceptionally well. On the MTOB benchmark, it scores 54.0 and 46.4 for half-book settings and 50.8 and 46.7 for full-book, beating Gemini Flash’s respective scores in both categories. This shows its capability to retain and reason over extended inputs, something especially valuable for tasks like summarization and document QA.
Coding benchmarks show a solid performance with a 43.4 on LiveCodeBench, outperforming Gemini Flash and GPT-4o and coming close to DeepSeek’s top-end score of 49.2. Importantly, Llama 4 Maverick achieves all this while maintaining a cost of $0.19–$0.49 per 1M tokens (3:1 input/output blend), which is dramatically lower than GPT-4o ($4.38) and comparable to Gemini Flash ($0.17), making it one of the most cost-efficient options available.
Baburajan Kizhakedath
