关注本地模型部署、推理引擎、量化和工作站运行工具。
test-llama-archs : generate dummy test vocab (#29084) Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp **Website:** - **Attestations:** - **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b11049/llama-
metal : support qwen4exp hc ops (#29000) Add support for the new DSV4 HC op variants used by qwen4exp: - hc_pre with per-element sigmoid gate (gated variant) - hc_post with identity mixing (comb == nullptr) Assisted-by: pi:llama.cpp/Qwen3.8-27B **Website:** -
## What's Changed `GET /api/show` now advertises each model's thinking controls and default: ```sh curl http://localhost:11434/api/show -d '{"model": "glm-5.3-flash:cloud"}' ``` ```json { "thinking": { "values": ["low", "high", "max"], "default": "max" } } ```
cuda : fix CUB argsort corruption caused by in-place keys (#28389) argsort_f32_i32_cuda_cub called the one-shot DeviceRadixSort::SortPairs API with d_keys_in == d_keys_out (temp_keys, temp_keys). CUB's internal double-buffer ping-pong requires distinct key buf
opencl: add support for bin kernel `flash_attn_f32_f16_bin` (#29046) * opencl: add `flash_attn_f32_f16_bin` * opencl: guarded prefill fa **Website:** - **Attestations:** - **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/rel
# Highlights *713 PRs from 237 contributors.* **New models in this release** (see the [cookbook](https://docs.sglang.io/cookbook) for all supported models): | Model | Type | PRs | Cookbook | |---|---|---|---| | GLM-5.3-Flash | Autoregressive | [#36507](https:/
hexagon: add ROLL op support (#29105) **Website:** - **Attestations:** - **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b11045/llama-b11045-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI e
hexagon: im2col update (#29103) * ggml-hexagon: accept 1D and padded IM2COL ops * ggml-hexagon: make pure-DDR IM2COL kernel is_2D-aware * ggml-hexagon: extend IM2COL DMA patch-embed fast path to 1D * ggml-hexagon: add blocked-staging general IM2COL DMA kernel
api: expose model thinking levels and defaults (#18473)
Amazon SageMaker AI shipped 13 inference launches in year-to-date across two deployment paths: fully managed endpoints and Amazon SageMaker HyperPod Inference. This post reviews each launch, from inference recommendations and capacity-aware instance pools to t
hexagon: HMX flash-attention head_dim padding (support DK=DV=72) (#26539) Allow HMX flash-attention to run with head_dim not a multiple of 64 (e.g. SigLIP head_dim=72), by operating on DK/DV rounded up to 64 with zero-filled tail lanes. **Website:** - **Attest
You’re deploying a model on a system. It starts up, prompts are getting responses. Now the hard question: Is this fast? Your instincts might lead you to send...
opencl: add bin kernel `kernel_gemm_noshuffle_q6_k_f32_32b_trans_ila_a8_bin` (#28678) * opencl: add A8 Q6_K non-MoE binary kernel * opencl: fix layout compatibility **Website:** - **Attestations:** - **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://githu
ggml : check for allocation failures to prevent crashes (#28149) * ggml : check for allocation failures to prevent crashes * wording **Website:** - **Attestations:** - **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/release
Model-Saver: Write the SWA pattern, 15 more architectures roundtrip (#29042) * llama: read the SWA pattern as a period or a per-layer array Add llama_model_base::load_swa_pattern(), which reads sliding_window_pattern either as one flag per layer or as a period
Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing add-on for Amazon EKS. It uses real-time GPU signals to send each inference request to the best-suited pod, cutting first-token latency by up to 82% with no changes to your m
ggml-webgpu: fix supports_op condition for GET_ROWS (#28978) * fix get_rows vec4 handling * Add src strides checking to vec4_aligned of get_rows and the new test case. **Website:** - **Attestations:** - **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://gi
ggml : handle graph buffer reservation failure (#26070) **Website:** - **Attestations:** - **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b11036/llama-b11036-bin-macos-arm64.tar.gz) - macOS Apple Silicon
vulkan: add IQ3_S MMQ matmul kernels (#28822) * vulkan: add IQ3_S MMQ matmul kernels * Make block_a_to_shmem do 2-byte loads (110 bytes is divisible by 2) * Align the check, IQ3_S is also using K tile size **Website:** - **Attestations:** - **macOS/iOS:** - [m
### Summary of major features and improvements * #### More GenAI coverage and framework integrations to minimize code changes * New models supported: * On CPU: Gemma-3n * On CPU, GPU: Kokoro-82M, Qwen3-VL-4B with EAGLE-3, Qwen3-ASR, Muse Glimmer 30B, Qwen3.8 2
## What's Changed - Added first-run setup when running `ollama`, with options to sign in or continue locally. Setup completion is shared with the desktop app on macOS and Windows. - Added `ollama://apps` to open the desktop app’s Apps page directly on macOS an
cli: add first-run onboarding shared with the desktop app (#18495)
The decode loop releases MLX's pool of freed buffers every 256 generated tokens, which is also how often the KV cache grows and drops its previous, smaller buffers. The check fires only when the token count lands exactly on a multiple of 256. Speculative decod
model is one package with three jobs: the contract between the runner and the architectures, the opened checkpoint, and building nn layers from checkpoint tensors. Its files did not say which was which. base.go carried the folded package's name over the interf
Nanya-backed DRAM designer PieceMakers began trading in Taipei on Sept. 16 on a bet that AI inference memory won’t be HBM. <p>Shane has a background in computer engineering and has worked as a freelance consultant in multiple industries. He has a strong affect
System performance, efficient infrastructure scaling and continuous software optimization are key levers that determine AI inference economics. Higher system performance means more tokens generated, resulting in higher revenue. Efficient scaling means throughp
llama.cpp build changes resulted in duplicate symbols between libllama and libmtmd. This moves the compat patch into libllama with exported symbols.
## What's Changed * MLX safetensors `ollama create` no longer experimental. GGUF model creation now requires using llama.cpp tooling for safetensor conversion and quantization. * Improved MLX memory handling on Apple Silicon * Runaway repeat token detection no
Power is a defining constraint for AI factories. As AI workloads demand a full compute platform to serve them, each component of that platform must maximize...
Manually tagging thousands of catalog products is slow and inconsistent. This walkthrough shows how to customize Qwen3-8B with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR) on Amazon SageMaker serverless model customiza
