Run Open-Source LLMs Locally on Mac with MLX (Step-by-Step Guide)

When to Use ​

Running open-source LLMs locally on a Mac fits three scenarios: privacy-sensitive work (documents and code never leave the device), offline environments, and low-latency debugging or cost control (no API fees). On October 16, 2026, Meta released the open-weight Muse Spark Open model in partnership with Apple, running the full model locally on two Mac Studios with 512GB memory via MLX — demonstrating MLX's local inference capability.

Step-by-Step Instructions ​

1. Check hardware and OS: Apple Silicon Mac (M1/M2/M3/M4 series), macOS 13 or later, Python 3.9 or later.
2. Install mlx-lm: run pip install mlx-lm in a terminal (use a virtual environment or uv for dependency management).
3. Pick a model by unified memory: 8GB suits 3B–7B quantized models, 32GB can run 30B-class quantized models, 64GB and above can run 70B-class.
4. Download and run the model: run python -m mlx_lm.generate --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit --prompt "Hello"; the first run downloads weights from Hugging Face automatically.
5. Start a local server: run python -m mlx_lm.server --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit --port 8080 for an OpenAI-compatible endpoint at localhost:8080.
6. Quantize your own model: run python -m mlx_lm.convert --hf-path MODEL_DIR -q to produce a 4-bit quantized version that uses less memory.

Notes ​

• Memory is the main bottleneck: memory needed ≈ parameters (in billions) × bits ÷ 8 GB; a 7B model at 4-bit needs roughly 4GB.
• The first run downloads weights over the network; afterwards the model works offline.
• Muse Spark Open (816 billion total / 42 billion active parameters) requires Mac Studios with 512GB memory to run fully locally; regular Macs are better suited to quantized small and medium models.
• MLX only supports Apple Silicon (M series); Intel Macs are not supported.

References ​

• MLX official repository (GitHub)
• mlx-lm documentation
• Meta: Muse Spark Open official blog