Can I run Nemotron 3.5 Lightning on Apple M2 (10-core GPU)?

Q: Can I run Nemotron 3.5 Lightning on Apple M2 (10-core GPU)?

Partially. Nemotron 3.5 Lightning needs CPU offload on Apple M2 (10-core GPU) (24 GB) — expect reduced speed.

Partially — with CPU offload.best quant:

19.9 GB of weights, plus 1.9 GB for the software that runs it and the smallest conversation it can hold, comes to 21.8 GB against the 18 GB this 24 GB device leaves free.

Decode speed

—

Usable context

—

Memory

24 GB

Spills to system RAM

Spills to system RAMest

Too large

What is quantisation? →

Plan B — rent it

Nemotron 3.5 Lightning is hosted by 6 providers from $0.17 per 1M output tokens (Io Net).Compare providers →

Nemotron 3.5 Lightning

31.6B · full specs & benchmarks →

Apple M2 (10-core GPU)

24 GB · everything it runs →