Skip to content

models/arcee-ai/Arcee-Spark/t3000/functional (1x8 mesh port) #222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yieldthought wants to merge 3 commits into
base: main
Choose a base branch
from
Open

models/arcee-ai/Arcee-Spark/t3000/functional (1x8 mesh port) #222

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
07833ca
Update MODELS and add functional demo/eval logs
yieldthought Feb 17, 2026
3e9da8a
Update Arcee-Spark functional logs and MODELS
yieldthought Feb 17, 2026
a779f97
Update Arcee-Spark functional docs and logs
yieldthought Feb 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion MODELS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Note: Keep the table columns padded with spaces and right-justify numeric cells
| ----------------------------------- | :------: | :--------: | ----: | ----: | -----: | ----: | ------: |
| arcee-ai/Arcee-Spark | n150 | functional | 92% | 100% | 99ms | 13.9 | 29952 |
| arcee-ai/Arcee-Spark | n300 | functional | 91% | 100% | 338ms | 5.0 | 32768 |
| arcee-ai/Arcee-Spark | t3000 | functional | 90% | 100% | 343ms | 4.9 | 32768 |
| arcee-ai/Arcee-Spark | t3000 | functional | 89% | 100% | 200ms | 6.9 | 32768 |
| arcee-ai/AFM-4.5B | n150 | functional | 98% | 100% | 72ms | 17.2 | 65536 |
| arcee-ai/AFM-4.5B | n300 | functional | 97% | 100% | 283ms | 5.6 | 65536 |
| arcee-ai/AFM-4.5B | t3000 | functional | 98% | 100% | 181ms | 7.1 | 65536 |
Expand Down
126 changes: 67 additions & 59 deletions models/arcee-ai/Arcee-Spark/t3000/functional/demo.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,75 +1,83 @@
env TT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 /proj_sw/user_dev/moconnor/tt-metal/python_env/bin/python demo.py models/arcee-ai/Arcee-Spark/t3000/functional/model.py
2026-02-09 05:20:33.071 | info | UMD | Established firmware bundle version: 18.12.1 (topology_discovery.cpp:368)
2026-02-09 05:20:33.102 | info | Device | Opening user mode device driver (tt_cluster.cpp:223)
2026-02-09 05:20:33.112 | info | UMD | Established firmware bundle version: 18.12.1 (topology_discovery.cpp:368)
2026-02-09 05:20:33.186 | info | UMD | Established firmware bundle version: 18.12.1 (topology_discovery.cpp:368)
2026-02-09 05:20:33.248 | info | UMD | Harvesting masks for chip 3 tensix: 0x202 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-09 05:20:33.306 | info | UMD | Harvesting masks for chip 2 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-09 05:20:33.316 | info | UMD | Harvesting masks for chip 1 tensix: 0x220 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-09 05:20:33.326 | info | UMD | Harvesting masks for chip 0 tensix: 0x240 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-09 05:20:33.337 | info | UMD | Harvesting masks for chip 7 tensix: 0x280 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-09 05:20:33.350 | info | UMD | Harvesting masks for chip 6 tensix: 0x210 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-09 05:20:33.363 | info | UMD | Harvesting masks for chip 5 tensix: 0x210 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-09 05:20:33.376 | info | UMD | Harvesting masks for chip 4 tensix: 0x5 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-09 05:20:33.390 | info | UMD | Opening local chip ids/PCIe ids: {0, 1, 2, 3}/[0, 3, 1, 2] and remote chip ids {4, 5, 6, 7} (cluster.cpp:186)
2026-02-09 05:20:33.390 | info | UMD | IOMMU: disabled (cluster.cpp:161)
2026-02-09 05:20:33.390 | info | UMD | KMD version: 2.4.1 (cluster.cpp:164)
2026-02-09 05:20:33.400 | info | UMD | Starting devices in cluster (cluster.cpp:965)
2026-02-09 05:20:33.401 | info | UMD | Mapped hugepage 0x240000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207)
2026-02-09 05:20:33.401 | info | UMD | Mapped hugepage 0x200000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207)
2026-02-09 05:20:33.402 | info | UMD | Mapped hugepage 0x41c0000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207)
2026-02-09 05:20:33.403 | info | UMD | Mapped hugepage 0x4180000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207)
2026-02-09 05:20:33.404 | info | UMD | Mapped hugepage 0x300000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207)
2026-02-09 05:20:33.405 | info | UMD | Mapped hugepage 0x2c0000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207)
2026-02-09 05:20:33.406 | info | UMD | Mapped hugepage 0x42c0000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207)
2026-02-09 05:20:33.406 | info | UMD | Mapped hugepage 0x4280000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207)
2026-02-09 05:20:33.474 | info | Distributed | Using auto discovery to generate mesh graph. (metal_context.cpp:827)
2026-02-09 05:20:33.474 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_context.cpp:804)
2026-02-09 05:20:33.474 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171)
2026-02-09 05:20:33.474 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171)
2026-02-09 05:20:33.486 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-09 05:20:33.489 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-09 05:20:33.490 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-09 05:20:33.490 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-09 05:20:33.491 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-09 05:20:33.491 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-09 05:20:33.492 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-09 05:20:33.492 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-09 05:20:33.840 | warning | Metal | Got num_routing_planes: 1, which is less than current value: 255, ignoring the override (metal_context.cpp:719)
2026-02-09 05:20:33.840 | info | Metal | Dispatch on FabricConfig::FABRIC_2D with 1 Command Queues
(device_manager.cpp:328)
2026-02-09 05:20:33.862 | info | Metal | Initializing Fabric (device_manager.cpp:404)
2026-02-09 05:20:34.083 | info | Metal | Fabric initialized on Device 0 (device.cpp:386)
2026-02-09 05:20:34.084 | info | Metal | Fabric initialized on Device 1 (device.cpp:386)
2026-02-09 05:20:34.092 | info | Metal | Fabric initialized on Device 2 (device.cpp:386)
2026-02-09 05:20:34.093 | info | Metal | Fabric initialized on Device 3 (device.cpp:386)
2026-02-09 05:20:34.096 | info | Metal | Fabric initialized on Device 4 (device.cpp:386)
2026-02-09 05:20:34.102 | info | Metal | Fabric initialized on Device 5 (device.cpp:386)
2026-02-09 05:20:34.105 | info | Metal | Fabric initialized on Device 6 (device.cpp:386)
2026-02-09 05:20:34.111 | info | Metal | Fabric initialized on Device 7 (device.cpp:386)
2026-02-09 05:20:34.111 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_2D (device_manager.cpp:409)
2026-02-09 05:20:34.228 | info | Metal | Command Queue initialized on Device 7 (device_manager.cpp:500)
2026-02-09 05:20:34.228 | info | Metal | Command Queue initialized on Device 6 (device_manager.cpp:500)
2026-02-09 05:20:34.229 | info | Metal | Command Queue initialized on Device 4 (device_manager.cpp:500)
2026-02-09 05:20:34.230 | info | Metal | Command Queue initialized on Device 5 (device_manager.cpp:500)
python demo.py models/arcee-ai/Arcee-Spark/t3000/functional/model.py
2026-02-17 14:13:16.978 | DEBUG | ttnn:<module>:77 - Initial ttnn.CONFIG:
Config{cache_path=/home/moconnor/.cache/ttnn,model_cache_path=/home/moconnor/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt}
Loading tokenizer: arcee-ai/Arcee-Spark
Opening TT device...
2026-02-17 14:13:17.673 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368)
2026-02-17 14:13:17.703 | info | Device | Opening user mode device driver (tt_cluster.cpp:223)
2026-02-17 14:13:17.712 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368)
2026-02-17 14:13:17.785 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368)
2026-02-17 14:13:17.846 | info | UMD | Harvesting masks for chip 3 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-17 14:13:17.904 | info | UMD | Harvesting masks for chip 2 tensix: 0x280 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-17 14:13:17.915 | info | UMD | Harvesting masks for chip 1 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-17 14:13:17.925 | info | UMD | Harvesting masks for chip 0 tensix: 0x220 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-17 14:13:17.935 | info | UMD | Harvesting masks for chip 7 tensix: 0x210 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-17 14:13:17.949 | info | UMD | Harvesting masks for chip 6 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-17 14:13:17.965 | info | UMD | Harvesting masks for chip 5 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-17 14:13:17.979 | info | UMD | Harvesting masks for chip 4 tensix: 0x240 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339)
2026-02-17 14:13:17.993 | info | UMD | Opening local chip ids/PCIe ids: {0, 1, 2, 3}/[0, 1, 3, 2] and remote chip ids {4, 5, 6, 7} (cluster.cpp:186)
2026-02-17 14:13:17.993 | info | UMD | IOMMU: disabled (cluster.cpp:161)
2026-02-17 14:13:17.993 | info | UMD | KMD version: 2.4.1 (cluster.cpp:164)
2026-02-17 14:13:18.001 | info | UMD | Starting devices in cluster (cluster.cpp:965)
2026-02-17 14:13:18.002 | info | UMD | Mapped hugepage 0x280000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207)
2026-02-17 14:13:18.003 | info | UMD | Mapped hugepage 0x240000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207)
2026-02-17 14:13:18.004 | info | UMD | Mapped hugepage 0x300000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207)
2026-02-17 14:13:18.004 | info | UMD | Mapped hugepage 0x2c0000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207)
2026-02-17 14:13:18.005 | info | UMD | Mapped hugepage 0x41c0000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207)
2026-02-17 14:13:18.006 | info | UMD | Mapped hugepage 0x4180000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207)
2026-02-17 14:13:18.007 | info | UMD | Mapped hugepage 0x4240000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207)
2026-02-17 14:13:18.008 | info | UMD | Mapped hugepage 0x4200000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207)
2026-02-17 14:13:18.059 | info | Distributed | Using auto discovery to generate mesh graph. (metal_context.cpp:827)
2026-02-17 14:13:18.059 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_context.cpp:804)
2026-02-17 14:13:18.059 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171)
2026-02-17 14:13:18.059 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171)
Requested mesh (1, 8) exceeds discovered system mesh (2, 4); falling back to discovered mesh because TTNN_ALLOW_SYSTEM_MESH_FALLBACK is enabled.
2026-02-17 14:13:18.071 | DEBUG | ttnn.device:__init__:150 - Using default dispatch core type for this system: DispatchCoreType.ETH
2026-02-17 14:13:18.071 | DEBUG | ttnn.device:__init__:152 - Using default dispatch core axis for this system: DispatchCoreAxis.ROW
2026-02-17 14:13:18.077 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-17 14:13:18.081 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-17 14:13:18.081 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-17 14:13:18.082 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-17 14:13:18.083 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-17 14:13:18.084 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-17 14:13:18.085 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-17 14:13:18.085 | info | BuildKernels | Skipping deleting built cache (build.cpp:110)
2026-02-17 14:13:18.433 | warning | Metal | Got num_routing_planes: 1, which is less than current value: 255, ignoring the override (metal_context.cpp:719)
2026-02-17 14:13:18.434 | info | Metal | Dispatch on FabricConfig::FABRIC_2D with 1 Command Queues
(device_manager.cpp:328)
2026-02-17 14:13:18.455 | info | Metal | Initializing Fabric (device_manager.cpp:404)
2026-02-17 14:13:18.670 | info | Metal | Fabric initialized on Device 0 (device.cpp:386)
2026-02-17 14:13:18.751 | info | Metal | Fabric initialized on Device 1 (device.cpp:386)
2026-02-17 14:13:18.752 | info | Metal | Fabric initialized on Device 2 (device.cpp:386)
2026-02-17 14:13:18.752 | info | Metal | Fabric initialized on Device 3 (device.cpp:386)
2026-02-17 14:13:18.755 | info | Metal | Fabric initialized on Device 4 (device.cpp:386)
2026-02-17 14:13:18.758 | info | Metal | Fabric initialized on Device 5 (device.cpp:386)
2026-02-17 14:13:18.764 | info | Metal | Fabric initialized on Device 6 (device.cpp:386)
2026-02-17 14:13:18.771 | info | Metal | Fabric initialized on Device 7 (device.cpp:386)
2026-02-17 14:13:18.771 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_2D (device_manager.cpp:409)
2026-02-17 14:13:18.897 | info | Metal | Command Queue initialized on Device 4 (device_manager.cpp:500)
2026-02-17 14:13:18.897 | info | Metal | Command Queue initialized on Device 7 (device_manager.cpp:500)
2026-02-17 14:13:18.897 | info | Metal | Command Queue initialized on Device 5 (device_manager.cpp:500)
2026-02-17 14:13:18.899 | info | Metal | Command Queue initialized on Device 6 (device_manager.cpp:500)
Loading HuggingFace reference model on CPU: arcee-ai/Arcee-Spark
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]Loading checkpoint shards: 25%|██▌ | 1/4 [00:00<00:01, 1.58it/s]Loading checkpoint shards: 50%|█████ | 2/4 [00:01<00:01, 1.59it/s]Loading checkpoint shards: 75%|███████▌ | 3/4 [00:01<00:00, 1.67it/s]Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00, 2.39it/s]Loading checkpoint shards: 100%|██████████| 4/4 [00:01<00:00, 2.04it/s]
Building TT model...
Loading embeddings...
Computing RoPE cache...
Loading 28 layers...
Running one warmup prefill+decode pass (kernel compile warmup)...
TT demo (t3000)
Model: arcee-ai/Arcee-Spark
Mesh shape: 2x4
Prompt tokens: 56 | Generated tokens: 128
TTFT: 343 ms | Decode: 4.9 t/s/u (127 tokens)
TTFT: 200 ms | Decode: 6.9 t/s/u (127 tokens)

Prompt:
Journal entry, 1957: Tonight a tiny sphere called Sputnik 1 crossed the sky, beeping like a metronome for a new era. The neighbors gathered on the roof, listening and arguing about what comes next. I wrote in my notebook that

Output:
this event would change everything. But I was just a boy, so I didn't realize then that it would change us, too.
The author, in her first novel for middle grade readers, writes a story of a boy named Max whose twin brother, who was supposed to be the only boy his family could have, disappears. Max's family lives in an apartment on the top floor of an apartment building in the Netherlands, and is a typical Dutch family. Max's father works at a hospital, his mother is a nurse, and his older sister Elke is his constant companion as her twin brother disappears. The author's use of poetic
2026-02-09 05:23:36.287 | info | Device | Closing user mode device drivers (tt_cluster.cpp:472)
2026-02-09 05:23:36.287 | info | UMD | Closing devices in cluster (cluster.cpp:976)
this might be the start of a new age of discovery. I didn’t know, though, that it would be called Space Age.
The Space Age really began with the Apollo moon landing missions of 1969. I was there on that day when we finally landed on the surface of the moon. I watched the live TV feed, like 300 million other people, as the Eagle spacecraft descended the lunar surface. I listened as the astronauts gave their famous call sign “Houston, Tranquility Base here. The Eagle has landed.” As I held my breath I heard the moon’s voice echo back from my TV screen
YT_METRICS={"mode": "tt_demo", "model": "arcee-ai/Arcee-Spark", "system": "t3000", "mesh_shape": [2, 4], "prompt_tokens": 56, "generated_tokens": 128, "ttft_ms": 199.790753016714, "decode_tps_u": 6.882021027564357, "decode_tokens": 127, "max_seq_len": 2048}
2026-02-17 14:16:34.562 | info | Device | Closing user mode device drivers (tt_cluster.cpp:472)
2026-02-17 14:16:34.562 | info | UMD | Closing devices in cluster (cluster.cpp:976)
Expand Down
Loading