From 07833ca7dec7241d8d8d8a3fb2998f3926a6b49b Mon Sep 17 00:00:00 2001 From: Mark O'Connor Date: Tue, 17 Feb 2026 10:40:38 +0000 Subject: [PATCH 1/3] Update MODELS and add functional demo/eval logs --- MODELS.md | 2 +- .../Arcee-Spark/t3000/functional/demo.log | 130 ++++++++++-------- .../Arcee-Spark/t3000/functional/eval.log | 124 +++++++++-------- 3 files changed, 138 insertions(+), 118 deletions(-) diff --git a/MODELS.md b/MODELS.md index 40a4afa..3330937 100644 --- a/MODELS.md +++ b/MODELS.md @@ -10,7 +10,7 @@ Note: Keep the table columns padded with spaces and right-justify numeric cells | ----------------------------------- | :------: | :--------: | ----: | ----: | -----: | ----: | ------: | | arcee-ai/Arcee-Spark | n150 | functional | 92% | 100% | 99ms | 13.9 | 29952 | | arcee-ai/Arcee-Spark | n300 | functional | 91% | 100% | 338ms | 5.0 | 32768 | -| arcee-ai/Arcee-Spark | t3000 | functional | 90% | 100% | 343ms | 4.9 | 32768 | +| arcee-ai/Arcee-Spark | t3000 | functional | 90% | 100% | 194ms | 7.3 | 32768 | | arcee-ai/AFM-4.5B | n150 | functional | 98% | 100% | 72ms | 17.2 | 65536 | | arcee-ai/AFM-4.5B | n300 | functional | 97% | 100% | 283ms | 5.6 | 65536 | | arcee-ai/AFM-4.5B | t3000 | functional | 98% | 100% | 181ms | 7.1 | 65536 | diff --git a/models/arcee-ai/Arcee-Spark/t3000/functional/demo.log b/models/arcee-ai/Arcee-Spark/t3000/functional/demo.log index d3ba9f7..0fac925 100644 --- a/models/arcee-ai/Arcee-Spark/t3000/functional/demo.log +++ b/models/arcee-ai/Arcee-Spark/t3000/functional/demo.log @@ -1,75 +1,85 @@ -env TT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 /proj_sw/user_dev/moconnor/tt-metal/python_env/bin/python demo.py models/arcee-ai/Arcee-Spark/t3000/functional/model.py -2026-02-09 05:20:33.071 | info | UMD | Established firmware bundle version: 18.12.1 (topology_discovery.cpp:368) -2026-02-09 05:20:33.102 | info | Device | Opening user mode device driver (tt_cluster.cpp:223) -2026-02-09 05:20:33.112 | info | UMD | Established firmware bundle version: 18.12.1 (topology_discovery.cpp:368) -2026-02-09 05:20:33.186 | info | UMD | Established firmware bundle version: 18.12.1 (topology_discovery.cpp:368) -2026-02-09 05:20:33.248 | info | UMD | Harvesting masks for chip 3 tensix: 0x202 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:20:33.306 | info | UMD | Harvesting masks for chip 2 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:20:33.316 | info | UMD | Harvesting masks for chip 1 tensix: 0x220 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:20:33.326 | info | UMD | Harvesting masks for chip 0 tensix: 0x240 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:20:33.337 | info | UMD | Harvesting masks for chip 7 tensix: 0x280 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:20:33.350 | info | UMD | Harvesting masks for chip 6 tensix: 0x210 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:20:33.363 | info | UMD | Harvesting masks for chip 5 tensix: 0x210 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:20:33.376 | info | UMD | Harvesting masks for chip 4 tensix: 0x5 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:20:33.390 | info | UMD | Opening local chip ids/PCIe ids: {0, 1, 2, 3}/[0, 3, 1, 2] and remote chip ids {4, 5, 6, 7} (cluster.cpp:186) -2026-02-09 05:20:33.390 | info | UMD | IOMMU: disabled (cluster.cpp:161) -2026-02-09 05:20:33.390 | info | UMD | KMD version: 2.4.1 (cluster.cpp:164) -2026-02-09 05:20:33.400 | info | UMD | Starting devices in cluster (cluster.cpp:965) -2026-02-09 05:20:33.401 | info | UMD | Mapped hugepage 0x240000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:20:33.401 | info | UMD | Mapped hugepage 0x200000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:20:33.402 | info | UMD | Mapped hugepage 0x41c0000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:20:33.403 | info | UMD | Mapped hugepage 0x4180000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:20:33.404 | info | UMD | Mapped hugepage 0x300000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:20:33.405 | info | UMD | Mapped hugepage 0x2c0000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:20:33.406 | info | UMD | Mapped hugepage 0x42c0000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:20:33.406 | info | UMD | Mapped hugepage 0x4280000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:20:33.474 | info | Distributed | Using auto discovery to generate mesh graph. (metal_context.cpp:827) -2026-02-09 05:20:33.474 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_context.cpp:804) -2026-02-09 05:20:33.474 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) -2026-02-09 05:20:33.474 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) -2026-02-09 05:20:33.486 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:20:33.489 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:20:33.490 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:20:33.490 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:20:33.491 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:20:33.491 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:20:33.492 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:20:33.492 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:20:33.840 | warning | Metal | Got num_routing_planes: 1, which is less than current value: 255, ignoring the override (metal_context.cpp:719) -2026-02-09 05:20:33.840 | info | Metal | Dispatch on FabricConfig::FABRIC_2D with 1 Command Queues - (device_manager.cpp:328) -2026-02-09 05:20:33.862 | info | Metal | Initializing Fabric (device_manager.cpp:404) -2026-02-09 05:20:34.083 | info | Metal | Fabric initialized on Device 0 (device.cpp:386) -2026-02-09 05:20:34.084 | info | Metal | Fabric initialized on Device 1 (device.cpp:386) -2026-02-09 05:20:34.092 | info | Metal | Fabric initialized on Device 2 (device.cpp:386) -2026-02-09 05:20:34.093 | info | Metal | Fabric initialized on Device 3 (device.cpp:386) -2026-02-09 05:20:34.096 | info | Metal | Fabric initialized on Device 4 (device.cpp:386) -2026-02-09 05:20:34.102 | info | Metal | Fabric initialized on Device 5 (device.cpp:386) -2026-02-09 05:20:34.105 | info | Metal | Fabric initialized on Device 6 (device.cpp:386) -2026-02-09 05:20:34.111 | info | Metal | Fabric initialized on Device 7 (device.cpp:386) -2026-02-09 05:20:34.111 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_2D (device_manager.cpp:409) -2026-02-09 05:20:34.228 | info | Metal | Command Queue initialized on Device 7 (device_manager.cpp:500) -2026-02-09 05:20:34.228 | info | Metal | Command Queue initialized on Device 6 (device_manager.cpp:500) -2026-02-09 05:20:34.229 | info | Metal | Command Queue initialized on Device 4 (device_manager.cpp:500) -2026-02-09 05:20:34.230 | info | Metal | Command Queue initialized on Device 5 (device_manager.cpp:500) +$ python demo.py models/arcee-ai/Arcee-Spark/t3000/functional/model.py +2026-02-17 10:29:00.000 | DEBUG | ttnn::77 - Initial ttnn.CONFIG: +Config{cache_path=/home/moconnor/.cache/ttnn,model_cache_path=/home/moconnor/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt} Loading tokenizer: arcee-ai/Arcee-Spark Opening TT device... +2026-02-17 10:29:00.593 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) +2026-02-17 10:29:00.624 | info | Device | Opening user mode device driver (tt_cluster.cpp:223) +2026-02-17 10:29:00.632 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) +2026-02-17 10:29:00.706 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) +2026-02-17 10:29:00.769 | info | UMD | Harvesting masks for chip 3 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 10:29:00.824 | info | UMD | Harvesting masks for chip 2 tensix: 0x280 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 10:29:00.834 | info | UMD | Harvesting masks for chip 1 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 10:29:00.845 | info | UMD | Harvesting masks for chip 0 tensix: 0x220 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 10:29:00.855 | info | UMD | Harvesting masks for chip 7 tensix: 0x210 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 10:29:00.869 | info | UMD | Harvesting masks for chip 6 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 10:29:00.882 | info | UMD | Harvesting masks for chip 5 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 10:29:00.895 | info | UMD | Harvesting masks for chip 4 tensix: 0x240 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 10:29:00.909 | info | UMD | Opening local chip ids/PCIe ids: {0, 1, 2, 3}/[0, 1, 3, 2] and remote chip ids {4, 5, 6, 7} (cluster.cpp:186) +2026-02-17 10:29:00.909 | info | UMD | IOMMU: disabled (cluster.cpp:161) +2026-02-17 10:29:00.909 | info | UMD | KMD version: 2.4.1 (cluster.cpp:164) +2026-02-17 10:29:00.917 | info | UMD | Starting devices in cluster (cluster.cpp:965) +2026-02-17 10:29:00.918 | info | UMD | Mapped hugepage 0x280000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 10:29:00.918 | info | UMD | Mapped hugepage 0x240000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 10:29:00.919 | info | UMD | Mapped hugepage 0x300000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 10:29:00.920 | info | UMD | Mapped hugepage 0x2c0000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 10:29:00.921 | info | UMD | Mapped hugepage 0x41c0000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 10:29:00.921 | info | UMD | Mapped hugepage 0x4180000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 10:29:00.922 | info | UMD | Mapped hugepage 0x4240000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 10:29:00.923 | info | UMD | Mapped hugepage 0x4200000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 10:29:00.978 | info | Distributed | Using auto discovery to generate mesh graph. (metal_context.cpp:827) +2026-02-17 10:29:00.978 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_context.cpp:804) +2026-02-17 10:29:00.979 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) +2026-02-17 10:29:00.979 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) +Requested mesh (1, 8) exceeds discovered system mesh dimensions (2, 4), but is allowed because device count fits (8 <= 8). +2026-02-17 10:29:00.986 | DEBUG | ttnn.device:__init__:150 - Using default dispatch core type for this system: DispatchCoreType.ETH +2026-02-17 10:29:00.986 | DEBUG | ttnn.device:__init__:152 - Using default dispatch core axis for this system: DispatchCoreAxis.ROW +2026-02-17 10:29:00.993 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 10:29:00.995 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 10:29:00.996 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 10:29:00.996 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 10:29:00.997 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 10:29:00.997 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 10:29:00.998 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 10:29:00.998 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 10:29:01.330 | warning | Metal | Got num_routing_planes: 1, which is less than current value: 255, ignoring the override (metal_context.cpp:719) +2026-02-17 10:29:01.330 | info | Metal | Dispatch on FabricConfig::FABRIC_1D with 1 Command Queues + (device_manager.cpp:328) +2026-02-17 10:29:01.352 | info | Metal | Initializing Fabric (device_manager.cpp:404) +2026-02-17 10:29:01.562 | info | Metal | Fabric initialized on Device 0 (device.cpp:386) +2026-02-17 10:29:01.563 | info | Metal | Fabric initialized on Device 1 (device.cpp:386) +2026-02-17 10:29:01.563 | info | Metal | Fabric initialized on Device 2 (device.cpp:386) +2026-02-17 10:29:01.564 | info | Metal | Fabric initialized on Device 3 (device.cpp:386) +2026-02-17 10:29:01.567 | info | Metal | Fabric initialized on Device 4 (device.cpp:386) +2026-02-17 10:29:01.569 | info | Metal | Fabric initialized on Device 5 (device.cpp:386) +2026-02-17 10:29:01.575 | info | Metal | Fabric initialized on Device 6 (device.cpp:386) +2026-02-17 10:29:01.581 | info | Metal | Fabric initialized on Device 7 (device.cpp:386) +2026-02-17 10:29:01.581 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_1D (device_manager.cpp:409) +2026-02-17 10:29:01.701 | info | Metal | Command Queue initialized on Device 4 (device_manager.cpp:500) +2026-02-17 10:29:01.703 | info | Metal | Command Queue initialized on Device 7 (device_manager.cpp:500) +2026-02-17 10:29:01.703 | info | Metal | Command Queue initialized on Device 6 (device_manager.cpp:500) +2026-02-17 10:29:01.704 | info | Metal | Command Queue initialized on Device 5 (device_manager.cpp:500) Loading HuggingFace reference model on CPU: arcee-ai/Arcee-Spark + Loading checkpoint shards: 0%| | 0/4 [00:00:77 - Initial ttnn.CONFIG: +Config{cache_path=/home/moconnor/.cache/ttnn,model_cache_path=/home/moconnor/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt} Loading model module: /localdev/moconnor/ttnn_models/models/arcee-ai/Arcee-Spark/t3000/functional/model.py Loading HuggingFace tokenizer... Loading HuggingFace reference model on CPU... -2026-02-09 05:24:48.118 | info | UMD | Established firmware bundle version: 18.12.1 (topology_discovery.cpp:368) -2026-02-09 05:24:48.150 | info | Device | Opening user mode device driver (tt_cluster.cpp:223) -2026-02-09 05:24:48.160 | info | UMD | Established firmware bundle version: 18.12.1 (topology_discovery.cpp:368) -2026-02-09 05:24:48.236 | info | UMD | Established firmware bundle version: 18.12.1 (topology_discovery.cpp:368) -2026-02-09 05:24:48.297 | info | UMD | Harvesting masks for chip 3 tensix: 0x202 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:24:48.353 | info | UMD | Harvesting masks for chip 2 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:24:48.363 | info | UMD | Harvesting masks for chip 1 tensix: 0x220 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:24:48.374 | info | UMD | Harvesting masks for chip 0 tensix: 0x240 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:24:48.383 | info | UMD | Harvesting masks for chip 7 tensix: 0x280 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:24:48.397 | info | UMD | Harvesting masks for chip 6 tensix: 0x210 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:24:48.411 | info | UMD | Harvesting masks for chip 5 tensix: 0x210 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:24:48.424 | info | UMD | Harvesting masks for chip 4 tensix: 0x5 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-09 05:24:48.437 | info | UMD | Opening local chip ids/PCIe ids: {0, 1, 2, 3}/[0, 3, 1, 2] and remote chip ids {4, 5, 6, 7} (cluster.cpp:186) -2026-02-09 05:24:48.437 | info | UMD | IOMMU: disabled (cluster.cpp:161) -2026-02-09 05:24:48.437 | info | UMD | KMD version: 2.4.1 (cluster.cpp:164) -2026-02-09 05:24:48.445 | info | UMD | Starting devices in cluster (cluster.cpp:965) -2026-02-09 05:24:48.446 | info | UMD | Mapped hugepage 0x240000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:24:48.447 | info | UMD | Mapped hugepage 0x200000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:24:48.448 | info | UMD | Mapped hugepage 0x41c0000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:24:48.448 | info | UMD | Mapped hugepage 0x4180000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:24:48.449 | info | UMD | Mapped hugepage 0x300000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:24:48.450 | info | UMD | Mapped hugepage 0x2c0000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:24:48.451 | info | UMD | Mapped hugepage 0x42c0000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:24:48.452 | info | UMD | Mapped hugepage 0x4280000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-09 05:24:48.504 | info | Distributed | Using auto discovery to generate mesh graph. (metal_context.cpp:827) -2026-02-09 05:24:48.504 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_context.cpp:804) -2026-02-09 05:24:48.504 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) -2026-02-09 05:24:48.504 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) -2026-02-09 05:24:48.516 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:24:48.519 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:24:48.519 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:24:48.520 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:24:48.520 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:24:48.521 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:24:48.521 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:24:48.522 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-09 05:24:48.852 | warning | Metal | Got num_routing_planes: 1, which is less than current value: 255, ignoring the override (metal_context.cpp:719) -2026-02-09 05:24:48.852 | info | Metal | Dispatch on FabricConfig::FABRIC_2D with 1 Command Queues - (device_manager.cpp:328) -2026-02-09 05:24:48.863 | info | Metal | Initializing Fabric (device_manager.cpp:404) -2026-02-09 05:24:49.081 | info | Metal | Fabric initialized on Device 0 (device.cpp:386) -2026-02-09 05:24:49.082 | info | Metal | Fabric initialized on Device 1 (device.cpp:386) -2026-02-09 05:24:49.082 | info | Metal | Fabric initialized on Device 2 (device.cpp:386) -2026-02-09 05:24:49.083 | info | Metal | Fabric initialized on Device 3 (device.cpp:386) -2026-02-09 05:24:49.086 | info | Metal | Fabric initialized on Device 4 (device.cpp:386) -2026-02-09 05:24:49.092 | info | Metal | Fabric initialized on Device 5 (device.cpp:386) -2026-02-09 05:24:49.095 | info | Metal | Fabric initialized on Device 6 (device.cpp:386) -2026-02-09 05:24:49.101 | info | Metal | Fabric initialized on Device 7 (device.cpp:386) -2026-02-09 05:24:49.101 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_2D (device_manager.cpp:409) -2026-02-09 05:24:49.210 | info | Metal | Command Queue initialized on Device 4 (device_manager.cpp:500) -2026-02-09 05:24:49.210 | info | Metal | Command Queue initialized on Device 6 (device_manager.cpp:500) -2026-02-09 05:24:49.210 | info | Metal | Command Queue initialized on Device 5 (device_manager.cpp:500) -2026-02-09 05:24:49.212 | info | Metal | Command Queue initialized on Device 7 (device_manager.cpp:500) + Loading checkpoint shards: 0%| | 0/4 [00:00 Date: Tue, 17 Feb 2026 12:58:08 +0000 Subject: [PATCH 2/3] Update Arcee-Spark functional logs and MODELS --- MODELS.md | 2 +- .../Arcee-Spark/t3000/functional/demo.log | 129 +++++++++--------- .../Arcee-Spark/t3000/functional/eval.log | 124 +++++++++-------- 3 files changed, 126 insertions(+), 129 deletions(-) diff --git a/MODELS.md b/MODELS.md index 3330937..e61cd5d 100644 --- a/MODELS.md +++ b/MODELS.md @@ -10,7 +10,7 @@ Note: Keep the table columns padded with spaces and right-justify numeric cells | ----------------------------------- | :------: | :--------: | ----: | ----: | -----: | ----: | ------: | | arcee-ai/Arcee-Spark | n150 | functional | 92% | 100% | 99ms | 13.9 | 29952 | | arcee-ai/Arcee-Spark | n300 | functional | 91% | 100% | 338ms | 5.0 | 32768 | -| arcee-ai/Arcee-Spark | t3000 | functional | 90% | 100% | 194ms | 7.3 | 32768 | +| arcee-ai/Arcee-Spark | t3000 | functional | 91% | 100% | 192ms | 7.3 | 32768 | | arcee-ai/AFM-4.5B | n150 | functional | 98% | 100% | 72ms | 17.2 | 65536 | | arcee-ai/AFM-4.5B | n300 | functional | 97% | 100% | 283ms | 5.6 | 65536 | | arcee-ai/AFM-4.5B | t3000 | functional | 98% | 100% | 181ms | 7.1 | 65536 | diff --git a/models/arcee-ai/Arcee-Spark/t3000/functional/demo.log b/models/arcee-ai/Arcee-Spark/t3000/functional/demo.log index 0fac925..3a9b0b5 100644 --- a/models/arcee-ai/Arcee-Spark/t3000/functional/demo.log +++ b/models/arcee-ai/Arcee-Spark/t3000/functional/demo.log @@ -1,66 +1,66 @@ -$ python demo.py models/arcee-ai/Arcee-Spark/t3000/functional/model.py -2026-02-17 10:29:00.000 | DEBUG | ttnn::77 - Initial ttnn.CONFIG: +$ TT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python demo.py models/arcee-ai/Arcee-Spark/t3000/functional/model.py +2026-02-17 12:48:24.421 | DEBUG | ttnn::77 - Initial ttnn.CONFIG: Config{cache_path=/home/moconnor/.cache/ttnn,model_cache_path=/home/moconnor/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt} Loading tokenizer: arcee-ai/Arcee-Spark Opening TT device... -2026-02-17 10:29:00.593 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) -2026-02-17 10:29:00.624 | info | Device | Opening user mode device driver (tt_cluster.cpp:223) -2026-02-17 10:29:00.632 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) -2026-02-17 10:29:00.706 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) -2026-02-17 10:29:00.769 | info | UMD | Harvesting masks for chip 3 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 10:29:00.824 | info | UMD | Harvesting masks for chip 2 tensix: 0x280 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 10:29:00.834 | info | UMD | Harvesting masks for chip 1 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 10:29:00.845 | info | UMD | Harvesting masks for chip 0 tensix: 0x220 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 10:29:00.855 | info | UMD | Harvesting masks for chip 7 tensix: 0x210 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 10:29:00.869 | info | UMD | Harvesting masks for chip 6 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 10:29:00.882 | info | UMD | Harvesting masks for chip 5 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 10:29:00.895 | info | UMD | Harvesting masks for chip 4 tensix: 0x240 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 10:29:00.909 | info | UMD | Opening local chip ids/PCIe ids: {0, 1, 2, 3}/[0, 1, 3, 2] and remote chip ids {4, 5, 6, 7} (cluster.cpp:186) -2026-02-17 10:29:00.909 | info | UMD | IOMMU: disabled (cluster.cpp:161) -2026-02-17 10:29:00.909 | info | UMD | KMD version: 2.4.1 (cluster.cpp:164) -2026-02-17 10:29:00.917 | info | UMD | Starting devices in cluster (cluster.cpp:965) -2026-02-17 10:29:00.918 | info | UMD | Mapped hugepage 0x280000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 10:29:00.918 | info | UMD | Mapped hugepage 0x240000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 10:29:00.919 | info | UMD | Mapped hugepage 0x300000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 10:29:00.920 | info | UMD | Mapped hugepage 0x2c0000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 10:29:00.921 | info | UMD | Mapped hugepage 0x41c0000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 10:29:00.921 | info | UMD | Mapped hugepage 0x4180000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 10:29:00.922 | info | UMD | Mapped hugepage 0x4240000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 10:29:00.923 | info | UMD | Mapped hugepage 0x4200000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 10:29:00.978 | info | Distributed | Using auto discovery to generate mesh graph. (metal_context.cpp:827) -2026-02-17 10:29:00.978 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_context.cpp:804) -2026-02-17 10:29:00.979 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) -2026-02-17 10:29:00.979 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) +2026-02-17 12:48:25.057 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) +2026-02-17 12:48:25.087 | info | Device | Opening user mode device driver (tt_cluster.cpp:223) +2026-02-17 12:48:25.097 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) +2026-02-17 12:48:25.173 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) +2026-02-17 12:48:25.234 | info | UMD | Harvesting masks for chip 3 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 12:48:25.292 | info | UMD | Harvesting masks for chip 2 tensix: 0x280 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 12:48:25.302 | info | UMD | Harvesting masks for chip 1 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 12:48:25.312 | info | UMD | Harvesting masks for chip 0 tensix: 0x220 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 12:48:25.322 | info | UMD | Harvesting masks for chip 7 tensix: 0x210 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 12:48:25.339 | info | UMD | Harvesting masks for chip 6 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 12:48:25.352 | info | UMD | Harvesting masks for chip 5 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 12:48:25.365 | info | UMD | Harvesting masks for chip 4 tensix: 0x240 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 12:48:25.381 | info | UMD | Opening local chip ids/PCIe ids: {0, 1, 2, 3}/[0, 1, 3, 2] and remote chip ids {4, 5, 6, 7} (cluster.cpp:186) +2026-02-17 12:48:25.381 | info | UMD | IOMMU: disabled (cluster.cpp:161) +2026-02-17 12:48:25.381 | info | UMD | KMD version: 2.4.1 (cluster.cpp:164) +2026-02-17 12:48:25.389 | info | UMD | Starting devices in cluster (cluster.cpp:965) +2026-02-17 12:48:25.390 | info | UMD | Mapped hugepage 0x280000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 12:48:25.391 | info | UMD | Mapped hugepage 0x240000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 12:48:25.392 | info | UMD | Mapped hugepage 0x300000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 12:48:25.393 | info | UMD | Mapped hugepage 0x2c0000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 12:48:25.393 | info | UMD | Mapped hugepage 0x41c0000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 12:48:25.394 | info | UMD | Mapped hugepage 0x4180000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 12:48:25.395 | info | UMD | Mapped hugepage 0x4240000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 12:48:25.396 | info | UMD | Mapped hugepage 0x4200000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 12:48:25.454 | info | Distributed | Using auto discovery to generate mesh graph. (metal_context.cpp:827) +2026-02-17 12:48:25.454 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_context.cpp:804) +2026-02-17 12:48:25.455 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) +2026-02-17 12:48:25.455 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) Requested mesh (1, 8) exceeds discovered system mesh dimensions (2, 4), but is allowed because device count fits (8 <= 8). -2026-02-17 10:29:00.986 | DEBUG | ttnn.device:__init__:150 - Using default dispatch core type for this system: DispatchCoreType.ETH -2026-02-17 10:29:00.986 | DEBUG | ttnn.device:__init__:152 - Using default dispatch core axis for this system: DispatchCoreAxis.ROW -2026-02-17 10:29:00.993 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 10:29:00.995 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 10:29:00.996 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 10:29:00.996 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 10:29:00.997 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 10:29:00.997 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 10:29:00.998 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 10:29:00.998 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 10:29:01.330 | warning | Metal | Got num_routing_planes: 1, which is less than current value: 255, ignoring the override (metal_context.cpp:719) -2026-02-17 10:29:01.330 | info | Metal | Dispatch on FabricConfig::FABRIC_1D with 1 Command Queues +2026-02-17 12:48:25.460 | DEBUG | ttnn.device:__init__:150 - Using default dispatch core type for this system: DispatchCoreType.ETH +2026-02-17 12:48:25.460 | DEBUG | ttnn.device:__init__:152 - Using default dispatch core axis for this system: DispatchCoreAxis.ROW +2026-02-17 12:48:25.467 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 12:48:25.470 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 12:48:25.470 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 12:48:25.471 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 12:48:25.471 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 12:48:25.472 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 12:48:25.472 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 12:48:25.473 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 12:48:25.827 | warning | Metal | Got num_routing_planes: 1, which is less than current value: 255, ignoring the override (metal_context.cpp:719) +2026-02-17 12:48:25.827 | info | Metal | Dispatch on FabricConfig::FABRIC_1D with 1 Command Queues (device_manager.cpp:328) -2026-02-17 10:29:01.352 | info | Metal | Initializing Fabric (device_manager.cpp:404) -2026-02-17 10:29:01.562 | info | Metal | Fabric initialized on Device 0 (device.cpp:386) -2026-02-17 10:29:01.563 | info | Metal | Fabric initialized on Device 1 (device.cpp:386) -2026-02-17 10:29:01.563 | info | Metal | Fabric initialized on Device 2 (device.cpp:386) -2026-02-17 10:29:01.564 | info | Metal | Fabric initialized on Device 3 (device.cpp:386) -2026-02-17 10:29:01.567 | info | Metal | Fabric initialized on Device 4 (device.cpp:386) -2026-02-17 10:29:01.569 | info | Metal | Fabric initialized on Device 5 (device.cpp:386) -2026-02-17 10:29:01.575 | info | Metal | Fabric initialized on Device 6 (device.cpp:386) -2026-02-17 10:29:01.581 | info | Metal | Fabric initialized on Device 7 (device.cpp:386) -2026-02-17 10:29:01.581 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_1D (device_manager.cpp:409) -2026-02-17 10:29:01.701 | info | Metal | Command Queue initialized on Device 4 (device_manager.cpp:500) -2026-02-17 10:29:01.703 | info | Metal | Command Queue initialized on Device 7 (device_manager.cpp:500) -2026-02-17 10:29:01.703 | info | Metal | Command Queue initialized on Device 6 (device_manager.cpp:500) -2026-02-17 10:29:01.704 | info | Metal | Command Queue initialized on Device 5 (device_manager.cpp:500) +2026-02-17 12:48:25.848 | info | Metal | Initializing Fabric (device_manager.cpp:404) +2026-02-17 12:48:26.019 | info | Metal | Fabric initialized on Device 0 (device.cpp:386) +2026-02-17 12:48:26.085 | info | Metal | Fabric initialized on Device 1 (device.cpp:386) +2026-02-17 12:48:26.086 | info | Metal | Fabric initialized on Device 2 (device.cpp:386) +2026-02-17 12:48:26.086 | info | Metal | Fabric initialized on Device 3 (device.cpp:386) +2026-02-17 12:48:26.089 | info | Metal | Fabric initialized on Device 4 (device.cpp:386) +2026-02-17 12:48:26.092 | info | Metal | Fabric initialized on Device 5 (device.cpp:386) +2026-02-17 12:48:26.098 | info | Metal | Fabric initialized on Device 6 (device.cpp:386) +2026-02-17 12:48:26.103 | info | Metal | Fabric initialized on Device 7 (device.cpp:386) +2026-02-17 12:48:26.103 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_1D (device_manager.cpp:409) +2026-02-17 12:48:26.280 | info | Metal | Command Queue initialized on Device 7 (device_manager.cpp:500) +2026-02-17 12:48:26.282 | info | Metal | Command Queue initialized on Device 4 (device_manager.cpp:500) +2026-02-17 12:48:26.282 | info | Metal | Command Queue initialized on Device 5 (device_manager.cpp:500) +2026-02-17 12:48:26.283 | info | Metal | Command Queue initialized on Device 6 (device_manager.cpp:500) Loading HuggingFace reference model on CPU: arcee-ai/Arcee-Spark - Loading checkpoint shards: 0%| | 0/4 [00:00:77 - Initial ttnn.CONFIG: +$ TT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python eval.py models/arcee-ai/Arcee-Spark/t3000/functional/model.py --model arcee-ai/Arcee-Spark --prompt_file prompts/bringup_eval_long.txt --max_new_tokens 100 --max_seq_len 32768 +2026-02-17 12:51:30.104 | DEBUG | ttnn::77 - Initial ttnn.CONFIG: Config{cache_path=/home/moconnor/.cache/ttnn,model_cache_path=/home/moconnor/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt} Loading model module: /localdev/moconnor/ttnn_models/models/arcee-ai/Arcee-Spark/t3000/functional/model.py Loading HuggingFace tokenizer... Loading HuggingFace reference model on CPU... - Loading checkpoint shards: 0%| | 0/4 [00:00 Date: Tue, 17 Feb 2026 14:31:22 +0000 Subject: [PATCH 3/3] Update Arcee-Spark functional docs and logs --- MODELS.md | 2 +- .../Arcee-Spark/t3000/functional/demo.log | 131 +++++++++--------- .../Arcee-Spark/t3000/functional/eval.log | 124 ++++++++--------- 3 files changed, 128 insertions(+), 129 deletions(-) diff --git a/MODELS.md b/MODELS.md index e61cd5d..cbd7773 100644 --- a/MODELS.md +++ b/MODELS.md @@ -10,7 +10,7 @@ Note: Keep the table columns padded with spaces and right-justify numeric cells | ----------------------------------- | :------: | :--------: | ----: | ----: | -----: | ----: | ------: | | arcee-ai/Arcee-Spark | n150 | functional | 92% | 100% | 99ms | 13.9 | 29952 | | arcee-ai/Arcee-Spark | n300 | functional | 91% | 100% | 338ms | 5.0 | 32768 | -| arcee-ai/Arcee-Spark | t3000 | functional | 91% | 100% | 192ms | 7.3 | 32768 | +| arcee-ai/Arcee-Spark | t3000 | functional | 89% | 100% | 200ms | 6.9 | 32768 | | arcee-ai/AFM-4.5B | n150 | functional | 98% | 100% | 72ms | 17.2 | 65536 | | arcee-ai/AFM-4.5B | n300 | functional | 97% | 100% | 283ms | 5.6 | 65536 | | arcee-ai/AFM-4.5B | t3000 | functional | 98% | 100% | 181ms | 7.1 | 65536 | diff --git a/models/arcee-ai/Arcee-Spark/t3000/functional/demo.log b/models/arcee-ai/Arcee-Spark/t3000/functional/demo.log index 3a9b0b5..839835c 100644 --- a/models/arcee-ai/Arcee-Spark/t3000/functional/demo.log +++ b/models/arcee-ai/Arcee-Spark/t3000/functional/demo.log @@ -1,66 +1,66 @@ -$ TT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python demo.py models/arcee-ai/Arcee-Spark/t3000/functional/model.py -2026-02-17 12:48:24.421 | DEBUG | ttnn::77 - Initial ttnn.CONFIG: +python demo.py models/arcee-ai/Arcee-Spark/t3000/functional/model.py +2026-02-17 14:13:16.978 | DEBUG | ttnn::77 - Initial ttnn.CONFIG: Config{cache_path=/home/moconnor/.cache/ttnn,model_cache_path=/home/moconnor/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt} Loading tokenizer: arcee-ai/Arcee-Spark Opening TT device... -2026-02-17 12:48:25.057 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) -2026-02-17 12:48:25.087 | info | Device | Opening user mode device driver (tt_cluster.cpp:223) -2026-02-17 12:48:25.097 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) -2026-02-17 12:48:25.173 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) -2026-02-17 12:48:25.234 | info | UMD | Harvesting masks for chip 3 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 12:48:25.292 | info | UMD | Harvesting masks for chip 2 tensix: 0x280 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 12:48:25.302 | info | UMD | Harvesting masks for chip 1 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 12:48:25.312 | info | UMD | Harvesting masks for chip 0 tensix: 0x220 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 12:48:25.322 | info | UMD | Harvesting masks for chip 7 tensix: 0x210 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 12:48:25.339 | info | UMD | Harvesting masks for chip 6 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 12:48:25.352 | info | UMD | Harvesting masks for chip 5 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 12:48:25.365 | info | UMD | Harvesting masks for chip 4 tensix: 0x240 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) -2026-02-17 12:48:25.381 | info | UMD | Opening local chip ids/PCIe ids: {0, 1, 2, 3}/[0, 1, 3, 2] and remote chip ids {4, 5, 6, 7} (cluster.cpp:186) -2026-02-17 12:48:25.381 | info | UMD | IOMMU: disabled (cluster.cpp:161) -2026-02-17 12:48:25.381 | info | UMD | KMD version: 2.4.1 (cluster.cpp:164) -2026-02-17 12:48:25.389 | info | UMD | Starting devices in cluster (cluster.cpp:965) -2026-02-17 12:48:25.390 | info | UMD | Mapped hugepage 0x280000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 12:48:25.391 | info | UMD | Mapped hugepage 0x240000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 12:48:25.392 | info | UMD | Mapped hugepage 0x300000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 12:48:25.393 | info | UMD | Mapped hugepage 0x2c0000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 12:48:25.393 | info | UMD | Mapped hugepage 0x41c0000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 12:48:25.394 | info | UMD | Mapped hugepage 0x4180000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 12:48:25.395 | info | UMD | Mapped hugepage 0x4240000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 12:48:25.396 | info | UMD | Mapped hugepage 0x4200000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) -2026-02-17 12:48:25.454 | info | Distributed | Using auto discovery to generate mesh graph. (metal_context.cpp:827) -2026-02-17 12:48:25.454 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_context.cpp:804) -2026-02-17 12:48:25.455 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) -2026-02-17 12:48:25.455 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) -Requested mesh (1, 8) exceeds discovered system mesh dimensions (2, 4), but is allowed because device count fits (8 <= 8). -2026-02-17 12:48:25.460 | DEBUG | ttnn.device:__init__:150 - Using default dispatch core type for this system: DispatchCoreType.ETH -2026-02-17 12:48:25.460 | DEBUG | ttnn.device:__init__:152 - Using default dispatch core axis for this system: DispatchCoreAxis.ROW -2026-02-17 12:48:25.467 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 12:48:25.470 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 12:48:25.470 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 12:48:25.471 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 12:48:25.471 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 12:48:25.472 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 12:48:25.472 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 12:48:25.473 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) -2026-02-17 12:48:25.827 | warning | Metal | Got num_routing_planes: 1, which is less than current value: 255, ignoring the override (metal_context.cpp:719) -2026-02-17 12:48:25.827 | info | Metal | Dispatch on FabricConfig::FABRIC_1D with 1 Command Queues +2026-02-17 14:13:17.673 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) +2026-02-17 14:13:17.703 | info | Device | Opening user mode device driver (tt_cluster.cpp:223) +2026-02-17 14:13:17.712 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) +2026-02-17 14:13:17.785 | info | UMD | Established firmware bundle version: 19.4.0 (topology_discovery.cpp:368) +2026-02-17 14:13:17.846 | info | UMD | Harvesting masks for chip 3 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 14:13:17.904 | info | UMD | Harvesting masks for chip 2 tensix: 0x280 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 14:13:17.915 | info | UMD | Harvesting masks for chip 1 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 14:13:17.925 | info | UMD | Harvesting masks for chip 0 tensix: 0x220 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 14:13:17.935 | info | UMD | Harvesting masks for chip 7 tensix: 0x210 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 14:13:17.949 | info | UMD | Harvesting masks for chip 6 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 14:13:17.965 | info | UMD | Harvesting masks for chip 5 tensix: 0x201 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 14:13:17.979 | info | UMD | Harvesting masks for chip 4 tensix: 0x240 dram: 0x0 eth: 0x0 pcie: 0x0 l2cpu: 0x0 (cluster.cpp:339) +2026-02-17 14:13:17.993 | info | UMD | Opening local chip ids/PCIe ids: {0, 1, 2, 3}/[0, 1, 3, 2] and remote chip ids {4, 5, 6, 7} (cluster.cpp:186) +2026-02-17 14:13:17.993 | info | UMD | IOMMU: disabled (cluster.cpp:161) +2026-02-17 14:13:17.993 | info | UMD | KMD version: 2.4.1 (cluster.cpp:164) +2026-02-17 14:13:18.001 | info | UMD | Starting devices in cluster (cluster.cpp:965) +2026-02-17 14:13:18.002 | info | UMD | Mapped hugepage 0x280000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 14:13:18.003 | info | UMD | Mapped hugepage 0x240000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 14:13:18.004 | info | UMD | Mapped hugepage 0x300000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 14:13:18.004 | info | UMD | Mapped hugepage 0x2c0000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 14:13:18.005 | info | UMD | Mapped hugepage 0x41c0000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 14:13:18.006 | info | UMD | Mapped hugepage 0x4180000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 14:13:18.007 | info | UMD | Mapped hugepage 0x4240000000 to NOC address 0x800000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 14:13:18.008 | info | UMD | Mapped hugepage 0x4200000000 to NOC address 0x840000000 (silicon_sysmem_manager.cpp:207) +2026-02-17 14:13:18.059 | info | Distributed | Using auto discovery to generate mesh graph. (metal_context.cpp:827) +2026-02-17 14:13:18.059 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_context.cpp:804) +2026-02-17 14:13:18.059 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) +2026-02-17 14:13:18.059 | info | Fabric | TopologyMapper mapping start (mesh=0): n_log=8, n_phys=8, log_deg_hist={2:4, 3:4}, phys_deg_hist={2:4, 3:4} (topology_mapper_utils.cpp:171) +Requested mesh (1, 8) exceeds discovered system mesh (2, 4); falling back to discovered mesh because TTNN_ALLOW_SYSTEM_MESH_FALLBACK is enabled. +2026-02-17 14:13:18.071 | DEBUG | ttnn.device:__init__:150 - Using default dispatch core type for this system: DispatchCoreType.ETH +2026-02-17 14:13:18.071 | DEBUG | ttnn.device:__init__:152 - Using default dispatch core axis for this system: DispatchCoreAxis.ROW +2026-02-17 14:13:18.077 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 14:13:18.081 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 14:13:18.081 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 14:13:18.082 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 14:13:18.083 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 14:13:18.084 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 14:13:18.085 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 14:13:18.085 | info | BuildKernels | Skipping deleting built cache (build.cpp:110) +2026-02-17 14:13:18.433 | warning | Metal | Got num_routing_planes: 1, which is less than current value: 255, ignoring the override (metal_context.cpp:719) +2026-02-17 14:13:18.434 | info | Metal | Dispatch on FabricConfig::FABRIC_2D with 1 Command Queues (device_manager.cpp:328) -2026-02-17 12:48:25.848 | info | Metal | Initializing Fabric (device_manager.cpp:404) -2026-02-17 12:48:26.019 | info | Metal | Fabric initialized on Device 0 (device.cpp:386) -2026-02-17 12:48:26.085 | info | Metal | Fabric initialized on Device 1 (device.cpp:386) -2026-02-17 12:48:26.086 | info | Metal | Fabric initialized on Device 2 (device.cpp:386) -2026-02-17 12:48:26.086 | info | Metal | Fabric initialized on Device 3 (device.cpp:386) -2026-02-17 12:48:26.089 | info | Metal | Fabric initialized on Device 4 (device.cpp:386) -2026-02-17 12:48:26.092 | info | Metal | Fabric initialized on Device 5 (device.cpp:386) -2026-02-17 12:48:26.098 | info | Metal | Fabric initialized on Device 6 (device.cpp:386) -2026-02-17 12:48:26.103 | info | Metal | Fabric initialized on Device 7 (device.cpp:386) -2026-02-17 12:48:26.103 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_1D (device_manager.cpp:409) -2026-02-17 12:48:26.280 | info | Metal | Command Queue initialized on Device 7 (device_manager.cpp:500) -2026-02-17 12:48:26.282 | info | Metal | Command Queue initialized on Device 4 (device_manager.cpp:500) -2026-02-17 12:48:26.282 | info | Metal | Command Queue initialized on Device 5 (device_manager.cpp:500) -2026-02-17 12:48:26.283 | info | Metal | Command Queue initialized on Device 6 (device_manager.cpp:500) +2026-02-17 14:13:18.455 | info | Metal | Initializing Fabric (device_manager.cpp:404) +2026-02-17 14:13:18.670 | info | Metal | Fabric initialized on Device 0 (device.cpp:386) +2026-02-17 14:13:18.751 | info | Metal | Fabric initialized on Device 1 (device.cpp:386) +2026-02-17 14:13:18.752 | info | Metal | Fabric initialized on Device 2 (device.cpp:386) +2026-02-17 14:13:18.752 | info | Metal | Fabric initialized on Device 3 (device.cpp:386) +2026-02-17 14:13:18.755 | info | Metal | Fabric initialized on Device 4 (device.cpp:386) +2026-02-17 14:13:18.758 | info | Metal | Fabric initialized on Device 5 (device.cpp:386) +2026-02-17 14:13:18.764 | info | Metal | Fabric initialized on Device 6 (device.cpp:386) +2026-02-17 14:13:18.771 | info | Metal | Fabric initialized on Device 7 (device.cpp:386) +2026-02-17 14:13:18.771 | info | Metal | Fabric Initialized with config FabricConfig::FABRIC_2D (device_manager.cpp:409) +2026-02-17 14:13:18.897 | info | Metal | Command Queue initialized on Device 4 (device_manager.cpp:500) +2026-02-17 14:13:18.897 | info | Metal | Command Queue initialized on Device 7 (device_manager.cpp:500) +2026-02-17 14:13:18.897 | info | Metal | Command Queue initialized on Device 5 (device_manager.cpp:500) +2026-02-17 14:13:18.899 | info | Metal | Command Queue initialized on Device 6 (device_manager.cpp:500) Loading HuggingFace reference model on CPU: arcee-ai/Arcee-Spark - Loading checkpoint shards: 0%| | 0/4 [00:00:77 - Initial ttnn.CONFIG: +python eval.py models/arcee-ai/Arcee-Spark/t3000/functional/model.py --model arcee-ai/Arcee-Spark --prompt_file prompts/bringup_eval_long.txt --max_new_tokens 100 --max_seq_len 32768 +2026-02-17 14:16:56.109 | DEBUG | ttnn::77 - Initial ttnn.CONFIG: Config{cache_path=/home/moconnor/.cache/ttnn,model_cache_path=/home/moconnor/.cache/ttnn/models,tmp_dir=/tmp/ttnn,enable_model_cache=false,enable_fast_runtime_mode=true,throw_exception_on_fallback=false,enable_logging=false,enable_graph_report=false,enable_detailed_buffer_report=false,enable_detailed_tensor_report=false,enable_comparison_mode=false,comparison_mode_should_raise_exception=false,comparison_mode_pcc=0.9999,root_report_path=generated/ttnn/reports,report_name=std::nullopt,std::nullopt} Loading model module: /localdev/moconnor/ttnn_models/models/arcee-ai/Arcee-Spark/t3000/functional/model.py Loading HuggingFace tokenizer... Loading HuggingFace reference model on CPU... - Loading checkpoint shards: 0%| | 0/4 [00:00