Comments on: Lots Of Questions On Google’s “Trillium” TPU v6, A Few Answers

By: Timothy Prickett Morgan

Timothy Prickett Morgan — Tue, 11 Jun 2024 19:36:10 +0000

In reply to Vengineer. Thanks. I will amend.

By: Slim Albert

Slim Albert — Tue, 11 Jun 2024 11:14:31 +0000

Interesting update of Google’s TPU devices (to Trillium, v6), with that 4.7x performance and 67% efficiency improvements over v5e. In terms of the “MLPerf Inference: Datacenter benchmark” (https://mlcommons.org/benchmarks/inference-datacenter/) It looks like this should give it 12 samples/sec (per accelerator) on gptj-99 offline inference (MLPerf v4.0) which compares nicely to Gaudi 2 at 10.5 samples/sec (per accelerator) on MLPerf v3.1. But then, Intel is now at Gaudi 3, with 2x to 4x the performance of Gaudi 2, and Nvidia’s H100-SXM improved from around 13 samples/sec on MLPerf v3.1 to about 30 samples/sec in v4.0, and so … TPUv7 might be the one to watch out for!

Nevertheless, the “custom optical ICI interconnect” (reconfigurable?) remains a high point of their tech in my view, nicely ahead of the curve in its deployment.

By: Vengineer

Vengineer — Tue, 11 Jun 2024 01:06:49 +0000

According to the paper below, there are two SparseCores in TPU v3.

https://arxiv.org/pdf/2304.01433

The article below also states that it is also installed in TPU v2.
https://www.semianalysis.com/p/google-ai-infrastructure-supremacy

By: Mickey Pearson

Mickey Pearson — Mon, 10 Jun 2024 20:50:08 +0000

(for obvious reasons) there is no mention of SW development costs associated with making this work. That hidden number could be the biggest one among all those that are mentioned.