
Googleã¯æè¿ããã£ãŒãã©ãŒãã³ã°ãå éããããã«ç¹å¥ã«èšèšãããããã»ããµã§ããTensor Processing Unit v2ïŒTPUv2ïŒãã¯ã©ãŠããµãŒãã¹ã®ãªã¹ãã«è¿œå ããŸããã ããã¯ãäžçåã®åºãå©çšå¯èœãªãã£ãŒãã©ãŒãã³ã°ã¢ã¯ã»ã©ã¬ãŒã¿ã®ç¬¬2äžä»£ã§ãããNvidia GPUã®ä»£æ¿åã§ãããšäž»åŒµããŠããŸãã æè¿ãç§ãã¡ã¯ç¬¬äžå°è±¡ã«ã€ããŠè©±ããŸããã å€ãã®äººãã Nvidia V100 GPUãšã®ãã詳现ãªæ¯èŒãæ±ããŸããã
ãã£ãŒãã©ãŒãã³ã°ã¢ã¯ã»ã©ã¬ãŒã¿ã客芳çãã€ææ矩ã«æ¯èŒããããšã¯ç°¡åãªäœæ¥ã§ã¯ãããŸããã ãããããã®ã«ããŽãªã®è£œåã®å°æ¥ã®éèŠæ§ãšè©³çŽ°ãªæ¯èŒã®æ¬ åŠã«ãããç¬ç«ãããã¹ããå®æœããå¿ èŠæ§ãæããŸããã ããã«ã¯ãæœåšçã«å¯Ÿç«ããåœäºè ã®æèŠãèæ ®ã«å ¥ããããšãå«ãŸããŸãã ãã®ãããGoogleãšNvidiaã®ãšã³ãžãã¢ã«é£çµ¡ãããã®èšäºã®ãã©ããã«ã€ããŠã³ã¡ã³ãããããã«æåŸ ããŸããã ãã€ã¢ã¹ããªãããšãä¿èšŒããããã«ãç¬ç«ããå°é家ãæåŸ ããŸããã ãã®ãããã§ãç§ãã¡ãç¥ãéãããããŸã§ã®TPUv2ãšV100ã®æãå®å šãªæ¯èŒãå€æããŸããã
å®éšã»ããã¢ãã
以äžã¯ã4ã€ã®TPUv2ïŒ1ã€ã®ã¯ã©ãŠãTPUã圢æïŒã4ã€ã®Nvidia V100ãšæ¯èŒããŠããŸãã äž¡æ¹ãšã64 GBã®åèšã¡ã¢ãªããããããåãéã®ãã¬ãŒãã³ã°ãµã³ãã«ã§åãã¢ãã«ããã¬ãŒãã³ã°ã§ããŸãã å®éšã§ã¯ãã¢ãã«ãåãæ¹æ³ã§ãã¬ãŒãã³ã°ããŸããCloudTPUã®4ã€ã®TPUv2ãš4ã€ã®V100ãåæ䞊ååæ£åŠç¿ã®ã¿ã¹ã¯ãå®è¡ããŸãã
ã¢ãã«ãšããŠãç»åãåé¡ããããã®äºå®äžã®æšæºããã³ãã³ãããŒã¯ã§ããImageNetã§ResNet-50ãéžæããŸãã ã åç §ResNet-50å®è£ ã¯å ¬éãããŠããŸãããCloud TPUãšè€æ°ã®GPUã®äž¡æ¹ã§åæã«ãã¬ãŒãã³ã°ããµããŒããããã®ã¯ãããŸããã
Nvidiaã¯ãè€æ°ã®V100çšã®Nvidia GPUã¯ã©ãŠãã§Dockerã€ã¡ãŒãžãšããŠå©çšå¯èœãªMXNetãŸãã¯TensorFlowå®è£ ã䜿çšããããšãæšå¥šããŸãã æ®å¿µãªããã倧éã®ãã¬ãŒãã³ã°ãµã³ãã«ã䜿çšããŠè€æ°ã®GPUã§äœæ¥ããå Žåãäž¡æ¹ã®å®è£ ãããã©ã«ãèšå®ã«ããŸãããäžèŽããªãããšãå€æããŸããã ç¹ã«ãåŠç¿çã®ã¹ã±ãžã¥ãŒã«ãå€æŽããå¿ èŠããããŸãã
代ããã«ãTensorFlow ãã³ãããŒã¯ãªããžããªããResNet-50å®è£ ãååŸããDockerã€ã¡ãŒãžïŒtensorflow / tensorflowïŒ1.7.0-gpuãCUDA 9.0ãCuDNN 7.1.2ïŒãšããŠèµ·åããŸããã Nvidiaãæšå¥šããTensorFlowå®è£ ãããå€§å¹ ã«é«éã§ãããMXNetããããããã«å£ã£ãŠããŸãïŒçŽ3ïŒ ã以äžãåç §ïŒã ããããããã¯ããŸãåæããŸãã ããã«ããã¬ãŒã ã¯ãŒã¯ã®åãããŒãžã§ã³ïŒTensorFlow 1.7.0ïŒã§ã®2ã€ã®å®è£ ãæ¯èŒãããšããè¿œå ã®å©ç¹ããããŸãã
Googleã§ã¯ãCloud TPUã®å ¬åŒTPUãªããžããªããTensorFlow 1.7.0ã§bfloat16ã䜿çšããããšãæšå¥šããŠããŸãã TPUãšGPUã®äž¡æ¹ã®å®è£ ã¯ã察å¿ããã¢ãŒããã¯ãã£ã§æ··å粟床ã®èšç®ã䜿çšããã»ãšãã©ã®ãã³ãœã«ã¯å粟床æ°ã§ä¿åãããŸãã
V100ãã¹ãã¯ãp3.8xlargeã€ã³ã¹ã¿ã³ã¹ïŒ16ã³ã¢Xeon E5-2686@2.30GHzãã¡ã¢ãª244 GBãUbuntu 16.04ïŒã§ã 4ã€ã®V100 GPU ïŒãããã16 GBã®ã¡ã¢ãªïŒã䜿çšããŠå®è¡ãããŸããã TPUãã¹ãã¯ãå°ããªã€ã³ã¹ã¿ã³ã¹n1-standard-4ïŒ2ã€ã®Xeon@2.3GHzã³ã¢ã15 GBã®ã¡ã¢ãªãDebian 9ïŒã§å®è¡ããã 4ã€ã®TPUv2 ïŒãããã16 GBã®ã¡ã¢ãªïŒããã®Cloud TPUïŒv2-8ïŒãå²ãåœãŠãããŸããã
2ã€ã®ç°ãªãæ¯èŒãè¡ããŸããã ãŸããåæããŒã¿ã®åž¯åå¹ ïŒ1ç§ãããã®ç»åæ°ïŒã®èŠ³ç¹ãããå¢åŒ·ãªãã§ãã€ãŸãå©çšå¯èœãªããŒã¿ããè¿œå ã®ãã¬ãŒãã³ã°ããŒã¿ãäœæããã«ããã©ãŒãã³ã¹ã調æ»ããŸããã ãã®æ¯èŒã¯åæã«äŸåãããI / Oã«ããã«ããã¯ã¯ãªããããŒã¿ã®å¢åŒ·ã¯çµæã«åœ±é¿ããŸããã 2çªç®ã®æ¯èŒã§ã¯ãImageNetã®2ã€ã®å®è£ ã®ç²ŸåºŠãšåææ§ã調ã¹ãŸããã
垯åå¹ ãã¹ã
åæããŒã¿ã® 1ç§ãããã®ç»åæ°ãã€ãŸããããŸããŸãªããããµã€ãºã§ã®ãªã³ã¶ãã©ã€ãã¬ãŒãã³ã°çšã®ããŒã¿ã®äœæã«ãã£ãŠã¹ã«ãŒãããã枬å®ããŸããã TPUã®å Žåãæšå¥šããããµã³ãã«ãµã€ãºã¯1024ã®ã¿ã§ãããèªè ããã®å€æ°ã®ãªã¯ãšã¹ãã«ãããä»ã®çµæãå ±åããŠããããšã«æ³šæããŠãã ããã

åæããŒã¿ããã³æ¡åŒµãªãã®ããŸããŸãªãµã³ãã«ãµã€ãºã§ã®ããã©ãŒãã³ã¹ïŒ1ç§ãããã®ã€ã¡ãŒãžïŒã ãµã³ãã«ãµã€ãºã¯ã°ããŒãã«ã§ããã€ãŸãã1024ã¯åã¹ãããã§ã®åGPU / TPUãããäžã®256ã®ãµã€ãºãæå³ããŸã
1024ã®ãã¬ãŒãã³ã°ãµã³ãã«ãµã€ãºã§ã¯ã垯åå¹ ã«å®è³ªçãªéãã¯ãããŸããïŒ TPUã¯ãããã«é²ãã§ãããããŒãžã³ã¯çŽ2ïŒ ã§ãã å°èŠæš¡ãªãã¬ãŒãã³ã°ãµã³ãã«ã§ã¯ãââäž¡æ¹ã®ãã©ãããã©ãŒã ã§åž¯åå¹ ãäœäžããGPUã®åäœããããã«åäžããŸãã ãã ããåè¿°ã®ããã«ããã®ãããªãã¬ãŒãã³ã°ãµã³ãã«ãµã€ãºã¯çŸåšTPUã«ã¯æšå¥šãããŠããŸããã
Nvidiaã®æšå¥šã«åŸã£ãŠã MXNet㧠GPUã䜿çšããå®éšãå®æœããŸãã ã Nvidia GPU Cloudã§å©çšå¯èœãªDockerã€ã¡ãŒãžïŒ mxnetïŒ18.03-py3 ïŒã§ResNet-50å®è£ ã䜿çšããŸããã ãã¬ãŒãã³ã°ãµã³ãã«ãµã€ãºã768ïŒ1024ã¯å€ãããïŒã®å ŽåãGPUã¯1ç§éã«çŽ3280ã®ç»åãåŠçããŸãã ããã¯ãTPUã®æè¯ã®çµæãããçŽ3ïŒ é«éã§ãã ãã ããåè¿°ã®ããã«ãMXNetå®è£ ã¯ãã®ãã¬ãŒãã³ã°ãµã³ãã«ãµã€ãºã§ã¯è€æ°ã®GPUã«ããŸãããŸãåæããªããããããã§ã¯TensorFlowå®è£ ã«çŠç¹ãåœãŠãŸãã
ã¯ã©ãŠãã®ã³ã¹ã
Cloud TPUïŒ4ã€ã®TPUv2ãããïŒã¯çŸåšãGoogle Cloudã§ã®ã¿å©çšå¯èœã§ãã ãã®ãããªèšç®ãå¿ èŠãªå Žåã«ã®ã¿ãVMã€ã³ã¹ã¿ã³ã¹ã«ãªã³ããã³ãã§æ¥ç¶ããŸãã V100ã«ã€ããŠã¯ãAWSã¯ã©ãŠããœãªã¥ãŒã·ã§ã³ã確èªããŸããïŒV100ã¯ãŸã Googleã¯ã©ãŠãã§å©çšã§ããŸããïŒã äžèšã®çµæã«åºã¥ããŠãåãã©ãããã©ãŒã ããã³ãããã€ããŒã®1ãã«ãããã®1ç§ãããã®ç»åæ°ãæ£èŠåã§ããŸãã
ããã©ãŒãã³ã¹ïŒ1ç§ããã1ãã«ãããã®ç»å
ã¯ã©ãŠãTPU | 4ÃV100 | 4ÃV100 | |
---|---|---|---|
ã¯ã©ãŠã | Googleã¯ã©ãŠã | AWS | AWSãªã¶ãŒããã€ã³ã¹ã¿ã³ã¹ |
æéãããã®äŸ¡æ Œ | 6.7ãã« | 12.2ãã« | 8.4ãã« |
1ç§ãããã®ç»å | 3186 | 3128 | 3128 |
ããã©ãŒãã³ã¹ïŒ1ç§ãããã®ãã«ãããã®ã€ã¡ãŒãžæ°ïŒ | 476 | 256 | 374 |
ãããã®äŸ¡æ Œã§ãCloud TPUã¯æ確ãªåè ã§ãã ãã ããããé·ãã¬ã³ã¿ã«ãŸãã¯æ©åšã®è³Œå ¥ãæ€èšããŠããå Žåãç¶æ³ã¯ç°ãªã£ãŠèŠããå ŽåããããŸãïŒãã ãããã®ãªãã·ã§ã³ã¯çŸåšCloud TPUã§ã¯äœ¿çšã§ããŸããïŒã äžèšã®è¡šã«ã¯ãAWSäžã®äºçŽæžã¿ã€ã³ã¹ã¿ã³ã¹p3.8xlargeã®12ãæåã®ã¬ã³ã¿ã«æéïŒåæããªãïŒãå«ãŸããŠããŸãã ããã«ããã1ãã«ã®çç£æ§ã374ã€ã¡ãŒãž/ç§ã«1ãã«å€§å¹ ã«åäžããŸãã
GPUã«ã¯ä»ã«ãèå³æ·±ããªãã·ã§ã³ããããŸãã ããšãã°ã Cirrascaleã§ã¯ãçŽ7500ãã«ïŒ1æéãããçŽ10.3ãã«ïŒã§4å°ã®V100ã䜿çšãããµãŒããŒã¬ã³ã¿ã«ãæ¯ææäŸããŠããŸãã ãã ãããã®æ©åšã¯AWSã®æ©åšïŒCPUã¿ã€ããã¡ã¢ãªãNVLinkãµããŒããªã©ïŒãšã¯ç°ãªããããçŽæ¥æ¯èŒããã«ã¯è¿œå ã®ãã¹ããå¿ èŠã§ãã
粟床ãšåæ
ããã©ãŒãã³ã¹ã¬ããŒãã«å ããŠãèšç®ãå®éã«ãæå³ã®ããããã®ã§ããããšãã€ãŸãå®è£ ãè¯ãçµæã«åæããããšã確èªãããã£ãã®ã§ãã 2ã€ã®ç°ãªãå®è£ ãæ¯èŒãããããå€å°ã®éžè±ãäºæ³ãããŸãã ãããã£ãŠããã®æ¯èŒã¯æ©åšã®é床ã ãã§ãªããå®è£ ã®å質ã®ææšã§ããããŸãã ããšãã°ãTPUã®å®è£ ã«ã¯éåžžã«ãªãœãŒã¹ãéäžçã«äœ¿çšããååŠçæé ãå«ãŸããå®éã«ã¯åž¯åå¹ ãç ç²ã«ãªããŸãã Googleã«ãããšãããã¯äºæ³ãããåäœã§ãã 以äžã§èŠãããã«ãããã¯æ£åœåãããŸãã
ImageNetããŒã¿ã»ããã®ã¢ãã«ããã¬ãŒãã³ã°ããŸãããã¿ã¹ã¯ã¯ãç»åãããã㪠ã ããªã㌠ã ãã¶ãªã©ã®1000ã®ã«ããŽãªã®ããããã«åé¡ããããšã§ãã ããŒã¿ã»ããã¯ããã¬ãŒãã³ã°çšã®130äžç»åïŒã142 GBïŒãšæ€èšŒçšã®50,000ç»åïŒã7 GBïŒã§æ§æãããŠããŸãã
åŠç¿ã¯ãµã³ãã«ãµã€ãº1024ã§90ãè¶ ããæ代ã«è¡ããããã®åŸãçµæãã³ã³ãããŒã«ããŒã¿ãšæ¯èŒãããŸãã TPUå®è£ ã¯1ç§éã«çŽ2796åã®ç»åãåŠçããGPUå®è£ ã¯1ç§éã«çŽ2839åã®ç»åãåŠçããŸãã ããã¯ãå¢åŒ·ããªãã«ããåæããŒã¿ã䜿çšããŠTPUãšGPUã®æ£å³é床ãæ¯èŒãã以åã®åž¯åå¹ ã®çµæãšã¯ç°ãªããŸãã

90ãšããã¯åŸã®2ã€ã®å®è£ ã®äžäœ1ã®ç²ŸåºŠïŒã€ãŸããåç»åã«ã€ããŠãæãä¿¡é Œæ§ã®é«ãäºæž¬ã®ã¿ãèæ ®ãããŸãïŒ
äžèšã®ããã«ãTPUã®å®è£ ã®90ãšããã¯åŸã®ããã1ã®ç²ŸåºŠã¯0.7 pp ããè¯ãã ããã¯åãã«è¶³ããªãããã«æãããããããŸãããããã®éåžžã«é«ãã¬ãã«ã§æ¹åãéæããããšã¯éåžžã«å°é£ã§ãã ã¢ããªã±ãŒã·ã§ã³ã«ãã£ãŠã¯ããã®ãããªå°ããªæ¹åãçµæã«å€§ãã圱é¿ããå ŽåããããŸãã
ã¢ãã«ã®ãã¬ãŒãã³ã°äžã«ãããŸããŸãªæ代ã®ããã1ã®ç²ŸåºŠãèŠãŠã¿ãŸãããã

2ã€ã®å®è£ ã®ãã¹ãã»ããã§äžäœ1äœã®ç²ŸåºŠ
äžèšã®ã°ã©ãã®åçãªå€åã¯ãåŠç¿é床ã®å€åãšäžèŽããŠããŸãã åæåŸåã¯ãTPUã®å®è£ ã®æ¹ãåªããŠããŸãã ããã§ã¯ã86ã®æ代ã®åŸãæçµçãªç²ŸåºŠã¯76.4ïŒ ã«éããŸãã GPUã®å®è£ ã¯é ããŠããã84ãšããã¯åŸã«75.7ïŒ ã®æçµç²ŸåºŠã«éããŸãããTPUã§ãã®ãããªç²ŸåºŠãéæããã«ã¯64ãšããã¯ã ããå¿ èŠã§ãã TPUåæã®æ¹åã¯ãããŒã¿ã®ååŠçãšæ¡åŒµã®æ¹åã«ãããã®ãšæãããŸããããã®ä»®èª¬ã確èªããã«ã¯ããå€ãã®å®éšãå¿ èŠã§ãã
è²»çšå¯Ÿå¹æã®é«ãã¯ã©ãŠãäŸ¡æ Œèšå®ãœãªã¥ãŒã·ã§ã³
æçµçã«ãç¹å®ã®ç²ŸåºŠãéæããããã«å¿ èŠãªæéãšã³ã¹ããéèŠã«ãªããŸãã 75.7ïŒ ïŒGPUãå®è£ ããããšã§éæãããæé«ã®ç²ŸåºŠïŒã®ã¬ãã«ã§ãœãªã¥ãŒã·ã§ã³ãæ¡çšããå Žåãå¿ èŠãªæ代ãš1ç§ãããã®ç»åã®ãã¬ãŒãã³ã°é床ã«åºã¥ããŠããã®ç²ŸåºŠãéæããããã®ã³ã¹ããèšç®ã§ããŸãã ããã«ãããæ代ã®éã«ã¢ãã«ãè©äŸ¡ããæéãšãã¬ãŒãã³ã°ãéå§ããæéããªããªããŸãã

75.7ïŒ ã®ããã1ã®ç²ŸåºŠãéæããããã®äŸ¡æ Œã * 12ãæéäºçŽæžã¿
äžèšã®ããã«ãçŸåšã®Cloud TPUã®äŸ¡æ Œèšå®ããªã·ãŒã§ã¯ã55ãã«ã§9æé以å ã«ImageNetã§ã¢ãã«ããŒããã75.7ïŒ ã®ç²ŸåºŠã§ãã¬ãŒãã³ã°ã§ããŸãïŒ 76.4ïŒ ã®åæãŸã§åŠç¿ããã«ã¯73ãã«ããããŸãã V100ã¯åæ§ã«é«éã§åäœããŸãããäŸ¡æ Œãé«ããåæãé ããšããœãªã¥ãŒã·ã§ã³ã³ã¹ããå€§å¹ ã«é«ããªããŸãã
ç¹°ãè¿ããŸãããæ¯èŒã¯å®è£ ã®å質ãšã¯ã©ãŠãã®äŸ¡æ Œã«äŸåããããšã«æ³šæããŠãã ããã
ãšãã«ã®ãŒæ¶è²»ã®éããæ¯èŒããããšã¯èå³æ·±ãã§ãããã ããããçŸåšãTPUv2ã®ãšãã«ã®ãŒæ¶è²»ã«é¢ããå ¬éæ å ±ã¯ãããŸããã
ãããã«
ResNet-50ã®åºæ¬çãªããã©ãŒãã³ã¹ã«ã€ããŠã¯ããã¹ãã§ã¯4ã€ã®TPUv2ãããïŒ1ã€ã®Cloud TPUã¢ãžã¥ãŒã«ïŒãš4ã€ã®V100 GPUãåçã«é«éã§ãïŒå·®ã¯2ïŒ ä»¥å ïŒã ããããå°æ¥ã®ãœãããŠã§ã¢æé©åïŒããšãã°ãTensorFlowãŸãã¯CUDAïŒã«ãããããã©ãŒãã³ã¹ãåäžããæ¯çãå€åããŸãã
ãã ããå®éã«ã¯ãã»ãšãã©ã®å Žåãäž»ãªããšã¯ç¹å®ã®ã¿ã¹ã¯ã§ç¹å®ã®ç²ŸåºŠãéæããããã«å¿ èŠãªæéãšè²»çšã§ãã çŸåšã®ã¯ã©ãŠãTPUã®äŸ¡æ Œèšå®ãšåªããResNet-50ã®å®è£ ã¯ãImageNetã§æéãšã³ã¹ãã®å°è±¡çãªçµæããããããçŽ73ãã«ã§ã¢ãã«ã76.4ïŒ ã®ç²ŸåºŠã§ãã¬ãŒãã³ã°ã§ããŸãã
詳现ãªæ¯èŒã®ããã«ãä»ã®åéã®ã¢ãã«ãšç°ãªããããã¯ãŒã¯ã¢ãŒããã¯ãã£ã®ãã³ãããŒã¯ãå¿ èŠã§ãã ãŸããåããŒããŠã§ã¢ãã©ãããã©ãŒã ãå¹æçã«äœ¿çšããããã«å¿ èŠãªåŽåãç解ããããšãèå³æ·±ãã§ãã ããšãã°ã粟床ãæ··åšããèšç®ã§ã¯ãããã©ãŒãã³ã¹ãå€§å¹ ã«åäžããŸãããGPUãšTPUã§ã¯ç°ãªãæ¹æ³ã§å®è£ ãããŸãã