"Implementing the TensorFlow Deep Learning Framework on Qualcomm’s Low-power DSP," a Presentation from Google
3 likes1,499 views
TensorFlow has been implemented on Qualcomm's low-power DSP to run neural networks for machine learning. Models run 8 times faster and use 1.4 watts of power on the Snapdragon 820 processor's Hexagon DSP compared to 5 watts on a CPU. Qualcomm tested gemmlowp on the HVX and achieved over 5 times the speed and much lower power usage than a CPU. End to end performance on InceptionV1 was around 90 ms using the HVX versus 700 ms on a CPU.
"Implementing the TensorFlow Deep Learning Framework on Qualcomm’s Low-power DSP," a Presentation from Google
- 1. Copyright © 2017 Google 1 Implementing the TensorFlow Deep Learning Framework on Qualcomm’s Low-power DSP Pete Warden May 2017
- 2. Copyright © 2017 Google 2 • Google’s open source library for machine intelligence • tensorflow.org launched in Nov 2015 • Used by many production ML projects 2
- 3. Copyright © 2017 Google 3 TensorFlow and HVX
- 4. Copyright © 2017 Google 4 • Models run 8X faster, and use 1.4 watts versus ~5 watts on CPU TensorFlow supports Qualcomm’s Hexagon DSP Qualcomm Snapdragon 820 Processor featuring the Hexagon DSP DragonBoard 820c
- 5. Copyright © 2017 Google 5 • Started with my Embedded Vision Alliance talk last year • “Eight bits are enough” • Became clear from conversations with Qualcomm that there were possibilities with their existing hardware in the Snapdragon 820 How did this happen?
- 6. Copyright © 2017 Google 6 • Qualcomm implemented a quick sanity test using gemmlowp, our open source math library • That demonstrated 100 GOPs/second on realistic workloads using the HVX • More than 5x speed of CPU • Power usage expected to be much lower Next Steps
- 7. Copyright © 2017 Google 7 • Gemmlowp project has m, n, k values for InceptionV1 matrix multiplies Benchmark Details https://github.com/google/gemmlowp/blob/master/test/benchmark.cc#L283
- 8. Copyright © 2017 Google 8 • Gemmlowp results indicated around 200 GOPs/s (versus 25 GOPs/s on CPU) • End to end turned out to be around 90 ms, versus 700 ms on CPU • Was a good predictor of performance Results 8
- 9. Copyright © 2017 Google 9 TensorFlow code is at https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/hvx Qualcomm code is https://source.codeaurora.org/quic/hexagon_nn/nnlib
- 10. Copyright © 2017 Google 10 • Works by assembling a batch of ops on the CPU • Then sends them off to HVX via FastRPC • HVX runs it within its own code loop • Signals the AP when it's done
- 11. Copyright © 2017 Google 11 • TensorFlow handles splitting up the graph between HVX and CPU • Same mechanism is available for other accelerators too
- 12. Copyright © 2017 Google 12
- 13. Copyright © 2017 Google 13
- 14. Copyright © 2017 Google 14
- 15. Copyright © 2017 Google 15 Embedded TensorFlow
- 16. Copyright © 2017 Google 16 We work closely with chip builders
- 17. Copyright © 2017 Google 17 • Examples • ARM’s Compute Library • Movidius’s mvTensor tool • CEVA’s conversion tools • Intel’s contributions to https://github.com/google/gemmlowp • Qualcomm’s HVX collaboration We work closely with chip builders
- 18. Copyright © 2017 Google 18 Why?
- 19. Copyright © 2017 Google 19 • Mobile App Developers (including Snapchat) • Device builders • Home • Drones • Industrial • Medical • Automotive Lots of demand
- 20. Copyright © 2017 Google 20 • Full support for eight bit • Full stack: researchers, data centers, mobile apps, embedded devices • Main framework at Google • Shipping for vision on many apps, including PhotoScan and Snapchat What’s TensorFlow particularly good at?
- 21. Copyright © 2017 Google 21 • Support for eight-bit training • On-device training (already being used by Google Keyboard) • Better export pipeline (Graph Transform Tool) • Raspberry Pi • Jetson TX1 experimental support • Other chips? • Many more examples Embedded TensorFlow Roadmap
- 22. Copyright © 2017 Google 22 • ARM and Intel added code to https://github.com/google/gemmlowp • Worked with many others to support TensorFlow file format for conversion pipelines • We’re always open to conversations about our requirements and porting Collaborations with hardware vendors
- 23. Copyright © 2017 Google 23 • TensorFlow hands-on training class from the Embedded Vision Alliance, July 13 in Santa Clara • We’re always looking for chips, tools, systems companies to collaborate with • Please get in touch! • [email protected] Future