Debugging & Profiling of AWS Lambda: ServerlessConf - IOpipe
1 like738 views
The document discusses profiling and debugging AWS Lambda by Erica Windisch, highlighting the limitations of traditional instrumentation tools in serverless environments. It addresses challenges such as restricted access to debugging features and the impracticality of using kernel-level instrumentation. The document also references resources and tools for local debugging and profiling within the Lambda framework.
Debugging & Profiling of AWS Lambda: ServerlessConf - IOpipe
- 1. ServerlessConf NYC’17 Profiling & debugging AWS Lambda Erica Windisch CTO, Founder IOpipe, Inc.
- 2. Erica Windisch CTO & Founder IOpipe, Inc. Former maintainer of Docker & OpenStack Building cloud, ops, and infrastructure tooling for over 15 years.
- 3. (Thanks & credit to Katie Bayes!)
- 5. Kernel Hardware & Virtualization Application Detail Accuracy ❖ Step-debugging ❖ CPU Profiling ❖ Heap snapshots & allocation profiles
- 6. Kernel Hardware & Virtualization Application Detail Accuracy ❖ Step-debugging ❖ CPU Profiling ❖ Heap snapshots & allocation profiles
- 7. Virtualization - Valgrind https://www.cprogramming.com/debugging/valgrind.html
- 8. Kernel Hardware & Virtualization Application Detail Accuracy ❖ Step-debugging ❖ CPU Profiling ❖ Heap snapshots & allocation profiles
- 11. — Brendan Gregg
- 12. Almost everything you need to know about kernel-level performance tracing & beyond http://www.brendangregg.com/
- 13. Kernel Hardware & Virtualization Application Detail Accuracy ❖ Step-debugging ❖ CPU Profiling ❖ Heap snapshots & allocation profiles
- 14. ptrace (i.e. gdb, strace, et al) https://sysdig.com/blog/sysdig-vs-dtrace-vs-strace-a-technical-discussion/
- 15. There’s an excessive number of tools (and they don’t work on Lambda)
- 17. Things that don’t work: Linux eBPF! $ iosnoop Tracing block I/O. Ctrl-C to end. /usr/bin/iosnoop: line 172: cd: /sys/kernel/debug/tracing: Permission denied ERROR: accessing tracing. Root user? Kernel has FTRACE? debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)
- 18. Things that don’t work: Linux eBPF! λ$ ls -l /sys ⨯ Error: function response: Command failed: ls -l /sys ls: cannot access /sys: No such file or directory
- 19. Things that don’t work: Linux perf
- 20. Things that don’t work: Linux perf λ$ cat /proc/sys/kernel/perf_event_paranoid 2
- 21. Kernel-level instrumentation is out of reach
- 22. HW+Virt instrumentation is (mostly) out of reach.
- 23. Application instrumentation is hard(er) on Lambda
- 24. Lambda Limitations ✤ Event-loop is out of our direct control ✤ Process command-line flags inaccessible ✤ Ptrace disabled! ✤ Linux kernel perf features not available on Lambda. ✤ Local dev+test is an evolving story ✤ Many languages, requiring different implementations ✤ Debugger breakpoints in prod aren’t generally viable
- 27. (Thanks Paul Maddox, Sanath Kumar Ramesh, & AWS!)
- 28. Debugging Lambda in the cloud github.com/trek10inc/ aws-lambda-debugger
- 29. Profiling Lambda in the cloud github.com/iopipe/ iopipe-plugin-profiler
- 38. iopipe.com