[Feature Request] Calculating FLOPs for computational graph operations

[MG2033]

Feature Request: Please consider adding a floating point operations calculator for computational graph operations.

We'd like to use it for the deep learning models.

cc @ezyang @gchanan @zou3519 @bdhirsh @heitorschueroff @VitalyFedyunin @ngimel

[soumith]

@cpuhrsch this is something you were interested to have as well right? also @shubho

[cpuhrsch]

@soumith, this will be important, but it's low priority for now as we're still experimenting outside of pytorch. We're setting something up similar to this, but not as part of any framework for now. Maybe we can port that code to pytorch later on, but I can't promise anything.

[shubho]

Would be nice to track which networks do well on which processors - and also roll up datacenter level metrics. @cpuhrsch are you building a tool outside of PyTorch that will take in a model specification and emit these numbers for every op?

[cpuhrsch]

@shubho for now we're doing this one op at a time. For example, we're building it out far enough to optimize a function such as torch.sum, but within C++ only. Presumably, as we work our way through this, a tool that will create these (and many more) metrics per op for a model could emerge. For now that'd be over-engineering our tooling.

[ahirner]

I've come across this script to count ops and params for basic layers. It's reasonably advanced (most 2D convs and pooling). Maybe it compliments your porting efforts @cpuhrsch.

[warmspringwinds]

If someone still needs this, we wrote a small script to do that:

https://github.com/warmspringwinds/pytorch-segmentation-detection/blob/master/pytorch_segmentation_detection/utils/flops_benchmark.py

Example of usage:

https://github.com/warmspringwinds/pytorch-segmentation-detection/blob/d5df5e066fe9c6078d38b26527d93436bf869b1c/pytorch_segmentation_detection/recipes/pascal_voc/segmentation/flops_counter.ipynb

[sovrasov]

Also I've improved the above mentioned scripts to support grouped convolutions and several other layers:
https://github.com/sovrasov/flops-counter.pytorch

[laoreja]

We can compute FLOPs of conv layers by hand or by existing repos easily.
But is there a way to compute FLOPs for non-traditional operations like topk, sparse sum and indexing, etc.?
Thanks!

[cpuhrsch]

@zheng-xq - Is this something that could be of interest to you?

[ttumiel]

Came across this library which also counts number of parameters and FLOPs used by a model given an input.

[matyasfodor]

I came across this repo: https://github.com/ceykmc/pytorch_model_summary AFAIK it's not distributed on pypi, but it gives you a pretty detailed summary.