Would result in some non-trivial speedups in places (like conversion of mixed-type DataFrame -> Panel)