Check whether GBQ Job is finished

[cgrin]

xref #8728

jobComplete can be False in query_reply. Simply checking for the existence of the field isn't enough, as a False value will cause a KeyError when checking for query_reply['totalRows']

[shoyer]

usually I would prefer not query_reply['jobComplete'] to query_reply['jobComplete'] is False

you could also consolidate these with while(not query_reply.get('jobComplete', False))

[cgrin]

@shoyer I like the get method. Added a comment to make it blatantly obvious why the loop is there.

[jreback]

@jacobschaer pls have a look

[jacobschaer]

Isn't this similar to #8728 ? We are waiting for Google's response to http://stackoverflow.com/questions/27810116/google-bigquery-request-too-large

[cgrin]

Looks like this is the same issue as #8728. The code is exactly the same. I can see in that issue that the existing code came right from Google's example, however the API reference for getQueryResults regarding the jobComplete key states:

Whether the query has completed or not. If rows or totalRows are present, this will always be true. If this is false, totalRows will not be available.

So clearly we need to check the value of jobComplete before moving on to grabbing totalRows. The thread you've referenced at SO is regarding an issue with inserts, why is that holding up this unrelated change?

[jacobschaer]

It's being held up because we wanted to get the intergration suite to pass before giving this the go ahead. The other PR I referenced had a couple of changes from @shaun-schaefer that was supposed to go in as well and needed testing. At this point though, I am wondering if we should just proceed with the understanding that the test suite has a known issue.

[cgrin]

Gotcha. I'd argue that having read_gbq() return data reliably is more valuable than passing all integration tests, especially since the BQ API is still evolving.

[jreback]

@jacobschaer how's the testing coming?

[jreback]

closing in favor of #8728 (its the older issue).