Master failing

[max-sixty]

@tswast, do you recognize whether this an issue of the creds on the pandas-gbq travis being expired?

https://travis-ci.org/pydata/pandas-gbq/jobs/428221719

Running with my creds works great: https://travis-ci.org/max-sixty/pandas-gbq

[tswast]

google.api_core.exceptions.NotFound: 404 GET https://www.googleapis.com/bigquery/v2/projects/pandas-travis/datasets?pageToken=pandas_gbq_499911: Not found: Token pandas_gbq_499911

This isn't a credentials issue. It's a bug in the BigQuery API when you have multiple dataset list operations running in parallel with dataset delete operations. Basically, this is what's going on:

1. We start listing datasets.
2. BigQuery gives us a pagination token that corresponds to a dataset ID.
3. Some other process deletes that dataset while we are handling the current page.
4. We use our pagination token to ask for the next page.
5. BigQuery gets confused because the pagination no longer corresponds to a dataset, because it has been deleted.

I've filed an issue for it internally, but since it can usually be worked around by retrying the job, it hasn't been prioritized.

I suspect there are probably a ton of datasets in the pandas-gbq test project that didn't get properly garbage collected, which is probably exasperating the issue on this project. I'll see about running a Travis build today to try and clean those up.

[tswast]

That other thing we might be able to do to minimize this is to download the full list of datasets before we start deleting them in our clean-up step.

Or re-use datasets with a longer-session pytest fixture so that we don't create so many.

[max-sixty]

Ah - that makes sense - it's not happening on my test suite because the project doesn't have enough datasets to require paging.
That does sound like a small bug, so if it's an annual deleting of datasets then that seems like a reasonable fix until that gets fixed...