Matlab and Big Query dB limit

4 views (last 30 days)
Alessandro Bartolini
Alessandro Bartolini on 28 Jun 2023
Answered: Manan Jain on 10 Jul 2023
This is part of my code in order to download data from Big Query Google cloud Platform using a SELECT function.. Everything works fine, however i can't download all selected values. I can't download more than 350 000 Floats. Is there a limit? Is it possible to remove this limit?
api = 'https://www.googleapis.com/bigquery/v2/';
access_token = <AccessToken>
options = weboptions( ...
'RequestMethod', 'post', ...
'MediaType', 'application/json',...
'Timeout', Inf);
query=['SELECT '...
'TIMESTAMP_MILLIS(timestamp) as datetime, '...
'VALUEKEY, ' ...
....
'deviceId '...
'WHERE '...
'(_PARTITIONTIME BETWEEN TIMESTAMP(''2023-06-28'') '...
'AND TIMESTAMP(''2023-06-25'')) '...
data = struct( ...
'query', query, ...
'useLegacySql', false);
r = webwrite([api '<PROJECT>' access_token], ...
data, ...
options);

Answers (1)

Manan Jain
Manan Jain on 10 Jul 2023
There is a limit on the response size of a BigQuery API request. For interactive queries, the response size limit is 10 MB compressed or 100 MB uncompressed. If your query result exceeds these limits, you'll need to use pagination techniques to retrieve the data in smaller chunks.
There is a quota limit on the number of bytes processed per day for each project in BigQuery. The specific quota limit depends on your project and billing plan. If you exceed this limit, you may encounter errors or additional charges.
It's worth noting that the limit you're experiencing might not be directly related to the number of selected values, but rather the total size of the result set. If the result set exceeds the response size limit, you'll need to modify your code to handle pagination and retrieve the data in multiple requests.
To overcome these limitations and efficiently retrieve a large result set, you can use techniques like pagination, streaming results, or exporting the data to a storage bucket and downloading it from there.
If you need to further investigate the specific limitations or quotas affecting your project, I recommend checking the official BigQuery documentation.

Products


Release

R2022a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!