Matlab and Big Query dB limit
4 views (last 30 days)
Show older comments
This is part of my code in order to download data from Big Query Google cloud Platform using a SELECT function.. Everything works fine, however i can't download all selected values. I can't download more than 350 000 Floats. Is there a limit? Is it possible to remove this limit?
api = 'https://www.googleapis.com/bigquery/v2/';
access_token = <AccessToken>
options = weboptions( ...
'RequestMethod', 'post', ...
'MediaType', 'application/json',...
'Timeout', Inf);
query=['SELECT '...
'TIMESTAMP_MILLIS(timestamp) as datetime, '...
'VALUEKEY, ' ...
....
'deviceId '...
'WHERE '...
'(_PARTITIONTIME BETWEEN TIMESTAMP(''2023-06-28'') '...
'AND TIMESTAMP(''2023-06-25'')) '...
data = struct( ...
'query', query, ...
'useLegacySql', false);
r = webwrite([api '<PROJECT>' access_token], ...
data, ...
options);
0 Comments
Answers (1)
Manan Jain
on 10 Jul 2023
There is a limit on the response size of a BigQuery API request. For interactive queries, the response size limit is 10 MB compressed or 100 MB uncompressed. If your query result exceeds these limits, you'll need to use pagination techniques to retrieve the data in smaller chunks.
There is a quota limit on the number of bytes processed per day for each project in BigQuery. The specific quota limit depends on your project and billing plan. If you exceed this limit, you may encounter errors or additional charges.
It's worth noting that the limit you're experiencing might not be directly related to the number of selected values, but rather the total size of the result set. If the result set exceeds the response size limit, you'll need to modify your code to handle pagination and retrieve the data in multiple requests.
To overcome these limitations and efficiently retrieve a large result set, you can use techniques like pagination, streaming results, or exporting the data to a storage bucket and downloading it from there.
If you need to further investigate the specific limitations or quotas affecting your project, I recommend checking the official BigQuery documentation.
0 Comments
See Also
Categories
Find more on Web Services in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!