how to download data from website?

89 views (last 30 days)
Lilya
Lilya on 23 Mar 2019
Commented: imran about 6 hours ago
Dear all,
I am trying to download data from the following website
https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/
my problem is I can not get the files, meaning only the html been located to my computer. (below what I used to locat the link into my machine)
url='https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/'
filename='A2019.nc'
outfilename=websave(filename,url)
what I need is getting the files separtly and read them.
Thanks for the help.
  3 Comments
SASSA
SASSA on 15 Aug 2024
repository.kaust.edu.sa/bitstream/handle/10754/673855/PhD%20Dissertation-Final_Lina%20Eyouni.pdf?sequence=4
The website you linked provides access to oceanographic data, but directly downloading individual files through code might be tricky. Here's a breakdown of what you're encountering and alternative approaches:
Understanding the Download Challenge:
  • The website likely uses a different download method than websave expects. It might require authentication or interact with the server differently.
Alternative Approaches:
  1. Manual Download:
  1. Command-line tools (for advanced users):
  • Tools like wget can be used to download files from websites. However, it might require additional configuration for NASA's specific setup. Refer to the wget documentation and NASA's download instructions (if available) for this approach.
Additional Tips:
  • NASA's OceanColor website provides information on download methods: SASSA Status
  • Consider using Python libraries specifically designed for downloading scientific data like earthpy or pydap. These libraries might offer better compatibility with NASA's data access methods.
imran
imran about 5 hours ago
The news of the expiry of the SASSA Gold Card had created panic for the beneficiaries of the SASSA grant. As a result, the crowd of people in the SASSA offices was increasing day by day. Therefore, Post Bank and SASSA officially announced that SASSA cards will not expire in December. These cards will continue to be used to withdraw SASSA grants even in 2024.

Sign in to comment.

Accepted Answer

Akira Agata
Akira Agata on 26 Mar 2019
How about the following?
url = 'https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/';
str = webread(url);
links = regexp(str,'https://([\w-]+\.)+[\w-]+(/[\w- ./?%&=]*)?(\.nc)','match')';
data = cell(size(links));
for kk = 1:numel(links)
data{kk} = webread(links{kk});
end
By running this, all the .nc files are downloaded and stored in the cell array data.
  12 Comments
Belinda Finlay
Belinda Finlay on 12 May 2020
Thank you for your assistance Akira-san.
Soo Mee Kim
Soo Mee Kim on 11 Jun 2020
Hi, I have the same problem to to download nc files website.
When I tried the following code, both codes gave html file.
Please let me know how to download nc file from the website.
[my own]
fname = 'A2019152.L3m_DAY_PAR_par_4km.nc';
downloadURL = 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2019152.L3m_DAY_PAR_par_4km.nc';
options = weboptions('Username', '<username>', 'Password', '<password>');
websave(fname, downloadURL, options);
[From Akira Agata's answer]
downloadURL = 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2019152.L3m_DAY_PAR_par_4km.nc';
options = weboptions('Username', '<username>', 'Password', '<password>');
data = webread(downloadURL, options);

Sign in to comment.

More Answers (7)

Tracy
Tracy on 1 Aug 2023
It looks like you are using MATLAB's websave function to download the file from the URL. However, the websave function is primarily used to download a file and save it to a specified location. It seems like you are trying to download multiple files, and it won't work as you expect because you are providing a URL of a directory, not a direct link to a specific file.
To download multiple files from the website, you will need to loop through the links in the directory and download each file individually. Additionally, you can use Python with the requests library to achieve this task more easily. Below is a Python script that demonstrates how to download multiple files from the website using the requests library:
python
import requests
import os
url = 'https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/'
output_directory = './downloaded_files/'
# Create the output directory if it doesn't exist
os.makedirs(output_directory, exist_ok=True)
response = requests.get(url)
if response.status_code == 200:
# Parse the HTML content to find links to files
file_links = []
lines = response.text.split('\n')
for line in lines:
if '<a href="' in line:
start_index = line.find('<a href="') + len('<a href="')
end_index = line.find('">', start_index)
file_link = line[start_index:end_index]
if file_link.endswith('.nc'): # Only consider links that point to .nc files
file_links.append(file_link)
# Download each file and save it to the output directory
for file_link in file_links:
file_url = url + file_link
out_file_path = os.path.join(output_directory, file_link)
response = requests.get(file_url)
if response.status_code == 200:
with open(out_file_path, 'wb') as f:
f.write(response.content)
print(f"Downloaded: {file_link}")
else:
print(f"Failed to download: {file_link}")
else:
print("Failed to fetch the URL.")
print("All files downloaded successfully.")

Loria Smith
Loria Smith on 5 Sep 2023
There are several ways of manual web scraping.
  1. Code a web scraper with Python. It is possible to quickly build software with any general-purpose programming language like Java, JavaScript, PHP, C, C#, and so on. ...
  2. Use a data service. ...
  3. Use Excel for data extraction. ...
  4. Web scraping tools

Hitesh
Hitesh on 3 Oct 2023
import requests
# Define the URL of the file you want to download
url = 'https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/A2019.nc'
# Specify the local filename where you want to save the downloaded file
filename = 'A2019.nc'
# Send an HTTP GET request to the URL
response = requests.get(url)
# Check if the request was successful (status code 200)
if response.status_code == 200:
# Open a local file with write-binary ('wb') mode and save the content of the response
with open(filename, 'wb') as file:
file.write(response.content)
print(f"File '{filename}' downloaded successfully.")
else:
print(f"Failed to download file. Status code: {response.status_code}")
There are various methods for manually collecting data from websites:
  1. Coding a Web Scraper with Python: You can create a web scraper using Python, a versatile programming language. Python offers libraries like BeautifulSoup and Scrapy that make web scraping easier.
  2. Utilizing a Data Service: Another option is to use a data service or API provided by the website, if available. This method allows you to access structured data without the need for web scraping.
  3. Leveraging Excel for Data Extraction: Microsoft Excel can also be used for data extraction. You can import data from web pages into Excel and then manipulate and analyze it as needed.
  4. Web Scraping Tools: There are various web scraping tools and software applications designed specifically for data extraction from websites. These tools often provide a user-friendly interface for collecting data without extensive coding.

ABDUL
ABDUL on 7 Oct 2024
Great insights on downloading data from websites! It’s fascinating how tools like MATLAB can simplify the process. Just as many of us need to navigate the SASSA appeal process to secure our benefits, having efficient methods to gather data online can be equally crucial in various projects. Thanks for sharing these tips!

ABDUL
ABDUL on 7 Oct 2024
This post is super helpful for anyone looking to download data from websites! The techniques you’ve outlined are great. It reminds me of how important it is to stay organized, much like when we’re going through a having the right data can make all the difference. Thanks for sharing these valuable insights! For more

Andrietta
Andrietta on 7 Oct 2024
"If you’re trying to download data from a website, web scraping techniques can help! There are some great resources for using MATLAB’s web functions to automate data retrieval. By the way, I was also searching for 'sassa payment dates 2024,' and realized combining this with web scraping could make tracking updates easier."

Andrietta
Andrietta on 7 Oct 2024
Edited: Andrietta on 7 Oct 2024
"Using MATLAB’s webread or websave functions is a great way to download data from websites. It really helped me out when I was looking to pull info related to sassa appeal for an automation project. Anyone working on something similar might find these functions useful!"

Categories

Find more on Downloads in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!