How to Bulk Download Files in Soracom Harvest
Written by: Andrew Black
Published: August 10, 2023
One of the most useful functions of Soracom Harvest, our serverless data collection and dashboard tool, is Soracom Harvest Files, which allows users to collect, store and visualize data from IoT devices. Harvest mainly handles images and logs, though has no limitations on what types of files can be uploaded. Harvest files can also be used to store IoT device firmware, which can then be downloaded and applied from the device side.
Data stored within Soracom Harvest can provide valuable on-site information that can be analyzed for long-term solutions. Some examples of things that can be empowered by this kind of data include:
- Predictive Maintenance – Complaining baselines from atmospheric sensors and tracking performance metrics of deployed machines can allow users to proactively perform maintenance measures to avoid costly downtime or performance degradation
- Visual Analysis – By analyzing camera images with machine learning, it could be possible to establish a pass/fail judgment during a manufacturing process.
- Asset Management – Properly maintained data logs can be integral to the proper management of deployed devices and their surrounding environments, and easily accessed data will only make things more efficient.
Of course, analyzing data at the scale needed to enable these kinds of activities will require users to retrieve numerous files at once. This is where things become a bit complicated. Harvest Files makes it very easy to download files one by one, but downloading multiple files at once can be a bit tedious. Luckily, there is a solution!
How to Access Soracom Harvest Files in Bulk
The Soracom API allows users to easily and reliably process tasks that would otherwise take significantly more time and manual effort performed from a web browser. A sample script, found here, can retrieve a large number of files at once. Let’s take a closer look at the API in use.
You will want to use the following APIs:
FileEntry:listFiles
which will retrieve a list of saved files, and
FileEntry:getFile
to get a file.
The sample script uses the authentication ID and authentic key secret passed as arguments to obtain a list of files to retrieve, and then repeats the download of each file.
def main(args):
# ... (omitted for brevity)
token = auth(auth_key_id, auth_key)
file_names = list_file(
base_path, search, limit_num_to_list, limit_num_to_list_per_req, last_evaluated_key, token
)
get_and_save_images(file_names, base_path, save_path, limit_size_to_files, delete, token)This will allow you to download multiple files, but there are also some things to keep in mind.
--limit_size_to_files
When downloading files from Soracom Harvest Files, there is a fee assessed based on the size of the downloaded file. This means that if you download all the files at once, it can become costly.
This API option will allow you to limit the size of the data when downloading multiples instances or multiple months in order to keep the costs down.
--last_evaluated_key
FileEntry:listFiles
The files stored in Harvest Files are stored alphabetically, so if you are breaking up your data into multiple downloads, you’ll want to add this option to start any ensuing downloads from the point you left off with your previous batch.
View the ReadME (Japanese) to learn more API options, usage, and details
Conclusion
By using the Soracom API, you can easily accomplish tasks that would be time-consuming and far less cost-effective when performed manually. You can utilize additional scripts, you can remove any human error that may exist by running it manually each time.
Note: This blog is based on an existing piece from our Japanese team member Takuya
………………
Got any questions about an IoT deployment? Soracom offers a number of powerful solutions that have made us a visionary in the IoT space. Chat with us today to learn how a partnership with Soracom can get your project up and running easily and securely. Contact us today!
