Applying build process to data analysis

I'd like to use the build process to process collected data. This is a multi staged process which includes bringing in and parsing the data, reducing the data, creating index to pertinent portions of the collection, analyze and create figure outputs. Toward that end, i'd like to dynamically specify input and output file collections via input arguments. I don't care to process the whole 'tree' of data everytime I build. I'd like to be able to specify a branch of the data with my arguments. Is there any way to specify input file collection details as an argument in the interest of exercising incremental builds?

Answer 1

[Dan]

Talk about not being able to see the trees for the forest ... I can simply add arguments to my buildfile function that I can use for setting up Task Inputs and outputs ...

My wrapper will specify details that can pass to the buildfile via arguments ...

Answer 2

[Dinesh]

Hi Dan,

MATLAB Build Tool supports configurable tasks that accept arguments, and task arguments participate in incremental build fingerprinting. A common approach is to pass a branch identifier (or file-collection specification) as a task argument and then construct the task's FileCollection inputs and outputs from that value. This allows you to process only the relevant subset of the data hierarchy while still benefiting from incremental execution. Keep in mind that Build Tool's up-to-date checks work best when the selected files are represented in the task's declared inputs and outputs, rather than being discovered solely within the task action.

Please refere to the following documentations for more information:

1. Create Tasks That Accept Arguments - MATLAB & Simulink
2. Improve Performance with Incremental Builds - MATLAB & Simulink

[Dan]

Thanks for your response, Dinesh. Specifically, I want my arguments to be used to affect .Inputs and .Outputs fields of the tasks themselves. I have poured through the referenced articles and I have seen how arguments can be passed into tasks but incremental building seems to be solely dependent upon the task fields, Inputs and Outputs. If you could provide a snippet of code that would show me how to dynamically affect those paths at the time the plan is built, I would be very appreciative. I am currently employing a wrapper function which I use to save at 'base' path to an environment variable which is then used to setup those parameters, Inputs and Outputs, to my liking but a wrapper function is clunky. It seems that there should be a way to pass, at buildtime or when creating the build file, arguments to affect those paths.