You can use a combination of the "exist", "ismember", and "superclasses" functions. This lets you see if a given string represents a class name and whether that class is a subclass of the MATLAB Unit Test superclass. The general syntax is as follows:
exist("classname", 'class') && ismember("superclassName", superclasses("classname"))
In this both "classname" and "superclassName" are strings. This will return logical 1 if "classname" is both a class and "superclassName" appears in the set of its superclasses. This can handle many levels of inheritance, since the "superclasses" function does not only include direct superclasses.