hdf5_msb

[svn-r24757] Change assertion to condition check/error return.  Should fix periodic
assertion failure in fheap test (in test that is expected to fail cleanly).

Fix type in H5detect.c

Tested: ostrich, platypus (h5committest - jam and koala having issues)