Add COPYRIGHT notice.
Since all affected files are new, the following copyright notice was used:
Copyright by The HDF Group (THG).
All rights reserved.
This file is part of H4H5TOOLS. The full H4H5TOOLS copyright notice,
including terms governing use, modification, and redistribution, is
contained in the files COPYING and Copyright.html. COPYING can be
found at the root of the source code distribution tree; Copyright.html
can be found at the root level of an installed copy of the electroni...
When netCDF-4 output is enabled, create HDF5 attributes as SIMPLE space even though it has only one element.
By default, h4toh5 creates an HDF5 attribute as SCALAR space if there is only element. If that attribute is of numeric type, netCDF-4 rejects it because it always assumes SIMPLE space for numeric type. Previously, this is handled for attributes in Vdata or SDS. After this fix, the same handling is performed for Image or Vgroup.
verify - tolerate unusual HDF-EOS2 files that define XDim and YDim.
When XDim and YDim are explicitly defined in the HDF-EOS2 files, verifier has given a false alarm.
Enhancement: h4toh5 splits an attribute into several pieces if an attribute is too long. (override Revision #387)
As mentioned in Revision #387, a long attribute makes problems. Previously, only a global SDS attribute was handled. Now, any long attribute from Image, Vgroup, Vdata, SDS, and annotations is handled. Previously, an attribute becomes a dataset to avoid the limitation, but this changelist overrides this, and splits it into several attributes.
Currently, the naming convention is u...
Enhancement: convert a global SDS attribute into an HDF5 dataset if the content is too long to be an HDF5 attribute
An HDF5 attribute seems to put a limit on the length. H5Acreate() rejected when the length is 65536. This changelist makes h4toh5 convert that long attribute into an HDF5 dataset only if
- the original attribute is a global SDS attribute
- the attribute name contains "metadata" or "Metadata"
Since it is not easy to figure out the maximum length, it still tries to create an...
Fix a bug occurring when an unlimited dimension with zero elements is converted without EOS2 option.
An internal function convert_sdszerounlimit() calls H4toH5all_dimscale(), but one of parameter may not be what the callee accepts. The fourth argument of H4toH5all_dimscale() should not be the fully-qualified dataset path, but convert_sdszerounlimit() has passed that.
This changelist adds additional parameter to convert_sdszerounlimit() so that the pure SDS name (or NULL) can be passed from ...
Fix a bug that may occur when an SDS has unlimited dimension.
h4toh5 calls SDgetchunkinfo() to get chunking information, but this function may fail if the SDS has unlimited dimension. Since an SDS with unlimited dimension cannot be chunked, this changelist makes h4toh5 check if an SDS has unlimited dimension by calling SDisrecord(). If it has, SDgetchunkinfo() is not called; otherwise, the flow is same.
It seems SDgetchunkinfo() does not always fail even though an SDS has unlimited dimensio...
Fix an error occurring when a Vdata is empty.
VSsetfields() returns FAIL if the vdata is empty. Since h4toh5 treats that FAIL as a fatal error, an empty Vdata resulted in a conversion failure.
After this changelist, h4toh5 does not call VSsetfields() if the Vdata is empty. This was revealed while testing the following HDF4 file:
/mnt/web/internal/ESDIS_files/datapool/abridged/HDF4/GES-DISC/TRMM_2A53_CSI_DARW/2A53_CSI.971128.13.DARW.1.HDF
A script that converts all files in the abridged HDF-EOS2 list.
This script converts each file in the abridged HDF-EOS2 list in two different ways:
1. EOS2 mode (-eos -nc4strict, or -eos -nc4strict -nc4fakedim)
2. default mode (default, or -latest)
Fix a bug regarding fake dimensions for non-EOS2 variables in the non-root group.
To check the name clash, h4toh5 keeps an internal hash table, but it was based on wrong assumptions; a dimension name is unique in a file. Previously, this assumption was abandoned, and h4toh5 began to prepend an EOS2 group name to distinguish dimensions among different groups.
Later, we found an HDF-EOS2 hybrid file that needs fake dimensions. We couldn't prepend an EOS2 group name for this case because the o...
Add -latest option for long attributes.
Some HDF-EOS2 files have very long attributes; They contain 65535 characters. It seems HDF5 has some limitation regarding the length of attribute. I got the following error:
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 0:
#000: ../../hdf5/src/H5Adeprec.c line 165 in H5Acreate1(): unable to create attribute
major: Attribute
minor: Unable to initialize object
#001: ../../hdf5/src/H5A.c line 481 in H5A_create(): unable to c...
Print appropriate error messages when a fake dimension cannot be created.
When -nc4fakedim is specified, dimensions attached to each non-HDF-EOS2 field are also converted. (Those dimensions are called fake dimensions, and the option name may need to be revised.) Those fake dimensions are needed only when a file is hybrid.
A non-HDF-EOS2 field and a non-HDF-EOS2 dimension can have the same name because HDF4 allows this. This is not allowed in HDF5, and it's not straightforward to twist the s...
Suppress bogus error messages when an error occurs.
Bogus error messages have been being printed. Although they are printed only when there is an error, those bogus messages can mislead users and even developers.
h4toh5 sometimes tries a special conversion, and tries another way if the first trial fails. During the first stage, error messages are accumulated even though they don't cause errors. This changelist clears those bogus messages.
Make h4toh5 detect 'nlon' and 'nlat' dimensions used by some files in lieu of 'XDim' and 'YDim'.
Most HDF-EOS2 grid fields refer to 'XDim' and 'YDim' for specifying where the data was measured. Using 'XDim' and 'YDim' is kind of standard and documented in the HDF-EOS2 reference. However, some files do not use 'XDim' and 'YDim'. Instead, they define their own dimensions such as 'nlon' and 'nlat', and all fields refer to those non-standard dimensions.
'XDim' and 'YDim' are important because h...
Verify values in XDim and YDim.
As the comments for the revision #373 states, XDim and YDim in the HDF-EOS2 grid do not have values because they are not necessary. However, some files contain values. In the previous revision, h4toh5 was modified to tolerate unnecessary XDim and YDim.
This change checks if the values in XDim and YDim are the same as expected values calculated by GDij2ll(). If the value is very different, h4toh5 rejects the file.
Allow HDF-EOS2 Dimensions with values.
HDF-EOS2 Grid data don't need to have values in XDim and YDim because they can be calculated. I found a bizarre file that has XDim and YDim values. Previously, that file had caused an assertion failure.
This changelist allows that.
Arrange some code for later changes.
The previous code didn't clearly distinguish data fetched from the file from calculated data, which makes hard to add more features.
Fix a bug regarding one-dimensional Longitude and Latitude geo-location field under Swath.
Previously, I assumed that Longitude and Latitude included in a Swath group are always two-dimensional. It turns out that there is at least an exception. The following file has one-dimensional Longitude and Latitude.
abridged/HDFEOS2/LaRC/MOP02/MOP02-20000303-L2V5.7.1.val.hdf
, which came from
ftp://l4ftl01.larc.nasa.gov/MOPITT/MOP02.003/2000.03.03/MOP02-20000303-L2V5.7.1.val.hdf
The HDF-EOS2 dis...
Convert UNLIMITED dimensions.
SD_UNLIMITED dimension is converted into a chunked HDF5 Dimensional Scales with H5S_UNLIMITED. Previously, UNLIMITED dimensions were not taken into consideration.
When a dimension map exists, adjusted longitude or latitude has been being created, and its name is something like "Longitude_0:-1_0:1".
Due to Revision #357 change, neither colon nor hyphen is not allowed; so, the above name should become "Longitude_0__1_0_1". This was correctly done, but "coordinates" attribute was still indicate "Longitude_0:-1_0:1".
After this fix, "coordinates" attribute will indicate "Longitude_0__1_0_1", which is netCDF-4 safe.
Make more room for the last parameters of GDfieldinfo() or SWfieldinfo().
This is related to Revision #356. At that time, I increased the size of buffer to 256.
We got an answer from an HDF-EOS2 developer saying that UTLSTR_MAX_SIZE (512) can be regarded as the maximum length. (I'm guessing that includes NULL.) So, I increased more.
Make HDF5 Dataset names and attribute names safe.
NetCDF-4 has a strict rule regarding object names; it only accepts the following pattern:
([a-zA-Z0-9_]|{UTF8})([^\x00-\x1F\x7F/]|{UTF8})*
Besides, I believe the first character should not be a digit like most programming languages. This changelist changes the object name if it is not suitable.
This is applied only when netCDF-4 mode is enabled.
Fix a buffer overflow and a problematic code based on wrong assumption.
An HDF-EOS2 API function GDfieldinfo() has a value-result argument giving the comma-separated list of dimensions. The problem is that it seems there is no way to figure out enough buffer size. I got the number 80 from one of HDF-EOS2 tools. (I guess it was heconvert or bindump.) This has worked fine, but Joe found one file that breaks this.
Also, that file revealed another problem. Previously, h4toh5 assumes that each g...
Fix many warnings GCC gives!
* remove all unused variable
* initialize uninitialized variables
Sometimes, this is not trivial because of complex flow. For memory allocation
and release, I made all uses and releases check the validity of the pointer so
that double-free or accessing invalid address will not happen. For HDF5
function pair like H5Gopen() and H5Gclose(), I believe that HDF5 checks whether
or not the argument of H5Gclose() is valid; so, I just initialized the
descriptor with -1....
Fix a bug regarding unlimited dimensions with no elements (bugzilla #1382)
Check emptiness of SDS using both SDcheckempty() and SDgetinfo() because SDcheckempty() had a bug. When no elements exist, h4toh5 calls H5Pset_chunk() with zero, which is not allowed. For this case, h4toh5 calculates a fair chunk size. Currently, h4toh5 tries to put 512 elements in one chunk.
