Add new products from Block/Revision: 0200C JPSS CDFCB-X.Add -p, --products option to display list of products.
Remove product list from -h, --help output.
Update tests and expected output files to reflect above changes.
Merge pull request #1 in JPSS/jpssnagg from bugfix/NAGG-6-incorporate-patch-from-cspp-sdr-adding to master* commit 'ea0c72bea0a888ea3d1ecac28c450dd1bbdc2e58':
Revised the patch to support multiple platforms because unlike the N_GEO_Ref attribute, which according to the JPSS Common Data Format Control Book Volume V is used only where the geolocation information for a particular data product is packaged separately, all data products including GEO products should have a Platform_Short_Name attribut...
Revised the patch to support multiple platforms because unlike the N_GEO_Ref attribute, which according to the JPSS Common Data Format Control Book Volume V is used only where the geolocation information for a particular data product is packaged separately, all data products including GEO products should have a Platform_Short_Name attribute: 1. Removed get_platform_short_name(), which I b...I've also added a few minor changes otherwise to correct an issue we
were seeing on a ppc64 system, to eliminate some compiler warnings, and
changes to expected output files to reflect the addition of the platform
short name to the nagg granule structures.
Added the ACKNOWLEDGEMENTS file.
Commit patch provided by Scott Mindock, CSPP SDR.The patch accomplishes two things.
1. It adds support for CrIS FS - allow aggregation.
2. It adds support for jpss-1. The platform name was hard coded to npp.
The patch reads the platform from an input hdf and uses on output.
"make check" fails with 59 errors because
1. The additional product has not been added to the help/usage message.
2. Names composed for nagg output files do not have th...
Update release version to 1.6.2, and documentation files for the 1.6.2 release.Add messages regarding the NAGG_MAX_GRANULES environment variable to override granule limits.
Revert changes in revisions 425 and 426. This removes the optimizations to presort input files and to skip geo files that were previously processed for a compatible data product. There was a test failure on ostrich with those changes when built with HDF5 trunk (1.10.0-1) and the xlc compiler. Nagg --debug on fill granules returned errors instead of printing the granule information as expected.
This version passes all tests.
Removed incorrect call to free a local buffer in nagg_get_granules.c that caused make check failures due to memory corruption with xlc, pgcc and icc on Macs. Also addressed a few compiler warnings.Two configuations still have tests that fail: xlc compiler on ostrich and clang on osx1010test, both linking to hdf5 trunk. The output files seem to be correct, but the creation field in some of the output file names are not replaced with YYYYMMDDSSssssss, resulting in differences from the expected output.
Add code to detect repeated reference to geolocation files and process them just once to avoid duplicating the same geo granule for every data product when multiple products are packaged in either input or output files.Presorting input files is needed to accomplish this by comparing each geolocation product input file to just the previous geo product input file.
Changed %ull to %llu for sprintf to print the orbit number to the output file name. %ull doesn't work with IBM xl compilers, but %llu works there and also for all the Mac and Linux machines.
Restored code from revision 419 that was partially removed by reverse merge in revision 420 for using environment variable MAX_GRANULES_NAGG environment variable to reduce or increase the number of granules allowed to be processed. Also, for processing all the granules at once it is necessary to reset the granules_selected limit as well as the granule_info limit. Removed the "Finished Printing ...All tests pass, including extensive tests.
Fix compose_output_fname function to find the correct ending time for the test with a large number of fill granules. Also corrected the expected output file for the test.
- Commit reverse merge to remove addition of input file presort and processing by filesets (subsets of input files and granules) (r 398). Keep --onefile and --nofill options.- Use ending time of last granule for the ending time field of output file names. This still does not work for the test with 4 beginning and 4 ending granules separated by 67 fill granules.
- Revise expected output files for tests for changes in filenames.
- Revise tests and expected output files for filter and layout options due to changed output of h5dump in HDF5 v1.8.17.
- Change tests for compression and layout changes to grep output for specific words such as COMPRESSION or CHUNKED in outup to avoid differences between HDF5 version for the VALUE output of a Fill_Value setting.- Add code to set maximum number of granule information pointers to value of environment variable "MAX_GRANULES_NAGG" if it exists instead of the defined limit NAGG_Granule_info_max, to allow users to temporarily raise the number of granules that can be processed by nagg.
- If the number of granules in a product group is not the same as the value of the AggregateNumberGranules attribute of the...
HDF5 files for new examples using --onefile and --nofill were too large to include in the source tarfile. They are posted separately; the url was added to the nagg user guide.
Moved 2 GMODO-SVM01_npp_d20130506_*.h5 files from test/testfiles to /mnt/hdf/JPSS/CLASS/GMODO-SVM01, and their tests from testnagg.sh to testnagg_more.sh. They are used to test --onefile and --nofill options, but increased the size of the release tarfile from 17 MB to 354 MB.
Merge changes - to add --onefile and --nofill command options. - to presort input files and process the input granules by subsets instead of all together.The incremental processing hasn't been integrated with the --onefile option. That will be addressed after the nagg 1.6.2-beta release.
Tested jam, platypus, kituo, gcc and icc.
Increased the size of the selected_granules array to match the maximum number of granules processed array to avoid memory corruption with more granules than the previous maximum wi th the --onefile option. These limits still apply for the size of an ouput aggregation, w hich should be eliminated for the full nagg 1.6.2 release.Tested kituo (CentOS7, gcc 4.8.2), 1885 files with --onefile option and with make check.
Removed commented code that was tried and abandoned.Tested playpus.
This line, and those below, will be ignored--
M src/nagg_select_granules.c
M src/nagg_main.c
testfiles/satms_tatms2m-dump.txt redro3S-dump.txt redro1-dump.txt redro3-dump.txt The line "Finished Printing Granules" was removed from the nagg_print_granules function so that it would not appear in the middle of a list of granules produced from multiple subsets of granules. Accordingly, the line has been removed from these expected output files.test/testnagg.sh
One test that used input granules that were inconsistent from an early file has been commented out. The test behaves differently when the files are pre-sorted. The resolution of the inconsistent granule problem is still uncertain.
src/nagg_presort_infiles.c
New file with functions for sorting files according to the beginning times of the aggregations contained.
src/...
Add presorting of the input files according to the date and start time fields in the file name (dYYYYMMDD_THHMMSSS). nagg_presort_infiles.c: new file with new function nagg_sort_inputfiles(), regex functions to select the fields, and comparison function to pass th HDqsort. nagg.h add prototype for the new function. nagg_main.c call the new fu...Tested jam and platypus, gcc and icc.
test/testnagg.sh: added explanation of available test script functions. began adding test ids to tests for matching with requirements document test/nagg_functions.sh added parameter to h5dump_grep option for future use of grep options src/nagg_write_granules.c added H5Rdefeference_ macro coutesy of Scot Breitenfeld to call the ...Tested on platypus with gcc and icc compilers
Update nagg version 1.6.1 release date. Check object ids with H5Iis_valid before closing at end of functions.Tested on jam, koala and platypus with gcc and icc.
nagg.h increased size of NAGG_Product_Path_size from 79 to 128 to eliminate segmentation fault for a product with a long name. nagg_get_granules.c added debug code to print numbers of open objects immediately before calling H5Fclose(). nagg_write_granules.c added debug code to print numbers of open objects immediately before calling H5Fclose(). added code to close input files on...Tested on platypus and jam with gcc and icc.
nagg_write_granules.c Output files were not closed due to open groups and attributes. These are now closed.
This became apparent when attempting to deaggregate the file resulting from the aggregation and packaging of 136 granules of 16 VIIRS products + GEO, which exceeded the 1024 system limit for open files.
test script and files
Increased the aggregation number to 129 to get a full bucket in the outpu...
nagg_write_granules.c Refactored code to extract create_granule_regrefs() function from copy_granule_hyperslabs. Added missing function comment headers. RELEASE.txt Added entries for JPSS-86 and JPSS-87 fixes. nagg_get_granules.c toofewgranules.txt Test failed when make check was run for a build configured with an absolute path to configure instead of a r...Tested jam and platypus, gcc & icc.
Fixed JPSS-86 and JPSS-87. Region references are now created for hyperslabs with no stored data because they contain entirely fill values. nagg will now handle files that have missing region references and create the missing references in the output files. While addressing these issues, a file was created with a 12 granule aggregation but only one granule dataset in the /Data_Products group...Tests and test files added for datasets with all fill values, for files with missing region references, and for files that can be opened but are missing granule datsets.
Tested on jam and platypus with gcc & icc and extensive data tests.
Initial fix for JPSS-86 and JPSS-87. Region references are now created for hyperslabs that have only fill values, even though no data is written to the hyperslab. Code is still needed to create missing references that were not written by an earlier version of nagg. Some memory leaks were also addressed, but others remain.Tested on jam and platypus with gcc & icc.
Larry Knox