Add new products from Block/Revision: 0200C JPSS CDFCB-X.
Add -p, --products option to display list of products.
Remove product list from -h, --help output.
Update tests and expected output files to reflect above changes.
Merge pull request #1 in JPSS/jpssnagg from bugfix/NAGG-6-incorporate-patch-from-cspp-sdr-adding to master
* commit 'ea0c72bea0a888ea3d1ecac28c450dd1bbdc2e58':
Revised the patch to support multiple platforms because unlike the N_GEO_Ref attribute, which according to the JPSS Common Data Format Control Book Volume V is used only where the geolocation information for a particular data product is packaged separately, all data products including GEO products should have a Platform_Short_Name ...
Revised the patch to support multiple platforms because unlike the
N_GEO_Ref attribute, which according to the JPSS Common Data Format
Control Book Volume V is used only where the geolocation information for
a particular data product is packaged separately, all data products
including GEO products should have a Platform_Short_Name attribute:
1. Removed get_platform_short_name(), which I believe was a clone
of get_geofile_refs(), and instead called get_platform_value() directly.
2. ...
Commit patch provided by Scott Mindock, CSPP SDR.
The patch accomplishes two things.
1. It adds support for CrIS FS - allow aggregation.
2. It adds support for jpss-1. The platform name was hard coded to npp.
The patch reads the platform from an input hdf and uses on output.
"make check" fails with 59 errors because
1. The additional product has not been added to the help/usage message.
2. Names composed for nagg output files do not have the correct platform
field (It contains the first t...
Update release version to 1.6.2, and documentation files for the 1.6.2 release.
Add messages regarding the NAGG_MAX_GRANULES environment variable to override granule limits.
Revert changes in revisions 425 and 426. This removes the optimizations to presort input files and to skip geo files that were previously processed for a compatible data product.
There was a test failure on ostrich with those changes when built with HDF5 trunk (1.10.0-1) and the xlc compiler. Nagg --debug on fill granules returned errors instead of printing the granule information as expected.
This version passes all tests.
Removed incorrect call to free a local buffer in nagg_get_granules.c that caused make check failures due to memory corruption with xlc, pgcc and icc on Macs. Also addressed a few compiler warnings.
Two configuations still have tests that fail: xlc compiler on ostrich and clang on osx1010test, both linking to hdf5 trunk. The output files seem to be correct, but the creation field in some of the output file names are not replaced with YYYYMMDDSSssssss, resulting in differences from the expect...
Add code to detect repeated reference to geolocation files and process them just once to avoid duplicating the same geo granule for every data product when multiple products are packaged in either input or output files.
Presorting input files is needed to accomplish this by comparing each geolocation product input file to just the previous geo product input file.
Changed %ull to %llu for sprintf to print the orbit number to the output file name. %ull doesn't work with IBM xl compilers, but %llu works there and also for all the Mac and Linux machines.
Restored code from revision 419 that was partially removed by reverse merge in revision 420 for using environment variable MAX_GRANULES_NAGG environment variable to reduce or increase the number of granules allowed to be processed. Also, for processing all the granules at once it is necessary to reset the granules_selected limit as well as the granule_info limit.
Removed the "Finished Printing Granules" line at the end of nagg_print_granules() and also removed the line from 4 expected results...
Fix compose_output_fname function to find the correct ending time for the test with a large number of fill granules. Also corrected the expected output file for the test.
- Commit reverse merge to remove addition of input file presort and processing by filesets (subsets of input files and granules) (r 398). Keep --onefile and --nofill options.
- Use ending time of last granule for the ending time field of output file names. This still does not work for the test with 4 beginning and 4 ending granules separated by 67 fill granules.
- Revise expected output files for tests for changes in filenames.
- Revise tests and expected output files for filter and layout o...
- Change tests for compression and layout changes to grep output for specific words such as COMPRESSION or CHUNKED in outup to avoid differences between HDF5 version for the VALUE output of a Fill_Value setting.
- Add code to set maximum number of granule information pointers to value of environment variable "MAX_GRANULES_NAGG" if it exists instead of the defined limit NAGG_Granule_info_max, to allow users to temporarily raise the number of granules that can be processed by nagg.
- If the nu...
Test files changed for corrections to filenames.
testnagg.sh tests all pass.
test_nagg_more.sh tests:
test with 1885 files fails, and in turn the test to deaggregate the aggregation.
--onefile tests fail.
Still need to add code to change the ending time field of the output filenames when writing the end of the aggregation.
Check in changes made to prevent selecting duplicate granules multiple times, particularly GEO granules that are read for each of multiple data products. Maybe this could be optimized to only read GEO granule for the first product granule read? There are also some debugging statements added. If commented out, they are commented with '//' for removal later.
Reduced make check failures from 38 to 15.
Separated product infile index for reading from index for writing.
Moved call to initialize file tracking structures from nagg_get_granules.c to nagg_main.c to initialize it for number of input files instead of number of files in fileset.
Checking in changes for processing files without limit.
15 failures remaining in make check, some temporary comments still to be removed, and code cleanup and comments needed, but I want to keep track of changes for the remaining bug fixes.
Progress toward processing granule subsets for large files.
Checking in because this version with "-t REDRO -n 32 /home/lrknox/JPSSnagg_current/test/testfiles/REDRO_npp_d20120513_t000*_noaa_ops.h5" will cause a segmentation fault in HDF5 1.8.16 after nagg fails.
HDF5 files for new examples using --onefile and --nofill were too large to include in the source tarfile. They are posted separately; the url was added to the nagg user guide.
Moved 2 GMODO-SVM01_npp_d20130506_*.h5 files from test/testfiles to /mnt/hdf/JPSS/CLASS/GMODO-SVM01, and their tests from testnagg.sh to testnagg_more.sh. They are used to test --onefile and --nofill options, but increased the size of the release tarfile from 17 MB to 354 MB.
Merge changes
- to add --onefile and --nofill command options.
- to presort input files and process the input granules by subsets instead of all together.
The incremental processing hasn't been integrated with the --onefile option. That will be addressed after the nagg 1.6.2-beta release.
Tested jam, platypus, kituo, gcc and icc.
Increased the size of the selected_granules array to match the maximum number of granules processed array to avoid memory corruption with more granules than the previous maximum wi
th the --onefile option. These limits still apply for the size of an ouput aggregation, w
hich should be eliminated for the full nagg 1.6.2 release.
Tested kituo (CentOS7, gcc 4.8.2), 1885 files with --onefile option and with make check.
Removed commented code that was tried and abandoned.
Tested playpus.
This line, and those below, will be ignored--
M src/nagg_select_granules.c
M src/nagg_main.c
testfiles/satms_tatms2m-dump.txt
redro3S-dump.txt
redro1-dump.txt
redro3-dump.txt
The line "Finished Printing Granules" was removed from the nagg_print_granules function so that it would not appear in the middle of a list of granules produced from multiple subsets of granules. Accordingly, the line has been removed from these expected output files.
test/testnagg.sh
One test that used input granules that were inconsistent from an early file has been comm...
Add presorting of the input files according to the date and start time fields in the file name (dYYYYMMDD_THHMMSSS).
nagg_presort_infiles.c: new file with new function nagg_sort_inputfiles(), regex functions to select the fields, and comparison function to pass th HDqsort.
nagg.h add prototype for the new function.
nagg_main.c call the new function before calling nagg_get_granules() to read granules in input files.
Makefile.* ...
