This is the FACT construction data ingest and verification scripts README

In order to ingest data, use either auxIngest.sh or rawIngest.sh appropriately.
Raw Ingest:
	- if file is corrupted, a tentative repair is done
		- add missing END tag
		- reduce file size so that it re-become compliant to fits
	- Keywords are checked and added or updated if required. 
	- checksum is added to the file
	- if file has been repaired successfully, then it is moved to the archive

Aux Ingest:
	- if file is corrupted, a tentative repair is done
		- add missing END tag
		- reduce file size so that it re-become compliant to fits
			- if more rows seem to be present, restore extra rows while the time value seems correct (i.e. around the expected value)
		- If mjdref was not substracted to Time column, Time column is updated accordingly
	- Keywords are checked and added or updated if required.
	- checksum is added to the file
	- if file has been repaired successfully, then it is moved to the archive	
	
	
Checked keywords:
	- TSTARTI
	- TSTARTF
	- TSTOPI
	- TSTOPF
	- DATE-OBS
	- DATE-END
	- TELESCOP
	- PACKAGE
	- ORIGIN
	- TIMEUNIT
	- MJDREF
	- TIMESYS

It takes source and dest folders as argument, the last arg being the string to append to log filenames.

You will need the programs which source is included in this folder: please compile them appropriately

The ingested data headers can be checked with checkForHeaderUpdatesAux/Raw.sh

while the actual data can be compared with checkRawFioleIntegrityAndContent.sh

This script will not work on aux data because the time column may have been updated. hence different files.