I need to save data files daily in a backup folder, but still keep an up-to-the-minute copy with a static name in a different folder so a database software (VDV) can import it into a database. I do not want this updated copy to get too big, though. Since it does not appear that Split can 'move' older data from a file, does anyone have any suggestions?
I also do not want to overly tax the datalogger by requesting a days worth of data at the end of each day, as we are collecting secondly data. We're running CR1000s on Ethernet (NL115) using Loggernet Admin.
Hello Matthew,
I think you have a couple of options. Here are things to consider:
While Split itself cannot "move" older data from a file, you can certainly move, rename, append, etc. data from a batch file (remember the DOS days?). LoggerNet's Task Master can run batch files, and it can also run Split's PAR files (see the Task Master help for an example on how to do this). One way I might accomplish what you want to do using Split is to process the file once daily (for instance, a little after midnight), generating a file with a day's worth of data. Then this file could be moved into a backup directory using a batch file. The original file is still available for VDV to ingest.
Another way to get a daily file is to use the TableFile instruction in the datalogger, to store a specified "chunk" of data to a file on the datalogger's USR drive (see the CRBasic help for info on TableFile and also USR drive). That file could then be collected from the datalogger and stored to a backup directory using LoggerNet's File Retrieval tab. Alternately, since you have an NL115, the datalogger could FTP that file to a specified FTP site.
Regarding managing the file size of the file being ingested by VDV, the most recent release of LoggerNet has an option in the Setup screen for "Maximum data file size". By default, it is set to -1 (no max), but you could set this to some value to manage the file size if needed. When the file size was reached, the existing file would be saved with an archival name, and a new file would be started with the original name.
Dana W.
* Last updated by: Dana on 7/22/2014 @ 2:10 PM *
Thank you for the suggestions. I am concerned that setting a maximum file size will periodically cause a day's data to be split between two files (the archived and the current), thus making the creation of daily files difficult, especially with the archived file changing names each time.
Maybe instead, at 00:00 or a time shortly after, I use Split to create a file with the previous day's data, which I archive. Then I use Split to create a file that includes data from the previous day up to the present (in case VDV was slow to read it), then I use a batch file to move this file and replace the current, original file that VDV is reading. I am concerned, however, that during the time it is creating this second file, a data record may have been read into the original file that I then lose when overwriting it. I could also possibly lose data from overwrites if Task Master doesn't run one day, if VDV doesn't read data for more than a day, or data can't be retrieved from a datalogger for more than a day.
I do not think I want to go the TableFile and File Retrieval or FTP route because there is not much free datalogger processor time due to the 1s scan intervals, and this lack of overhead may cause problems when transferring the relatively large files.