hi,
I have a sensor that outputs a ASC II data every 6 seconds.
The data output is something like this. D,2,0,2.27e3,6.0,5.875,66784,0,308
I want to store a 30 seconds average data using 6 seconds data.
this is my code :
'Define or declare a variable to gather from CPC (Raw data) and then store it as a processed variable
SequentialMode
Public PTemp, batt_volt
Public Rawdata as String * 60
Public splitRawdata(8)
Public NBytesReturned
DataTable(aerosol,1,-1)
DataInterval(0,30, sec, 10)
Minimum(1, batt_volt, FP2, 0,False)
Sample (1, splitRawdata(1), FP2)
Sample (1,splitRawdata(2), FP2)
Sample (1, splitRawdata(3), Float)
Sample(1,splitRawdata(6), Float)
Average ( 1,splitRawdata(3), Float, False)
EndTable
BeginProg
SerialOpen(com1, 115200,0,0,60)
Scan (6, sec, 0,0)
Battery (batt_volt)
SerialInRecord(com1,Rawdata, &h44,0,&h0D, NBytesReturned,01)
SplitStr(splitRawdata(),Rawdata,",",8,4)
CallTable aerosol
NextScan
End Prog
Can anyone tell me if this is correct way of averaging the data?
Also do I have to work with the buffer size? A single thread of data should be 50 bytes or less. Do I need to increase the buffer size for averaging?
And what other changes do i have to make?
Thanks
Aerosol
* Last updated by: Aerosol on 7/5/2011 @ 1:46 PM *
The average instruction within the data table is where the averaging processing happens, so the source variable does not need to do anything special. The important thing is that the data table is 'called' on each scan, which is what you're doing. Presumably the average instruction initializes and manages its own buffer internally, behind the scenes. The code looks ok to me.
Thanks Kirving.
Two questions!
a) before the average command is executed, all the 6 seconds data from the sensor is stored somewhere in the logger , right?. I don't have to specify the buffer or some kind of memory for those data?
b) My sensor also output an S record, pretty much same as D record in my first mail. But the manual of sensor says I need to change the mode value ( value right after D, in the record string) of 2 to 4 to get the S record along with the D record. ( i have put the mode in square bracket)
D,[2],0,2.27e3,6.0,5.875,66784,0,308
Using a different terminal emulator , I was able to change the mode and access the D and S record which looks something like this..
D,4,0,2.27e3,6.0,5.875,66784,0,308
S, 1.00,972,20.0,60.0,50.0
D,4,0,2.27e3,6.0,5.875,66784,0,308
S, 1.00,972,20.0,60.0,50.0
on and on...
> a) before the average command is executed, all the 6 seconds data from the sensor is stored somewhere in the logger , right?. I don't have to specify the buffer or some kind of memory for those data?
That's correct, to my knowledge. The logger handles that when it 'compiles' the program; it sees the average command and sets up whatever buffers it needs.
> b) ...
I don't understand the question, but pretty much anything you're doing in a terminal emulator could also be done in the program by sending the right strings to the instrument.
sorry for the confusion.
The data output from my sensor contains both the D and S record or strings.
I don't know how the mode value can switch between D or S or both.
What string of commands do i need to read the S record?
Can you suggest?
It looks like you're parsing each line into an array, using comma as the separator, so it might be enough to test the first entry and do different things if it's 'S' or 'D'. For instance, you might define an array to hold the D values and one to hold S values (and maybe define 'alias' names for those entries), then log the results.
Hey Kirving,
Now I have another issue with data logging.
The data output mode of sensor (RS232 serial ASCII output) could be changed, so that it can be used to send data every seconds. While using a seperate terminal emulator, I use SM,2,10 command format to change the mode.
With the CR basic program, here is what i wrote (almost the same compared to the last program in the thread, with some addition).
SequentialMode
Public PTemp, batt_volt
Public Rawdata As String * 60
Public splitRawdata(8)
Public NBytesReturned
Public Outstring As String * 10
Alias splitRawdata(1) = mode
Alias splitRawdata(2) = Flags
Alias splitRawdata(3) = CN
Alias splitRawdata (4) = ST
Alias splitRawdata(5) = LT
Alias splitRawdata(6) = CNT
Alias splitRawdata(7) = PH
Alias splitRawdata(8) = RP
DataTable(aerosol2, 1,-1)
DataInterval(0,1,sec,10)
Sample (8,splitRawdata(1),Float)
EndTable
BeginProg
SerialOpen(com1, 115200,0,0,60)
Outstring = "SM,2,10" + CHR(13)+ CHR(10)
SerialOut(com1,Outstring,"",0,100)
Scan (1, sec, 0,0)
Battery (batt_volt)
SerialInRecord(com1,Rawdata, &h44,0,&h0D, NBytesReturned,01)
SplitStr(splitRawdata(),Rawdata,",",8,4)
CallTable aerosol2
NextScan
End Prog
The problem or the confusion is that I can see a lag of 1 seconds for the stored data in the logger compared to what the sensor displays in real time. How to avoid that?
I want to store the data at the same time the logger is measuring the data( i think the logger's measurement time and sensor output time are same).
Help me out.
* Last updated by: Aerosol on 7/20/2011 @ 5:27 PM *
I can't look into the details now, but maybe you'll need to scan at something faster than a second to see the results in real time?
if the sensor is sending me at 1 seconds interval , what do you think the scan should be?
Is there a way to determine the possible scan time?
I haven't used the loggers quite this way, and don't have access to the Windows help files, so can just try to offer some general points.
The serial data is retrieved on the logger by the serialInRecord() function, which has to either wait for the data to become available or perhaps it's already in the buffer. In either case, that function returns when a record is available, at which point you can parse and log it.
In a free-running system, or perhaps with a scan of 1/10 second or so, the above function would presumably block until ready to return, and maybe would be more to your liking. Maybe you'd get skipped scan counts, I don't know
As an exercise, consider if the data was produced once per day, and your scan was also once per day. At some time in the day, the data would be written by the source machine, and it would sit in a buffer somewhere. At some other time, midnight perhaps, your logger program runs, gets the data and you can process it. In this case, you would only see the 'current' data at midnight, but it was produced possibly up to 24 hours previously.
If, in this scenario, you instead scan every hour, then you'll get the data within an hour of when it's generated.
I think that's maybe what's happening, albeit at 1 second intervals instead of a day.
Ok. Kirving let me try that.
I have added Serialflush at the end of splitstr. I was getting some different values especially for splitRawdata(3) which is very important for me.
D,4,0,2.27e3,6.0,5.875,66784,0,308 is the serial data thread from the sensor.
When I try to compare the 1 seconds value by looking at the display of sensor and real time in the logger, I can see some discrepency on the fourth value in the string i.e (splitRawdata(3)). That value is a 4 digit number and I can see the difference in the last two number. Why is the logger reporting like this?
A few comments on this discussion to help:
a) The average option does not keep a buffer of measurements. As with most other statistical instructions (the exception being median), the logger stores partial statistics and uses these to do the final calculation when it has to store the result. In the case of the average it stores a running total and count of measurements. When it outputs it divides one by the other and resets both to zero for the next period. This means there is not extra cost in memory terms of running averages or most other statistics for log periods.
b) The Serialinrecord only processes data from the serial buffer, i.e. it must already have come into the logger. Simply put, it scans the buffer and pulls out the oldest or most recent whole data record and writes it to a string. If it cannot find a whole record you can get it to either write the last whole record it saw to the string or write NaN to indicate missing data.
c) There will always be issues with serial sensors that spit out data on their own time base, mainly because the clocks in the sensor will drift relative to the logger, so even if you set the output interval to be the same, if the sensor clock runs slow, you will occasionally miss a value or if it runs fast it will output an extra value.
Serialinrecord can help with problem (c). If the sensor runs slow and a value is missing you can duplicate the last one, or if fast and you are getting the most recent record the logger will ignore the extra record.
You will always expect the logger to run behind the sensor to some degree without some interaction between the two sides. To get the tightest synchronisation you need to switch to polling the sensor (if it allows this) so the logger can trigger a measurement when it wants it.
My suspicion is that there is a delay in which the sensor echos the data out the serial port or that waiting on the CR (&h0D) is holding up the ingest one cycle.
It would be informative if you could enter the terminal mode while the program is running. Capture the output of two modes and post them here.
a) W (serial watch) -> 9 (com1) -> Y (ASCII)
and
b) W (serial watch) -> 9 (com1) -> N (Binary)
Hey sam,
This is the terminal output.
ASC
R 12:34:54.90 D,2,0,4.79e3,1.0,0.954,22879,0,321
R 12:34:55.90 D,2,0,4.75e3,1.0,0.954,22700,0,319
R 12:34:56.90 D,2,0,4.82e3,1.0,0.954,23043,0,321
R 12:34:57.90 D,2,0,4.78e3,1.0,0.954,22853,0,321
Binary
R 12:36:56.89 44 2C 32 2C 30 2C 34 2E 37 32 65 33 2C 31 2E 30 2C 30 2E 39
R 12:36:56.89 35 34 2C 32 32 35 37 37 2C 30 2C 33 31 38 0D
R 12:36:57.89 44 2C 32 2C 30 2C 34 2E 37 31 65 33 2C 31 2E 30 2C 30 2E 39
R 12:36:57.89 35 35 2C 32 32 35 33 36 2C 30 2C 33 31 35 0D
R 12:36:58.89 44 2C 32 2C 30 2C 34 2E 36 36 65 33 2C 31 2E 30 2C 30 2E 39
R 12:36:58.89 35 35 2C 32 32 32 36 34 2C 30 2C 33 31 36 0D
The binary output has two rows coming out. it could be two types of record. Is it two types of data string. My sensor sends two data record. The one shown in the ASC is D record.
Let me know.
Aerosol,
Thanks for the output. It appears the data is consistently output 900 milliseconds after the second. I am guessing that the 1 second delay (logging) is definitely a timing issue - i.e the data logger scan has processed the buffer before the new record from the sensor for that second has actually arrived at the data logger. Time/data synchronicity is always a tough one when working with two independent devices with independent clocks.
CRBasic does not provide a way of handling serial interrupts. The best we can do is continually a) check if there is data in the serial buffer or b) check to see if the ComPortIsActive. Below is one way you could try catching the data at the moment it comes in.
------------
Const COM = COM1
Public Rawdata As String * 60
Public splitRawdata(8)
Public NBytesReturned
Public Outstring As String * 10
DataTable(aerosol2, 1,1000)
Sample (8,splitRawdata(1),Float)
EndTable
BeginProg
Scan (1,Sec,0,0)
NextScan
SlowSequence
SerialOpen(COM,115200,0,0,200)
Outstring = "SM,2,10" + CHR(13)+ CHR(10)
SerialOut(COM,Outstring,"",0,100)
Do While TRUE
If SerialInChk (COMRS232) Then
SerialInRecord(COM,Rawdata, &h44,0,&h0D, NBytesReturned,01)
SplitStr(splitRawdata(),Rawdata,",",8,4)
If NBytesReturned Then
CallTable aerosol2
NBytesReturned = 0
EndIf
EndIf
Loop
EndSequence
EndProg
-------------