RING BUFFER 18LF4620 Oshonsoft

[camerart]

C, it does not matter much what kind of buffer you use, it is how you handle the interrupt and the data it gets. Think in time frames of 1 millisecond (mS)

First, try to get the interrupt RX routine to determine if the received characters are valid, and in the proper position in the string. For example, if you expect "$" as the first character and "A" as the 10th character, then the interrupt routine keeps track of what position your RX buffer is at, and if the expected character is not "correct", you zero the pointers/length and just go onto reading the next character. If length pointer happens to go past the buffer length (maximum message length), there is something wrong, and simply zero your pointers and counter in the interrupt routine and wait for next character. Do so until you get the right data stream and proper terminator. At that point, you set a flag that the UART RX buffer has a valid string and is complete. Nothing else!
All this can be done in the interrupt routine in the order of micro seconds (uS), maybe as much as 100uS for slower clock rates. That is 1/10th the rate at which characters arrive at the UART at 9600 baud (approx 1mS each). Keep it simple, use lots of flags and pointers, and parse the incoming characters as "valid" ONLY, do not interpret the string itself, that is for the main program to do! You basically filter the RX characters for valid data format only within the interrupt routine. Anything invalid, you reset things within the interrupt routine.
You have to think in terms on microseconds, a chip can do hundreds, if not thousands of instructions between each UART character. Your main program can then parse the RX buffer when it can, usually within the time frame before the next incoming string is complete and valid. After that point, the interrupt routine ignores all RX characters until the main program has processed the message and cleared the flags.

For example, I have an RX interrupt routine. looking for a string of characters. First I check OERR, FERR, and such at start of interrupt routine. First character has to be "FE", and so on, 5th character has to be a "03". End of message is "FD" If any of those requirements are not correct, I zero my pointers and wait for the proper message within the interrupt routine. Simple...
Now, I do something different in the main loop. Assume I have a valid RX buffer of 20 characters. Flags are set to show the length of message, and string is complete and correct format by the interrupt routine. Main program now detects valid message via those flags and I save the pointers/length into new variables, and start to copy the RX buffer into another buffer, a "working buffer" Since copying characters from buffer to buffer is so fast (<20uS at 8Mhz clock), I can simply transfer 5 or 10 characters and then reset the interrupt flags/pointers while I copy the rest of the working buffer. This way, I allow more messages to be received by the interrupt routine while I still work on the work buffer in the main program loop. This is perfectly safe, as it will take a whole millisecond (mS) before the interrupt routine will trigger again, and for one character only. Next character will take another whole 1mS. By then, I've saved the entire working buffer, and main program can take its time processing it. No interaction between RX interrupt routine and its RX buffer vs the main program loop and its "working buffer"

I know this can be overwhelming to some. You have to think in terms of microprocessor speeds, and your UART speeds (or other data). The micro can process so much data between UART characters, you just have to think like a microprocessor to get the timing right, and see that 1mS for a UART character is a very, very long time.

Hi S,
Thank you for that information. Prevously 'J' had shown me how to time the INTERRUPT in Oshonsoft and the answer is a you suggest app, 8uS.

The problem I'm having is when WRITING to the BUFFER, an increment VARIABLE is used, how do I increment through the BUFFER without affecting the next incoming DATA? So far I've been using the same VARIABLE, but I'm sure this isn't correct.
C

 

[sagor1]

That is why I use two buffers, one in main program, one in the interrupt routine. Each has its own pointer and buffer length definitions.
Until the RX interrupt routine has a complete, valid string, there is nothing to process. Let the interrupt routine do its work and then let the main program know it has something valid to work on only when data is complete and valid.
You have 2 choices. You can copy the buffer to a working buffer, if you have enough ram space. Or, just copy the RX data that you NEED into a smaller buffer. Or, you disable the UART when you have a complete valid string and process that RX buffer. Other choice was that ring buffer concept, but that uses lots of ram as well.
The question is, is it all that bad if you turn off the UART (or UART interrupt enable) for several mS to process the RX buffer, then turn it back on once done? Yes, you may miss a string of data, but if it is coming in fairly frequently, does it really matter? With the UART off, can you quickly parse the RX buffer, determine what data you need from it (save critical parts of information), then zero the pointers and re-enable the UART as fast as you can? The UART can then overwrite the RX buffer if you have extracted the data you "need". If you can do that within several mS, you may not miss any incoming data anyway.
If you leave the UART on but disable its interrupt, you still have 2mS to process the RX buffer by the main program due to the RX FIFO caching the first 2 characters. Main program should not be doing anything that "takes time" like serial output while processing the RX buffer. Serial output is a killer, it takes so much time, and you cannot read flags from the RX interrupt routine in time. The real solution to that is to use TX interrupts as well, and send data only during TX interrupts (TXREG is empty). Using Oshonsoft HSEROUT kills a lot of time, and you cannot process anything in the main loop while that command is running. If you output one character at a time, you are still using up at least 1mS of time. That is a lot of time in the main program loop to be wasting...
Think of it this way, you can use interrupts to trigger timers, UART RX received characters, UART TX buffer empty and read for next TX character, and many other things - all interleaved at the same time. All the main program has to do is check the status of each of those interrupt routines.

 

[camerart]

That is why I use two buffers, one in main program, one in the interrupt routine. Each has its own pointer and buffer length definitions.
Until the RX interrupt routine has a complete, valid string, there is nothing to process. Let the interrupt routine do its work and then let the main program know it has something valid to work on only when data is complete and valid.
You have 2 choices. You can copy the buffer to a working buffer, if you have enough ram space. Or, just copy the RX data that you NEED into a smaller buffer. Or, you disable the UART when you have a complete valid string and process that RX buffer. Other choice was that ring buffer concept, but that uses lots of ram as well.
The question is, is it all that bad if you turn off the UART (or UART interrupt enable) for several mS to process the RX buffer, then turn it back on once done? Yes, you may miss a string of data, but if it is coming in fairly frequently, does it really matter? With the UART off, can you quickly parse the RX buffer, determine what data you need from it (save critical parts of information), then zero the pointers and re-enable the UART as fast as you can? The UART can then overwrite the RX buffer if you have extracted the data you "need". If you can do that within several mS, you may not miss any incoming data anyway.
If you leave the UART on but disable its interrupt, you still have 2mS to process the RX buffer by the main program due to the RX FIFO caching the first 2 characters. Main program should not be doing anything that "takes time" like serial output while processing the RX buffer. Serial output is a killer, it takes so much time, and you cannot read flags from the RX interrupt routine in time. The real solution to that is to use TX interrupts as well, and send data only during TX interrupts (TXREG is empty). Using Oshonsoft HSEROUT kills a lot of time, and you cannot process anything in the main loop while that command is running. If you output one character at a time, you are still using up at least 1mS of time. That is a lot of time in the main program loop to be wasting...
Think of it this way, you can use interrupts to trigger timers, UART RX received characters, UART TX buffer empty and read for next TX character, and many other things - all interleaved at the same time. All the main program has to do is check the status of each of those interrupt routines.

Hi S,
If I look at previous programs, where the INTERRUPT/BUFFER is working, but giving some sort of error. (This is why I'm starting again) I can see MSG1, which is a STRING, and I think this is what you would call the second BUFFER.

Here is a screen shot of an example message [11111$22222W33333 ] with a $ at the HEAD and W at the TAIL of the message.
Using message as the second BUFFER.

Step1 How do I get [ $ ] into MSG1?
Step2 How do I get [ 22222 ] into MSG1? (In words, not programming language, unless Oshonsoft)

I get the impression, that if the BUFFER is zeroed, then it could be classed as LINEAR BUFFER and if it LOOPS then it could be classed as a RING BUFFER.

I'll post the result.
C

 

[camerart]

Hi,
I've gone round in circles, and arrived in the past where there were 2x VARIABLES for the 2x LOOPS, WRITE and READ.
I'll persue this line for a while.
EDIT: The above paid off, as I've now got something in a READ STRING
C

 

[jjw]

Hi,
I've gone round in circles, and arrived in the past where there were 2x VARIABLES for the 2x LOOPS, WRITE and READ.
I'll persue this line for a while.
EDIT: The above paid off, as I've now got something in a READ STRING
C

 

[jjw]

You change the Rxhead and Rxtail in main and in ISR at the same time.-> a mess.
Start reading the message in the main only when the message is complete.

 

[camerart]

You change the Rxhead and Rxtail in main and in ISR at the same time.-> a mess.
Start reading the message in the main only when the message is complete.

Hi J,
And I thought I was having a good day
C

 

[Ian Rogers]

Hi J,
And I thought I was having a good day
C

You are.... I will find that piggin routine.... It makes it effortless... I may have to rewite and send it to you.... On the plus side I found my software USART with interrupt function...

 

[sagor1]

C, for your example string of 11111$22222W33333 , you don't bother saving the 1's or the 3's, you drop them in the interrupt routine. In the interrupt routine, you start by dropping characters until you see a valid start character, "$". Set a flag in the interrupt routine that you have started receiving a valid string (valid start character). Continue until you see the "W", and which point you set a flag that a valid string is in the buffer, and save the length of the string in a variable. Clear the "valid start" flag, so next character(s) are dropped as well. Main program then checks "valid string complete" flag, and reads the buffer characters up to the "length" you set in the interrupt routine. Once main program has read all the buffer, main program resets the flags and counters of the interrupt routine. Interrupt routine keeps running in background, but dumps the unwanted characters....
By saving just the data you need from a serial string of data, you can reduce the buffer lengths to just the maximum length you need for the data only. You do not need a buffer as long as the entire string if you don't need all the data in that string. Have the interrupt routine store only what is valid and drop the rest (just read the register to clear it and exit).
Something like this code. It may not be correct everywhere because I can't compile it without complete interrupt setup. You have to figure the rest out. This example shows how to read only what is needed from a UART string, and dumps the rest. Main program processes the string only when there is a valid complete string:

AllDigital
Dim i As Byte
Dim j As Byte
Dim char As Byte

Dim uart_buff(40) As Byte
Dim uart_complete As Bit
Dim uart_len As Byte
Dim uart_valid As Bit

uart_complete = False
uart_valid = False
uart_len = 0

'Set up your HSEROPEN and interrupt enable flags here


main:

If uart_complete = True Then
'process the uart data
For i = 0 To uart_len
    j = uart_buff(i)  'do what you want here
Next i

    'when finished...
    uart_len = 0
    uart_complete = False
    uart_valid = False
Endif

Goto main  'Main loop

End                                               

'Interrupt routine
Save System
'Do usual checks for OERR, etc...


char = RCREG
If uart_complete = True Then Goto wayout  'Nothing to do, old string has not been read by main yet
'Above simply dumps all characters until main program has processed the current saved string

If uart_valid = True Then  'have a valid string in progress, save everything until end of string
    uart_buff(uart_len) = char
    uart_len = uart_len + 1
'Can check for valid length pointer here if desired, to prevent overflow of buffer
    If char = "W" Then Goto end_string
    
Else  'not valid start of string yet. Check if start char is there
    If char = "$" Then  'Valid start character
        uart_valid = True
        uart_buff(0) = char  'HAS to be first buffer character
        uart_len = 1`  'We know next character will be location 1
    Endif
Endif

Goto wayout  'Skip any other processing.
'Above stores character only if valid string in progress or it is first start
'character ($). Otherwise, routine just jumps to "wayout", exit interrupt routine
'You can change the logic around to your liking.

end_string:  'reached valid ending
uart_comlete = True

wayout:
Resume

 

[camerart]

Hi S, I and J,
I was pleased about simply getting the MAIN LOOP to put anything in the VARIABLE. I wasn't really showing programming skills

Thanks for any CODE I will try to read it, but for me it takes too long, apart for Oshonsoft routines.

I'm seem to be getting conflicting advice! My logic tells me that as the INTERRUPT is adding digits to the BUFFER, they can be taken off at the same time, but I keep being told the whole sentence must be put into the VARIABLE first before removing. Any clarity welcome.

The CODE I posted appears to WRITE and READ the sentence digits without using HSERIN or OUT, is this possible?
C.

 

[sagor1]

Yes, it is possible to do TX and RX on a UART without using HSEROUT or HSERIN. Using HSEROPEN just sets up the ports and baud rate generator, you do not have to use his other routines if you don't want to.
It is possible to read a buffer while the interrupt routine is still filling it with characters, as with your "head" and "tail" examples. It all comes down to how much data you are receiving, how fast it is coming in, and whatever your main program is doing with that data. If you need a completed string, you cannot process just a part of it - you need all of it. So why read only a part of the string while it is still coming in?
I find that for smaller strings of data, it is rather pointless for the main program to start reading the input buffer while it is still filling up, as the main program is just wasting time waiting for the rest of the string before it can process the final result. That is, I find no point in processing the input string until it is complete. The processing of the input string by the main program can be done very quickly, before the next character even comes in to the UART. A UART at 9600 baud is SLOW, there is a lot of time between characters, So, if the MAIN program starts any processing on an incomplete string, it has to waste time waiting for the next character or end of message terminator. You can do better things with the MAIN program processing power.
Again, in my opinion, a ring buffer is more useful when there is lots of important data coming in FAST, and you have to process every byte for information as soon as possible. Typical use would be a 2 byte A/D value coming in at 115200 baud, and you have to read and average the value quickly. However, reading a 60 character string at 9600 baud, of which you use only 10 bytes of that information, it is a waste of memory space and compute power to store all of that string and later try to extract what you need. Extract what you need as it comes in (by interrupt routine filtering).

So, you have to decide on the style you want to use. For lots of serial data, of which you need all of it and it is coming in rather fast, then a ring or circular buffer makes sense. For a string of data that comes in slow, of which you only need a small portion of it, then a simple capture by the interrupt routine of the critical data into a small string is more efficient. It also depends somewhat on what else the MAIN program is doing and how fast it has to respond to any incoming data. At 9600 baud, the MAIN program has LOTS of time to do other things between incoming characters.

I can't teach you how to "think" like a processor at the 1mS time frames. You have to go over it yourself and think of how fast things are happen inside the code you are writing, and how fast the CPU can process the data, and how fast the data comes into the system in the first place.

 

[camerart]

Yes, it is possible to do TX and RX on a UART without using HSEROUT or HSERIN. Using HSEROPEN just sets up the ports and baud rate generator, you do not have to use his other routines if you don't want to.
It is possible to read a buffer while the interrupt routine is still filling it with characters, as with your "head" and "tail" examples. It all comes down to how much data you are receiving, how fast it is coming in, and whatever your main program is doing with that data. If you need a completed string, you cannot process just a part of it - you need all of it. So why read only a part of the string while it is still coming in?

I find that for smaller strings of data, it is rather pointless for the main program to start reading the input buffer while it is still filling up, as the main program is just wasting time waiting for the rest of the string before it can process the final result. That is, I find no point in processing the input string until it is complete. The processing of the input string by the main program can be done very quickly, before the next character even comes in to the UART. A UART at 9600 baud is SLOW, there is a lot of time between characters, So, if the MAIN program starts any processing on an incomplete string, it has to waste time waiting for the next character or end of message terminator. You can do better things with the MAIN program processing power.
Again, in my opinion, a ring buffer is more useful when there is lots of important data coming in FAST, and you have to process every byte for information as soon as possible. Typical use would be a 2 byte A/D value coming in at 115200 baud, and you have to read and average the value quickly. However, reading a 60 character string at 9600 baud, of which you use only 10 bytes of that information, it is a waste of memory space and compute power to store all of that string and later try to extract what you need. Extract what you need as it comes in (by interrupt routine filtering).

So, you have to decide on the style you want to use. For lots of serial data, of which you need all of it and it is coming in rather fast, then a ring or circular buffer makes sense. For a string of data that comes in slow, of which you only need a small portion of it, then a simple capture by the interrupt routine of the critical data into a small string is more efficient. It also depends somewhat on what else the MAIN program is doing and how fast it has to respond to any incoming data. At 9600 baud, the MAIN program has LOTS of time to do other things between incoming characters.

I can't teach you how to "think" like a processor at the 1mS time frames. You have to go over it yourself and think of how fast things are happen inside the code you are writing, and how fast the CPU can process the data, and how fast the data comes into the system in the first place.

Yes, it is possible to do TX and RX on a UART without using HSEROUT or HSERIN. Using HSEROPEN just sets up the ports and baud rate generator, you do not have to use his other routines if you don't want to.
It is possible to read a buffer while the interrupt routine is still filling it with characters, as with your "head" and "tail" examples. It all comes down to how much data you are receiving, how fast it is coming in, and whatever your main program is doing with that data. If you need a completed string, you cannot process just a part of it - you need all of it. So why read only a part of the string while it is still coming in?
My mistake! I am only saving it in a STRING before READING and PARSING.

I find that for smaller strings of data, it is rather pointless for the main program to start reading the input buffer while it is still filling up, as the main program is just wasting time waiting for the rest of the string before it can process the final result. That is, I find no point in processing the input string until it is complete. The processing of the input string by the main program can be done very quickly, before the next character even comes in to the UART. A UART at 9600 baud is SLOW, there is a lot of time between characters, So, if the MAIN program starts any processing on an incomplete string, it has to waste time waiting for the next character or end of message terminator. You can do better things with the MAIN program processing power.
I am under the impression that each INTERRUPT stores 1x digit. The Main program also stores each digit as it can. I think this is controlled by a FLAG or it skips past it, so no holdup. Once all of the relevant parts of the sentence are IN, it then starts PARSING
Again, in my opinion, a ring buffer is more useful when there is lots of important data coming in FAST, and you have to process every byte for information as soon as possible. Typical use would be a 2 byte A/D value coming in at 115200 baud, and you have to read and average the value quickly. However, reading a 60 character string at 9600 baud, of which you use only 10 bytes of that information, it is a waste of memory space and compute power to store all of that string and later try to extract what you need. Extract what you need as it comes in (by interrupt routine filtering).
App 80% f the DATA collected is useful, and I don't know how fast it will be needed, but as it's a flying machine, then I would like to Err on the fastest method.

So, you have to decide on the style you want to use. For lots of serial data, of which you need all of it and it is coming in rather fast, then a ring or circular buffer makes sense. For a string of data that comes in slow, of which you only need a small portion of it, then a simple capture by the interrupt routine of the critical data into a small string is more efficient. It also depends somewhat on what else the MAIN program is doing and how fast it has to respond to any incoming data. At 9600 baud, the MAIN program has LOTS of time to do other things between incoming characters.
The MAIN program needs to be as fast as possible.

I can't teach you how to "think" like a processor at the 1mS time frames. You have to go over it yourself and think of how fast things are happen inside the code you are writing, and how fast the CPU can process the data, and how fast the data comes into the system in the first place.
I'm slowly getting the idea of timings, but as it's incredibly fast, it is not that easy.
A couple of posts ago, 'J' seemed to suggest the Linear method that has be used for years (I started this because there were some errors), and he does understand Oshonsoft and this program, so I may have to change bak to linear??
C

 

[Pommie]

The only processing you should do in the IRS is to store recieved values in the buffer. Your main code then does the processing of the recieved values. I'm in rural Australia at the moment and away from my computer but could try to write example code later (or tomorrow) if you're still struggling.

Mike.

 

[rjenkinsgb]

The only processing you should do in the IRS is to store recieved values in the buffer.

... And increment the buffer pointer, plus setting it back to the buffer start if it passes the end.

No character comparisons, no changing any other values relating to the buffer.

Everything else is done in the part of the program that reads characters out of the ring buffer.

 

[camerart]

The only processing you should do in the IRS is to store recieved values in the buffer. Your main code then does the processing of the recieved values. I'm in rural Australia at the moment and away from my computer but could try to write example code later (or tomorrow) if you're still struggling.

Mike.

Hi M,
This is how the DATA is being stored at the moment.
CODE would be welcome, if I can't proceed, but as this is an excersise, Hold back on the writing for now, thanks.
C.

 

[camerart]

... And increment the buffer pointer, plus setting it back to the buffer start if it passes the end.

No character comparisons, no changing any other values relating to the buffer.

Everything else is done in the part of the program that reads characters out of the ring buffer.

Hi R,
As #35, thanks.
C

 

[camerart]

Hi,
Up to now, I've thought that the buffer, has a snake that's eating it's tail when it's being stored. I now think that the snake is laid down and doesn't chase anything, but sits there till it is 'eaten' by the next input digit, starting at the head. I'm guessing then, that the buffer needs to have a little time, for the MAIN LOOP to process the DATA, before over writing the DATA.

As it is at the moment: At each INTERRUPT the next digit is put into the buffer [<10 uS ].
Unless there is anyone who thinks differently, then as 'J' and perhaps others have suggested, the MAIN LOOP doesn't need to do anything till the TAIL, has been put into the buffer.

If I understand correctly, the only advantage of a RING BUFFER over a LINEAR is, if the MAIN LOOP was processing the DATA at the same time as the DATA is being written, allowing for a smaller buffer, so I think for me, that the LINEAR is easier to implement.

I'll carry on with a LINEAR BUFFER only being PARSED after the whole sentence has been stored in it. This is how it's been for a year or two. Now I've got to remember what the problem was that started me on this quest.
C.

 

[Pommie]

What is the data exactly? Is it a NMEA sentence?

Mike.

 

[camerart]

What is the data exactly? Is it a NMEA sentence?

Mike.

Hi M,
There are 3x types of sentence, from 3x different sources, selected in the program by an electronic DATASWITCH.
Here are examples: The useful DATA shown in red
$QEIDEG,111,W
$REMOTE,12,20,50,W
$GPRMC,123519,A,4807.038,N,01131.000,W,022.4,084.4,230394,003.1,W*6A?
C.

 

[rjenkinsgb]

You can just use the output part of the ring buffer as the data input for your original receive and parse routine.

Any time the buffer output pointer is different to the input pointer, get the value at the output point and treat it as you originally treated the received serial byte, loading it in to a linear buffer.


Your original problem was the processing or parsing time of a full buffer, if I remember right? It had to be completed between two interrupts.

Adding a small circular buffer ( eg. 8 or 16 bytes) gives you almost the time it would take the circular buffer to fill, to parse and do whatever you want with a complete received string, before timing matters again.

After that is complete and the main loop continues, it can rapidly move the content from the circular buffer and "catch up" with data coming in.