Real-time data logging of joystick: techniques, 300Bytes/s

[astronomerroyal]

Dear all,

I'm using a joystick to control some stepper motors, and I'd like to be able to record the joystick so that I can repeat motions exactly. I'd like to know how best to save this real-time stream of data.

(Click to enlarge?)


I'm using 18F4620 @20MHz.

1. The joystick (two axis analogue, connected to 10-bit AD pins on MCU) and a couple of slider pots are sampled.
2. This generates a block >10 bytes in size.
3. Doing this 30 times/sec = 300 bytes/s.
4. Recording for, say, 30s means a total of about 10KBytes.

I'm sure this is fairly standard stuff, but I've searched online for a discussion of real-time data logging MCU techniques, and not found anything really satisfying or tailored to PIC18F peripherals etc.

Some issues:
1) On 18F PICs, variables in RAM have a nominal size limit of 256 bytes. Although linker script can be modified for larger vars, this is not doable when a large 10KB block is required.
2) Internal EEPROM of 18F4620 (256bytes) is not large enough.

Options considered:
1) External EEPROM I2C. Writing each block of 10Bytes takes about 1ms on 100kHz I2C bus.
2) External EEPROM SPI. Much faster than I2C.
3) Buffer largish blocks of data in RAM and write whole pages to external EEPROM, either I2C or SPI. Presumably this is more efficient than 1). or 2).
4) SRAM? No idea. Volatile is okay.

The process has to be reversible, as I need to eventually read this data for real-time playback.

Words from the wise?
Millions of thanks in advances,
Tom.

 

[nsaspook]

Dear all,

I'm using a joystick to control some stepper motors, and I'd like to be able to record the joystick so that I can repeat motions exactly. I'd like to know how best to save this real-time stream of data.

(Click to enlarge?)
View attachment 41968

I'm using 18F4620 @20MHz.

1. The joystick (two axis analogue, connected to 10-bit AD pins on MCU) and a couple of slider pots are sampled 30 times/sec.
2. This generates a block >10 bytes in size.
3. Doing this 30 times/sec = 300 bytes/s.
4. Recording for, say, 30s means a total of about 10KBytes.

I'm sure this is fairly standard stuff, but I've searched online for a discussion of real-time data logging MCU techniques, and not found anything really satisfying or tailored to PIC18F peripherals etc.

Some issues:
1) On 18F PICs, variables in RAM have a nominal size limit of 256 bytes. Although linker script can be modified for larger vars, this is not doable when a large 10KB block is required.
2) Internal EEPROM of 18F4620 (256bytes) is not large enough.

Options considered:
1) External EEPROM I2C. Writing each block of 10Bytes takes about 1ms on 100kHz I2C bus.
2) External EEPROM SPI. Much faster than I2C.
3) Buffer largish blocks of data in RAM and write whole pages to external EEPROM, either I2C or SPI. Presumably this is more efficient than 1). or 2).
4) SRAM? No idea. Volatile is okay.

The process has to be reversible, as I need to eventually read this data for real-time playback.

Words from the wise?
Millions of thanks in advances,
Tom.

Data compression.
Basic example...
Only save the differences in samples and time/counts between the changes(if not in constant motion). You need a timer(adc routine) and counter variable (30*time of data in seconds without overflow ).

data start sequence mark
0: 0,count
0 to 233: +233,count
233 to 100: -133,count
etc...
data stop sequence mark

 

[Electronworks]

Personally I would use an SPI addressed E2 memory. The SPI bus is fast and much easier to diagnose than I2C. You will need 6ms for the E2 memory to program itself. Do you really need 300samples/s as this is pushing it with E2. If you can bring this speed down to 150sps, you might be in with a chance...

Failing that, you can get battery backed SRAM (see Dallas semiconductor), or buy a normal RAM and make sure you do not disconnect the power.... or download to battery backed SRAM then download that data (at your leisure) to E2.

Ah, the options are endless....

 

[Nigel Goodwin]

Why save the joystick data?, why not just save the data sent to the stepper motor?.

 

[astronomerroyal]

Data compression.

I generate 10bytes every time I read the ADCs. Only very simplistic motions will allow sufficient compression to even approach a 30s recording using only 256bytes of ram or internal EEPROM i.e. I'd only be able to change the stepper speeds once per second, on average - that won't yield/guarantee real-time playback.

Personally I would use an SPI addressed E2 memory. The SPI bus is fast and much easier to diagnose than I2C. You will need 6ms for the E2 memory to program itself. Do you really need 300samples/s as this is pushing it with E2. If you can bring this speed down to 150sps, you might be in with a chance...

Ah, the options are endless....

By E2 do you mean EEPROM? I do like the idea of the brute-force shoveling of data onto an EEPROM. I'm only generating a packet of data 30 times a second, not 300/s. That's 33ms to deposit each packet onto the EEPROM. I believe this could even be done on a slow I2C bus, but I'll have to give it a shot - I've got working I2C and SPI EEPROM code somewhere.

Why save the joystick data?, why not just save the data sent to the stepper motor?.

What do you mean? Record the time of each pulse that gets sent to the stepper driver?

Thanks for the suggestions. I was wondering if there was a an 'obvious best solution' that I wasn't aware of, but it increasingly feels like the EEPROM is the way to go. I believe a 512kb EEPROM can store over 3 minutes of uncompressed data recording - plenty.

 

[Pommie]

I would just use something like a 24xx1025 which is 128k of EEPROM on I²C. However, as already mentioned, it's the motor positions you need to store. How will you know the start position of the steppers?

Mike.

 

[astronomerroyal]

I would just use something like a 24xx1025 which is 128k of EEPROM on I²C. However, as already mentioned, it's the motor positions you need to store. How will you know the start position of the steppers?

Mike.

Pleased to hear it because the lonesome chip in the picture above is a 24LC512. Okay, I see what Nigel was getting at. It's a pan-tilt video mount for tracking moving things - since the timebase is a regular 1/30s tick, speed and location are trivially related - I just need to ensure a record of one or the other is stored exactly. Gadget keeps an absolute record of position by counting ticks - there's a homing function based upon this to guarantee positional repeatability.

When the user is in control the function ReadJoystick() is run every 1/30th of a second. This gives the user nice real-time control. When in playback mode everything is the identical except that it runs ReadMemory() instead. It's sort of a like a ghost joystick. The difference in terms of code is just a single flag. Works really beautifully - I hardly ever nail an algorithm like this - guaranteeing that the motion played back is identical to whatever the user inputs. Only issue is memory, but I think the EEPROM is the fix.

By the way, you GLCD Unicorn thread was an absolute Godsend. Full of clean ideas. I added some things, but for the most part it's your code running on the GLCD above.


Thanks,
Tom.

 

[nsaspook]

Pleased to hear it because the lonesome chip in the picture above is a 24LC512. Okay, I see what Nigel was getting at. It's a pan-tilt video mount for tracking moving things - since the timebase is a regular 1/30s tick, speed and location are trivially related - I just need to ensure a record of one or the other is stored exactly. Gadget keeps an absolute record of position by counting ticks - there's a homing function based upon this to guarantee positional repeatability.

When the user is in control the function ReadJoystick() is run every 1/30th of a second. This gives the user nice real-time control. When in playback mode everything is the identical except that it runs ReadMemory() instead. It's sort of a like a ghost joystick. The difference in terms of code is just a single flag. Works really beautifully - I hardly ever nail an algorithm like this - guaranteeing that the motion played back is identical to whatever the user inputs. Only issue is memory, but I think the EEPROM is the fix.

By the way, you GLCD Unicorn thread was an absolute Godsend. Full of clean ideas. I added some things, but for the most part it's your code running on the GLCD above.


Thanks,
Tom.

One other option would be a SD/MMC card interface. SD/MMC Mini Board if needed https://www.futurlec.com/Mini_Logic.shtml
Microchip Technology User Forums

 

[astronomerroyal]

One other option would be a SD/MMC card interface. SD/MMC Mini Board if needed Logic Converter Mini Board
Microchip Technology User Forums

Thanks. This might be a good excuse to try that too - I have de Lucio's 'Flying the PIC24' book which makes it look straightforward.

 

[astronomerroyal]

Well, just to follow-up; I implemented the I2C EEPROM, 24LC512 @ 100kHz and it's working wonderfully. I'm writing small 10byte packets, 30 times a second - and reading them back at 30/s - without any problems. So it seems to be a good solution.

Thanks again,
Tom.

 

[Pommie]

Congratulations. It's always good to hear of a successful outcome.

Mike.

 

[DirtyLude]

Why are you limited to a 100Khz I2C bus? Why can't you up it to 400Khz?

 

[astronomerroyal]

I used 100kHz only to judge where the limitations lay. I could certainly ramp it up to 400kHz now.

For visual diagnostic purposes I had an 470Ohm+LED on the open-drain I2C bus, and the ~10mA current apparently killed the I2C hardware in the 18F4620. I don't know if it was the magnitude of the current or its rapid on/off switching that did the damage. Has never been a problem in the past. Seems okay with a 1kOhm+LED - should I be using some sort of transistor in there if I want an LED indicator light directly on the I2C bus?