Shifting large amounts of data around

[bh00]

Hi,

I'm at the research stages of a project which will basically merge multiple DMX streams on a highest-takes-precedence basis. I will have 4 inputs and one output, and for each of the 512 channels it will choose whichever of the 4 inputs has the highest value for this channel and then transmit that particular one.

The specifications for DMX are "Data are transmitted serially at 250 kbit/s and is grouped into packets of up to 513 bytes, called 'slots' in DMX512-A. Data are sent with 1 start bit and 2 stop bits, LSB first. The start of a packet is signified by a break of at least 88 uS, extended from 44 uS in the 1986 standard, and when receivers detect the break they reset their receiving code. Then up to 513 bytes are sent. The first byte is always the "Start code" byte.".

I envisage I'll have to have a micro (PIC?) running with a high clock (20MHz?) which will read in the serial data from all 4 ports and store all of it in a large EEPROM (I'm guessing parallel memory would be more suited to this task). Then another PIC can sit and read the memory as well (by piggybacking the data lines of the memory) and then do the calculations and then transmit the DMX.

However I'm just wondering if this is going to be feasable. At the high transfer rates with the large amounts of data will PIC chips/standard memory be able to keep up or is it easily going to handle it? Anyone got any tips on how else I could achieve this?

Cheers

 

[3v0]

I do not know what DMX is.

How much memory is needed for buffering?

Whe use EEPROM to buffer a data stream?

Shared memory is a bit on the ungly side.
How fast would a single processor have to be to handle the load?

 

[Sceadwian]

Use cheaper pics and dedicated one chip for each input streams and one for the output stream, that will give you all the time you need to process, and since each pic only has to buffer one stream and the extra pics add extra memory you may be able to do it. A simple SPI bus with a basic byte sized protocol so the pics can communicate who should have precidence would connect all the pics. At the very least you'd have to use two pics for receive because you'd need hardware uarts. and I don't know of any pics that have more than 2 (someone else chime in)

 

[Pommie]

Four channels at 250Kb/s = 1Mb/s. Your not going to manage with a 20MHz pic. Also, EEPROM is very slow to write to. This looks more like an application for a DSP rather than a Pic.

Mike.

 

[Sceadwian]

DMX is 250 kbit/s not Kbyte/s Pommie. That's only 125 Kb/s the notion difference is subtle by 8 times slower =)
I just thought of it but this might be more suited to an AVR than a PIC. Most PIC's divide their running clock by 4 to get their system clock. AVR's run directly at their clock speed. I've heard some PICs have internal PLL's but I've always found it a nice feature to have the system clock and the driven clock be the same.

 

[Pommie]

At 1Mbit/s and a 20MHz clock you have 5 clock cycles per bit. As there is no PIC with 4 serial ports and doing it in software is out of the question then I suggest PICs (@20MHz) are not a viable solution. BTW, I recently tried to buffer a 19200baud serial stream in EEPROM and it couldn't manage it.

DSPics and the 24 series may be able to handle this. I have no experience of these.

Mike

 

[Sceadwian]

Sure they're viable, and well suited for it. You just use more than one MCU and an SPI network to link them together, then you have four input buffers (each MCU can handle it's own DMX stream using only SRAM) and one output buffer which communicates via SPI protocol to it's network a couple bytes here and there to determine who has precedence, then the master transmitter signals to the appropriate slave receiver the go ahead to empty it's buffer and the master transmit the output stream. Just because he's dealing with 4 input's at 250 kbits doesn't mean it's a 1meg stream. The bandwidth is still just 250kbits it's just being filtered and prioritized. eeprom will definitely not be fast enough it's for storing seldom used variables and settings anyways, not mean to be used for buffering, it would fail very quickly anyways.

At 1Mbit, you have 20 cycles per bit on an AVR btw.. That's why I suggested them, they are far superior to PIC's in the I/O department.

 

[3v0]

The processor per stream is a good way to go but it will cost 4x the memory. To ensure that any stream can buffer to a given depth all streams/processors have to have that amount of memory

Each processor could transmit its own packets as needed. Wire the outputs together and tristate when inactive. Use 2 USART chips and there would be no bit banging. The only traffic on the SPI buss would be arbitration.

 

[hjames]

Doing 4 simultaneous 250kbit data receives at the same time without hardware assistance is a bit tricky. Doing a single 250kbit async means that you need to find the bit start to within say 1/10 of a bit time = 400nsec, which is barely doable in software.

If use use a clock monster like the scenix, it becomes reasonable, except for the fact that the SX family doesn't have anywhere near enough RAM space.

I would recommend either a system which has multiple hardware UARTs (either discrete, or multiple processor) - on a parallel bus, or a more practical CPLD/FPGA based approach. You can dump this entire thing into a $15 FPGA, add a hundred or so lines of code, and be done with it.

BTW, multiport memory isn't required, just have the memory run at channels*max data rate and interleave everything. Also, the trickiest thing is probably trying to get the timing of the signals - how is the output synchronized versus the inputs - what sort of delays are allowed - how out of sync are the signals going to get. The main specification that needs to be written down is the behaviour of the output signal when the input signals are out of sync with each other.

 

[philba]

I agree with hjames. I'd go for uarts with a built fifo, I think you can get dual uarts. Interleaving is a fine technique. Also, I would not use serial busses but stay parallel (probably byte width). In fact, you may not need to route the data through the pic but rather just use it as a traffic cop (detecting packet start/stop, priority and output control). Build a little sequencer for the data from UART to the ram and data output. I guess that takes you a long way towards a FPGA or similar design.

Interesting design problem.

I'd invest in a logic analyzer for this project.

 

[Sceadwian]

An AVR running at 16mhz has a clock length of 62.5ns's. It could do two channels in software, but only because the baud rate is fixed on all ports though I wouldn't want to be the one to code it. Seeing as how the baud rate is fixed and easily divisible by common MCU crystals there should be no real sync issues, as long as the remote clock accuracy can be trusted. As long as you pick an AVR or AVR's with enough total memory for the four/five buffers you should be fine. I don't really see this as being that complicated. The OP would have to more clearly state exactly how the packets need to be prioritized and what possible orders they may come in and how many packets need to be buffered. If it's just one packet each from each stream and then they get sent out in a prioritized manner the entire project could be done fairly easily.

 

[bh00]

Hi guys,

It seems like a multiple chip with dual UART approach might be a lot easier to do than trying to get it all onto one chip. DMX is being constantly transmitted/received - not just when the values change. But this means that between PICs data could just be updated when it actually changes which could reduce transfer quite a bit unless all inputs are simultaneously changing all the channels. I would imagine I need another PIC using its UART transmit to generate the DMX data as this would need to be transmitted without time lagging or anything and would need to be transmitted reliably. **broken link removed** seems to indicate its possible to transmit 256 of the 512 channels quite easily with a standard PIC at 8MHz so I'll probably have a good look at that code.
About the prioritisation thing - basically each input will send their value for channel 1, then channel 2, etc. It will take the highest value for each channel from the inputs and then output that one. So input 1 might have 255 for channels 1 and 2 which would get priority over input 2 if that was only having 128 for the two channels. However the inputs won't be syncronised and would likely arrive at different times, hence the need for the buffering.

Edit: One limitation I think I can spot is that the PICs don't seem to have over 368B of RAM. I'm guessing that to buffer it through the PIC you'd need a byte of RAM per channel and therefore it won't fit all the 512 channels in? So I need some RAM with at least 512Bytes?

Hope that helps clear a few bits up.

 

[bh00]

Hmm I'm having difficulty in working out how to store the data as its being read/retransmitted. Anyone know of the kind of RAM I need? I'm guessing it needs to be parallel as that will help make it a lot faster. Some sort of fast storage with many read/write cycles. Looking for about 1-4Kb chips really.

Cheers

 

[Nigel Goodwin]

Look for old static RAM - 2Kx8 CMOS ones used to be common, and you can get larger ones now - not cheap though, static RAM is far more expensive than dynamic.

 

[philba]

You don't say where you are but electronic goldmine (usa) has the hm6116 (2Kx8) for 3.49 I don't know the speed (100/150/200 nS) but it will probably be good enough.

edit: jameco has quite a few and if you don't mind SMDs, they are pretty cheap (1.99 for 2Kx8). the site is really really really slow today. Did I mention they are slow?

 

[Oznog]

EEPROM is not only slow, but it also has limited write endurance. Writing at high speeds for long periods would yield a short operational life.

This does sound like some very difficult number crunching. I would suggest that you look at the 33F parts, which are FAR more powerful. Not only do they run at 40 MIPs as opposed to 10 MIPs of the 16F/18F, but they're 16-bit cores and most code runs far more efficiently (requires fewer cycles to perform a task). But it's hard to estimate whether even this will meet your requirements.

 

[Sceadwian]

A lot of the Atmega devices from Atmel have 4k of memory. That's enough to buffer 7 DMX sized packets and still have 500 or so bytes for program use. The Atmega64 comes to mind. Two UARTS SPI and 4k of memory, two of those using the SPI bus should be able to handle the task at 16mhz.

 

[bh00]

I've had a word with a few colleagues of mine who know a bit more about this stuff - they seem to suggest a PIC18F252(0) on each input with another polling it for data and generating the output would be sufficient. I might try and get a couple to experiment with.

 

[philba]

The only thing I would caution you about is that debugging multiprocessor designs can be tricky. I definitely suggest a logic analyzer...

 

[Sceadwian]

In order to prioritize all the inputs all the data is going to have to end up on one chip, so one chip per input stream would be a bad idea, but two dual UART chips with a high speed SPI bus to share the UARTS would be ideal.