Continue to Site

Welcome to our site!

Electro Tech is an online community (with over 170,000 members) who enjoy talking about and building electronic circuits, projects and gadgets. To participate you need to register. Registration is free. Click here to register now.

  • Welcome to our site! Electro Tech is an online community (with over 170,000 members) who enjoy talking about and building electronic circuits, projects and gadgets. To participate you need to register. Registration is free. Click here to register now.

Decoding and graphing BTc encoded data

Status
Not open for further replies.

vsmGuy

New Member
I am amazed by RB's work, especially the BTc encoder!

I have been meaning to decode and graph BTc encoded data (arrays generated by his encoder) on a COMPUTER.

I have modelled a RC filter using PERL as this allows quick prototyping and, unfortunately, the kind of graphs I get is not exact (maybe similar to my un trained eye, except at extreme ends).

Also unlike the BTc encoder drawn waveform, the waveform I see from my data starts to oscillate between the power rail and ground after some time.

Here is how I model the RC filter :

Code:
V(t) = V_FINAL + (V_INITIAL - V_FINAL) * (e ** (-t / RC ));
where,
VFINAL will be 0V if bit to be sent is low. It will be VCC if bit to be sent is high

The value of RC is the value suggested by RB's application :

R = 2966E
C = 0.22uF

Hence, in my code I have

Code:
my $RC = ( 2966 ) * ( 0.22 / ( 10 ** 6 ));

This is exactly what the subroutine "V" in my code returns:

Code:
my $samplingFreq = 11000; # In Hz
my $samplingPeriod = ( 1 / $samplingFreq ); # In seconds

my $e = 2.7186362391351841722168029758227;

my $RC = ( 2966 ) * ( 0.22 / ( 10 ** 6 ));

my $numberOfSamples = 17464;

# Returns V(t) = V_FINAL + (V_INITIAL - V_FINAL) * (e ** (-t / RC ));
# VFINAL will be 0V if bit to be sent is low. It will be VCC if bit to be sent is high

my $OUTPUT_AMPLITUDE = 5;

sub V
{
    my ( $t, $BIT_TO_BE_SENT,  $V_INITIAL ) = @_;
    
    my $V_FINAL = $OUTPUT_AMPLITUDE * $BIT_TO_BE_SENT;
    
    $t *= -1;
    
    return ( $V_FINAL + (( $V_INITIAL - $V_FINAL ) * ( $e ** ( $t / $RC )))); # Models rise as well as decay
}

I put RB's encoded data in an array called 'sound_data' and I have a loop that iterates through it a byte (element) at a time (outer for loop), extracting a bit for every time period (inverse of the sampling frequency) from the MSB to the LSB (inner for loop) before progressing to the next bit.

This loop writes out the time and amplitude in comma separated format that I later load into Excel and graph the data, and try and match it with the graph shown by RB's Encoder application.

Code:
my $V = 0;

for( my $dx = 0; $dx < @sound_data; ++$dx )
{
    my $t = (8 * $dx) * $samplingPeriod;

    #for(my $bit = 1; $bit <= ( 2 ** 7 ); $bit <<= 1)
    for(my $bit = ( 2 ** 7 ); $bit > 0 ; $bit >>= 1)
    {
        $V = V( $t,  (( $bit & $sound_data[$dx] ) ? 1 : 0 ), $V);
    
        printf "%f, %f\n", $t, $V;
        
        $t += $samplingPeriod;
    }
}

My final target is to create WAV file from the encoded data (arrays of data).

This should be easy to do once I have a 1:1 match of my generated graph with RB's graph. .. the place where I need some help right now :confused:

Whole package up for grabs at :

https://sites.google.com/site/vimalsshankar/btc

I also attached images of "What it should be"(RB's BTC graph) and "What It is" (What Excel shows my data as)

As you can see, my data shows that my filter starts osciallting widly just after a few samples and does not follow RB's BTC waveform as shown as all! :eek:
 

Attachments

  • WhatItIs.png
    WhatItIs.png
    8.8 KB · Views: 791
  • WhatItShouldBe.png
    WhatItShouldBe.png
    20.9 KB · Views: 738
Last edited:
Hi vsmGuy, and thanks for the compliments with my BTc Encoder. :)

Decoding BTc 1bit sound back to 8bit wave is not easy at all. I've thought about it for a lot of years but there is really no easy way, that's why i have never included BTc->Wave decoding in any version of the Encoder. I am very interested in what you are doing and if it can lead to a successful decoding system.

The problem is that the BTc encoding of the wave->1bit data encoding is very lossy. For every sound sample the encoder is generating both a 0 and a 1 bit, (based on the RC curve) then choosing the "predicted" bit that is closest to the original wave. The encoding algoithms are shown on my
Roman Black's BTc 1bit Sound Encoding Algorithms
(BTc Algorithms page)

So each BTc bit will have an error, that may be + or - to the original sample, by quite a large amount. Then the next bit will have another error that will tend to correct the previous error(s), because every new bit chooses the best option (+ or -) to get back to (closest to) the original waveform.

This randomness of the error is extremely hard to reverse engineer being that the + or - for that bit is unknown, as were the bits before it that were also unknown. Also the predictive nature of the algorithm means there is a time issue as the BTc bit is predicted in advance of the new wave bit.

Sorry I don't know PERL so I cant help you much there, but one thing seems an issue is that my algorithm uses the middle part of the 5v (ie the orig wave is reduced to half amplitude and then biased at 2.5v centre, so it goes from 1.25v to 3.75v only). Your first diagram seems to have a 0v-5v swing which would not be possible given that RC time constant and that bitrate. If you change that it should get you back close to the BTc waveshape. Again, check my encoding algorithms on the link above, reproducing the BTc waveform from the BTc data should not be that hard.

I really don't think you can reverse encode the BTc data back to a wave based on a per-sample system for the reasons I stated above. The only way I think it CAN work is to do it the same way your ear does, by a low pass averaging of the total waveform. By averaging each sample with the samples before AND after (remember it's predictive) you should be able to remove most of the PCM noise and get a re-coded (not decoded) wave that is hopefully something like the original wave.

Hope that helps.
 
Thanks a lot Roman for the detailed answer!

Well what I am trying to do is simple really - it should mimic the waveform that your encoder application plays when one clicks on "Play BTC Sound".

I am not trying to do anything else - I need to hear what the sample might sound like out of the uC output pins!

What I want to do is to create offset pointers into sound. For example, say I record :

"Counting up. Now going down"

As you can see, we have a lot of data that is common :

"Count" "ing"

"go" "ing"

So if I can save a pointer to ofsset and lenmgth into "ing" of "Count" "ing", I just need to store "go" and I get "go" "ing" for free!

Something like that :)

I noticed you also have a pointer creation facility in your tool - is this the same?

To summrize - my tool will allow you and anyone else to play the NTc encoded binary output WITHOUT actual hardware :)

Is there something available to do that right now?
 
Hi, re the pointers, yes the new version of BTc Encoder v3.0 handles sound pointers for you, this was a feature people had asked for that allows you to join sounds into one "library" or just to play separate parts of sounds as you said above. It really makes the system useful for makng a PIC talk etc as it can speak any word from within the sound library.

As for playing the BTc encoded 1bit sound, my software already does that (during/after encoding), there's a button for it.

If you want to play other 1bit BTc data as a wave file, you can simulate the PIC hardware playback of the 1bit stream. Sorry there is no tool available that currently does playback simulation (apart from actual playback in real hardware).

If you want to code up a simulator, you can reverse the encoding algorithm. From my algorithm page; here is the procedure for encoding 1bit sound at BTc8;

Code:
variables;
	NewSample = the new audio sample, scaled to 1/4-3/4 Vcc range
	LastVoltage = the cap voltage (audio output) from the last bit
	NewVoltage = the cap voltage (audio output) after we encode this new bit
	NewBit = the newly encoded BTc bit

encoding;
1. generate a 1bit;
	distance = (Vcc - LastVoltage) / 8
	Voltage1 = LastVoltage + distance
2. generate a 0bit;
	distance = (LastVoltage - Gnd) / 8
	Voltage0 = LastVoltage - distance
3. Find which bit is closest to the desired audio sample;
	if Voltage1 is closest to NewSample then; NewBit =1, and NewVoltage = Voltage1
	if Voltage0 is closest to NewSample then; NewBit =0, and NewVoltage = Voltage0
4. Save the cap voltage and repeat for the next audio sample;
	LastVoltage = NewVoltage
	Repeat

So to simulate playback from a PIC to a RC filter, you can basically reverse the encoding algoithm. This will simulate the analog voltage on the cap in the RC filter of the playback hardware. The simulation procedure would look like this;

Code:
variables;
	TheBit = the BTc bit (from your data file)
	LastVoltage = the cap voltage (audio output) from the last bit
	NewVoltage = the cap voltage (audio output) after we simulate this new bit (this is the 0-255 "wave file" output)
	
1. Start at middle of waveform;
	LastVoltage = 128
	NewVoltage = 128

2. Get the next bit;
	If bit==1; NewVoltage = (LastVoltage + ((256 - LastVoltage)/8) )
	If bit==0; NewVoltage = (LastVoltage - (LastVoltage/8) )

3. Now NewVoltage is the "wave" output sample, so save it

4. Keep a copy of the cap voltage
	LastVoltage = NewVoltage

5. Go back to 2, until all bits are simulated.

You should be able to make that a PERL script without too much hassle. The simulation does assume you know what algorithm as used in encoding (in this case was alg1) and assumes you know what the BTc fineness value was (in this case it was BTc8).

Because of the binary nature of the time constant it's extremely easy to simulate the RC filter resonse, in that for every bit (every sample) the voltage on the cap moves 1/8th of the way towards MAX (for a 1 bit) and 1/8th of the way toward ZERO (for a 0 bit). It's 1/8th because it was encoded at BTc8 obviously.

The simulation code above will give you an unsigned 8bit "wave" file with each sample from 0-255, however the waveform will slightly reduced and be closer to the centre of the range since it was originally encoded from a 1/2 amplitude waveform.

Now that simulated waveform will NOT be a perfect reproduction of the original wave due to the lossy nature of 1bit encoding. It will contain quite a lot of PCM noise "aliasing" etc that normally in hardware is reduced by low-pass filtering in the amp and speaker. If you want to play this as a wave file you can low-pass filter this farily easily to get an IDEA of what it would sound like, one technique I used was just to average every wave sample with the sample before it, this smoothes much of the PCM noise and gives an approximation of how the BTc 1bit playback will sound in actual hardware after filtering.
 
Hi, re the pointers, yes the new version of BTc Encoder v3.0 handles sound pointers for you, this was a feature people had asked for that allows you to join sounds into one "library" or just to play separate parts of sounds as you said above. It really makes the system useful for makng a PIC talk etc as it can speak any word from within the sound library.

So you mean so say, that if I have a sample that sounds :

"Counting up. Now going down"

Then I can access "Count", "ting", "up", "go", "down" from my hardware as pointers into a look up table? :confused:

I did click around the set offset options in your application but found it hard to understand.

Had I exported the data to C/ASM after setting the pointers, would have got some different kind of output?

Could you help me out a little bit by giving on overview of this option :)

As for playing the BTc encoded 1bit sound, my software already does that (during/after encoding), there's a button for it.

Yes, but I am trying to figure out how to play data from middle of the array and I have not figure out the pointer option in your application yet :eek:

If you want to play other 1bit BTc data as a wave file, you can simulate the PIC hardware playback of the 1bit stream. Sorry there is no tool available that currently does playback simulation (apart from actual playback in real hardware).

This is exactly what I am trying to do :D

Thanks a lot Mr RB!

If I still don't get the thing to work with all the information, maybe I should dig a well and die inside it :(

Thanks again for sharing this wondeful stuff with us!
 
So you mean so say, that if I have a sample that sounds :

"Counting up. Now going down"

Then I can access "Count", "ting", "up", "go", "down" from my hardware as pointers into a look up table? :confused:
...

Sure, that was the main change I did for BTc Encoder v3.0.

1. Open your wave file that says "Counting up. Now going down"

2. Goto Edit->edit library pointers

3. Enter the sample position (number) for the end of sound0 "Count"
(you can click PLAY to hear the sound for sound0)

4. goto sound1, put in the end point sample number for the end of sound1 "ing" (you can play it to test it like above)

5. keep going until you have all sound pointers allocated.

6. Save as .SL sound library file, this is a wave file that keeps your sound pointers (in a 1024 byte header) as well as the wave data

7. Export the sound as .BTL BTc library file, this is the BTc 1bit sound data to play on your PIC, but also has the same sound pointers (in the same 1024 byte header).

8. now your PIC or AVR can automatically play any of the words because it has the 1bit sound data AND all the sound pointers.

As I said above that was the feature people wanted me to add to my 1bit sound, so that there can be multiple counds in the one file (up to 256 sounds) and the file also contains the sound pointers so your PIC or AVR playback hardware can just select "play sound0" or "play sound4" etc.

There is more about the sound pointers and sound library files on this page;
Official homepage of TalkBotBrain.com
where you can also get cheap ready-made hardware to play sounds with a PIC and 1 Megabit eeprom included.

Please let me know if/when you get the simulator coded, if you like I can put a link on my webpage so other people using 1bit sound can make use of your work too. :)
 
Thanks for the walkthrough RB!

I feel like a fool though now.

Well, the thing is that I got the simulation working fine. The code generates the wav file and I get to hear the sound as I hear it from the uC without a LPF.

My code is unpolished though - only meant to simulate BTC8/1bit arrays .. will this limitation be OK?

I can work with you though to make sure it can simulate any of your algos - that would be much better :)

The nice thing using my Perl code however is that it will run on Linux or any OS that has a Perl interpreter - guess that increases the reach of your code :)

Will update this site with links, code and samples so you can get the code to update to your site and others can refer back when they want.

I have a few questions for you though :

1. The .BTL BTc library file that contains the pointers - this seems to be in a binary format? (Is the layout documented here : https://www.talkbotbrain.com/index.htm#libformat ? )

In that case it seems that the sample length pointed to by each pointer need to be multiples of 32 samples (and the last sample can be smaller?)

For example, I am thinking of simulating DTMF tones that can be as small as 2 - 3 bytes!

Can I have the pointers pointing to these samples? (or will extra space be consumed in padding the 2 -3 bytes data to 4 bytes to align it to sample/chunk size?)

2. Do we not simply bit bang the array elements out the uC pin? Then why did not my original equation fail to simulate the waveforms?

Although now the graphs I do have a 100% match with the ones generated by your application, I am still wondering why the original equation lifted striaght out of a college textbook fails to reproduce the final waveform?

Of course my model is wrong as evident from the fact that now I get exact matches after listening to you, but the routine in the last but one post you wrote is more of a digital approximation than an analog one?
 
Last edited:
Great! Glad you got the simulator working, as i said it would be easy to code up that's the beauty of BTc the sheer simplicity of it.

The help file .HLP included with BTc Encoder has a lot of info on sound pointers and the library header format.

Re sample size per-sound in the BTc library file it is limited to 32 samples (4 bytes) as that is the smallest sound size. That limitation was chosen to allow a 2byte address (in PIC playback) to be able to reference 2 megasamples of sound. A 32 sample resolution is fine enough for most sounds considering that's like a millisecond or so.

I'm not sure why your orignal PERL app didn't work, but it was trying to do the RC filter modelling the hard way using floating point. The whole benefit of my BTc system is that it only needs BINARY divisions and a few add/subtracts etc, its extremely fast and bulletproof.

Please PM me when you have updated your page, and I will put a link on my BTc page to your page so people can access your work, or I can even put your PERL files on my page if you think your web page won't be there permanently as that might be better. And we should probably continue any development via PM's instead of on the open forum as it might irritate the moderators doing it in this thread. :)
 
Hi Roman!

I would prefer you host the files on your site as that will ensure everything is available at one place - I just cannot stop emphasizing how exciting your project is!

Do PM me your mail address so that we can correspond without bloating this thread (You rightly pointed out that the conversation has been only between us) but I will help you post latest updates to your site.

I do have a question though - the model of the complete decoding system that constitutes a uC and a RC filter with us taking the output across the capacitor does follow the exponential formulae with the voltage across the uC pin feeding the capacitor being either 5v or 0v at any sampling period - does it not?

Could you please, in that case help me solve the large discrepancy between the waveforms that I am getting between the BTc waveform and the odd waveform I am getting from my script?

:-D
 
...
I do have a question though - the model of the complete decoding system that constitutes a uC and a RC filter with us taking the output across the capacitor does follow the exponential formulae with the voltage across the uC pin feeding the capacitor being either 5v or 0v at any sampling period - does it not?

For the 1bit algorithm (that we have been talking about) that is correct.

Could you please, in that case help me solve the large discrepancy between the waveforms that I am getting between the BTc waveform and the odd waveform I am getting from my script?

Just use the procedure i suggested above and convert that to PERL. Which I thought you already did? You said it was working ok to reproduce the graph waveform 100%.

If you like, PM me your PERL script for the procedure I specified above and i'll try to help with it. I'm not going to debug your original PERL script because it didn't work and it was a terrible way of doing the task compared to my procedure.
 
Last edited:
If you like, PM me your PERL script for the procedure I specified above and i'll try to help with it. I'm not going to debug your original PERL script because it didn't work and it was a terrible way of doing the task compared to my procedure.

:D

I was talking about the original script. The one based on your last discussion works 100%.

Oh well .. I will mail you a finished version of the scrip and some supporting file soon.. gove me a day or to to make sure it is presentable :eek:

Thanks again for all the support (and any insights into why the original script did not work at all - does it not model the BTc decoding process as in real life?)
 
Hi RB!

Latest files available here https://sites.google.com/site/vimalsshankar/btc

3.zip : The Perl script that transforms the BTc encoded output to WAV
18Bit.zip : BTc 8, 1 bit encoded output of 11k8.wav availble on the site

I would like to add a functionality to your application :

The BTc encoder application should output the BTc encoded data in true C array "format" : a list of bytes, ( helps to run in a C compiler as well as PERL etc )

If you don't have time to do this yourself, I would be more than happy to get this done!

:)
 
Cool. I'll check out your latest files. :)

Umm, my software already does export the BTc sound data as C bytes. See File->Export PIC MikroC file .C But if you want to add some different or extra functionality then sure please go ahead.

I'll PM you after I check your files. :)
 
:D

Actually, the C format you export to is like this :

Code:
void sound_data1()    org (1 * 256) 
{ 
   asm retlw 0x56 ; 
   asm retlw 0xAB ; 
   asm retlw 0x2B ;
...

I want it like this :

Code:
const char sound_data1[] =
{ 
   0x56, 0xAB, 0x2B...
};
 
I finally got around to putting your PERL stuff on my BTc Encoder web page;
BTc Sound Encoder v3.0 - free sound software

This is so cool :D

Sorry for the delay there I forgot all about it :D and thank you for the work you did on that. :)

No need to apologize at all. Thanks again for sharing so many nice things with the world :)

I am really interested in the TalkBotBrain Project. If you need some work done for which you are short on time, please let me know.
 
Status
Not open for further replies.

Latest threads

New Articles From Microcontroller Tips

Back
Top