Oscilloscope trace of sound signal

[Andrew T.]

How fragile is this AudioSystem code?

'AudioTrace' aims to draw an oscilloscope style trace
of the sounds currently being played through the PC's
audio system, but it is based on the assumption that it
is getting a stereo, 16 bit signal.

E.G. the command line output on my system is..
Format: PCM_SIGNED 44100.0 Hz, 16 bit,
stereo, 4 bytes/frame, little-endian

You can browse the files here..
http://www.physci.org/test/oscilloscope/

It contains the source code (an SSCCE, of course), as
well as a launchable jar file of the same (for those
even more pushed for time!).

In subdirectory 'screenshots' you can find typical
output of bird song and Veruca Salt's 'Only You Know',
as well as a JPEG of the intro dialog.

In 'sounds' I have included the two samples from which
the birdsong screenshots were grabbed. (I do not have
any distribution rights for the music.)

Do I need to worry about other formats of sound?

What output do you get in the JOptionPane/command line?

Does the screen output look like an oscilloscope trace
from an harmonic analog signal?

( Note, the code will become an open source screensaver
as part of the saverbeans project, and can currently be
downloaded from URL above. BTW - the 'meaty part' is in
the painComponent() method of AudioPlotPanel, a default
access class included in AudioTrace.java )

AndrewT.

 

[Chris Uppal]

'AudioTrace' aims to draw an oscilloscope style trace
of the sounds currently being played through the PC's
audio system, but it is based on the assumption that it
is getting a stereo, 16 bit signal.

E.G. the command line output on my system is..
Format: PCM_SIGNED 44100.0 Hz, 16 bit,
stereo, 4 bytes/frame, little-endian

Bad news from here, I'm afraid. I've tried it on two machines, one a WinXP
laptop and the other a Win2K deskside box. In both cases the printed output is
just as in your example, but the display doesn't reflect the sounds that are
playing. On one machine the display is just a flat line, on the other there is
a certain amount of jitter which I /guess/ is quiet noise from whatever device
it is actually listening to.

I tried to find something that would send sound to whatever that device is (or
might be), but all the sources on this box (CDs DVDs mp3s midi files) all seem
to end up being "rendered" via what Windows calls "Wave". I have only one
physical sound card in either machine.

Let me know if you want me to run any more diagnostics.

-- chris

 

[Chris Uppal]

I tried to find something that would send sound to whatever that device
is

Ah, found it. It's listening to the microphone input. Maybe I misunderstood
what the program was intended to do.

-- chris

 

[Andrew T.]

Ah, found it. It's listening to the microphone input. Maybe I misunderstood
what the program was intended to do.

You did not misunderstand, I stuffed up.

I had been lulled into a false sense of security by
my own system, in which both lines found seemed
to be streaming the exact same data.

If you would be so kind, try compiling the source* with..
static boolean allLines = true;
...to get a trace of every line.

My original idea was to display all lines to the user
and ask them to 'choose the interesting looking one',
but the seemingly identical traces on my own system
caused me to recompile it false and forget about it.

* I will upload a new version of the jar that displays all
lines, but I will not be able to upload it till tomorrow.

( Thanks for your attention thus far.

Andrew T.

 

[Andrew T.]

....
* I will upload a new version of the jar that displays all
lines, but I will not be able to upload it till tomorrow.

Or maybe tonight..
http://www.physci.org/test/oscilloscope/audiotraces.jar

The new jar is user configurable to show all lines
found (the default) and change the no. of frames
sampled (default 256, like before).

I also added a green border (shrugs).

The new source has replaced the old, available here..
http://www.physci.org/test/oscilloscope/

Andrew T.

 

[Chris Uppal]

If you would be so kind, try compiling the source* with..
static boolean allLines = true;
..to get a trace of every line.

Done that. The output now is:

Format: PCM_SIGNED 44100.0 Hz, 16 bit, stereo, 4 bytes/frame, little-endian
Format: PCM_SIGNED 44100.0 Hz, 16 bit, stereo, 4 bytes/frame, little-endian

I added some extra tracing too, and it appears that the first line corresponds
to some sort of virtual sound device maintained by Windows which is an alias
for the second. Thge second is the real sound device in this laptop. I
remembered that I had a USB headset and plugged that in too (which
automatically adds a new hardware sound device to Windows and makes the virtual
device point to it), and the output changed to:

Format: PCM_SIGNED 44100.0 Hz, 16 bit, stereo, 4 bytes/frame, little-endian
Format: PCM_SIGNED 44100.0 Hz, 16 bit, stereo, 4 bytes/frame, little-endian
Format: PCM_SIGNED 44100.0 Hz, 16 bit, stereo, 4 bytes/frame, little-endian

In both cases the top two lines produced similar (but not actually identical)
traces -- clearly showing the same sounds, but possibly subject to some sort of
timing differences. Which may in part be because the program pushes the
machine to 100% CPU so maybe the software in the virtual device can't quite
keep up.

In both cases the only lines displayed were connected to microphones (two
different mics in the second test).

One more test. Unplugging the headset (and so reverting to the original 1
virtual + 1 real device), and activated the recording control aspect of the
Windows (ultra confusing) volume control applet. It seems that the recording
control suggests 4 possible sources of input:
Mono Mic
Sterio Out
Microphone
Telphony
I tried all of them, and in each case ended up with the same format as above
(even the "mono" mic was stereo). Selecting the "Stereo out" option seems to
connect the input to the current output (why?) and -- at last -- your program
showed the current sound output. Unfortunately this feature seems to be in the
device itself rather than software, because the headset driver only offers the
"Microphone" option.

I'd try this on a Linux box too, but my Linux installation runs in VMWare over
a Windows host OS. And fathoming what's going on when we have Java's sound
system sitting on top of the Linux sound system (whatever /that/ is), on top of
VMWare's peculiar hackery, on top of Windows wierdness, would undoubtedly make
my brain explode...

-- chris

 

[Oliver Wong]

Selecting the "Stereo out" option seems to
connect the input to the current output (why?) and -- at last -- your
program
showed the current sound output.

This is sort of soundcard specific, but most cards have a similar
option. They are alternatively called "Stereo Mixer", "Wave Mixer", and I've
even seen "What U Hear (tm)" on a SounbBlaster Live!(tm).

Sometimes you do indeed want to record "What U Hear(tm)", e.g. to record
a voice chat with someone. It's better than holding the microphone up close
to your speakers.

Other times, you want to record what you speak into a microphone, and
not record any of the sound that may be coming from your computer. So you'd
pick "Microphone".

- Oliver

 

[Andrew T.]

Thanks for the comprehensive reply. I'll have a closer look off-line.

This is sort of soundcard specific, but most cards have a similar
option. ...

Thanks for the extra info.

Still hoping to get test results from more boxes.
The most important questions are..

That URL for the quick launch jar is..

Andrew T.

 

[Andrew T.]

Chris Uppal wrote:

I tried all of them, and in each case ended up with the same format as above
(even the "mono" mic was stereo). Selecting the "Stereo out" option seems to
connect the input to the current output (why?) and -- at last -- your program
showed the current sound output.

I missed that bit in my original reading, and have been working
under the assumption that the problem was signal levels..
(shrugs) in any case, I made some significant changes based on
the results of a second PC that had 'amplification issues' ..perhaps
this version might work better for you.

(It is a worry you had to go that deeply into system config. to
see anything though..)

My message (prepared off-line) is..
Current traces calculate the audio level by looking at
the large order byte, ignoring the small order byte
completely. *

There are some significant changes to the audio trace
code, mostly based on the fact that a third machine
displayed results very similar to what you described.

Most significant in the changes is that the visual
'gain' of the signal can be configured,
http://www.physci.org/test/oscilloscope/screenshots/sinetone.png
the screnshot shows a gain of 1 as it displays the signal
generated by the 'Audio Tone' jar to it's left.

Audio Tone is a jar here,
http://www.physci.org/test/oscilloscope/tone.jar
it generates a more predictable signal - a simple
sine tone with a harmonic (if you want).

The default volume is 20% (for good reason), but for these
screenshots I ramped it up to 90% volume.

I suppose you, Chris can especially appreciate how different
this will appear at gain 1 on your system. (Having seen it
on the 3rd PC, I can vouch it is - quite oddly boring).

I am guessing you'll need to push the gain beyond 7
(the default is 5) before you see a trace ranging this big,
and then it will be very blocky. *

Second, now there is an option to DisplayEachByte** of the four
bytes in each audio frame.
http://www.physci.org/test/oscilloscope/screenshots/sinetoneallbytes-a.png
The size of the small order bytes
has been knocked down to 1/8 the display height at 'gain 1',
by gain 8 it should be clipping.
http://www.physci.org/test/oscilloscope/screenshots/sinetoneallbytes-b.png
Here's a screenshot of 'all lines' showing a gain of 1 and seven
in the separate trace panels.

** The reason I added this option is because I cannot for the life
of me figure how to combine these bytes with the larger order
bytes in any sensible way.

* Which has become especially important since most of
the audio trace detail of two of the three PC's tested,
lies in those small order bytes.

Increasing the gain can amplify the visual trace to the
point of doing something interesting, but I need the small
bytes to give the trace any detail.

BTW
- You can click the trace(s) to toggle the gain
through 15 and back to 1.
- All the screenshots were done with a fade rate
of 100% (clear old traces) in order to simplify the image,
the default is a slowish fade for each trace - much prettier.
- the byte conversion has been shifted to the
'frameToSignedDoubles' method (bottom of AudioTrace source)
- Tooltips galore - most notably on the entire Visual Gain
and All Bytes panels.
- The default settings are visually complex. To simplify the
trace, flip the default values of the check boxes and increase
fade rate to 100%.
- All files available from
http://www.physci.org/test/oscilloscope/

Andrew T.

 

[Andrew T.]

Andrew T. wrote:
.....

Second, now there is an option to DisplayEachByte** of the four
bytes in each audio frame.
http://www.physci.org/test/oscilloscope/screenshots/sinetoneallbytes-a.png

Oops! Visitors cannot directly access the screenshots.
Go through http://www.physci.org/test/oscilloscope/screenshots/
and they should be viewable by clicking the links of the directory
listing.

Andrew T.

 

[Chris Uppal]

I missed that bit in my original reading, and have been working
under the assumption that the problem was signal levels..
(shrugs) in any case, I made some significant changes based on
the results of a second PC that had 'amplification issues' ..perhaps
this version might work better for you.

It's pretty and rather fun, but I'm afraid the basic problem is still there.
On my the machine program doesn't read output audio, only input (mic, etc). I
can still tell the audio device driver to connect the audio input to its audio
output, but that's the only way to capture audio output. If I wind the gain
up then I just see random noise.

Second, now there is an option to DisplayEachByte** of the four
bytes in each audio frame.

** The reason I added this option is because I cannot for the life
of me figure how to combine these bytes with the larger order
bytes in any sensible way.

I modified the last part of frameToSignedDoubles(), the bit after the diagram,
to read:

=================
if (displayEachByte) {
// transform this 'small order' bytes to doubles
// ranging from -1/8 to 1/8
d[0] = b[1]/256.0;
d[1] = b[3]/256.0;
d[2] = b[0]/(256.0*smallByteDampingFactor);
d[3] = b[2]/(256.0*smallByteDampingFactor);
}
else {
for (int cc = 0; cc < channels; cc++) {
d[cc] = (b[cc*2+1]*256 + b[cc*2])/32678.0;
}
}

return d;
=================

and it seems to work correctly. Of course that's stll hard-wiring the
assumption of 16-bit little-endian audio.

-- chris

 

[Andrew T.]

Andrew T. wrote: .....
I modified the last part of frameToSignedDoubles(), the bit after the diagram,
to read: ....
else {
for (int cc = 0; cc < channels; cc++) {
d[cc] = (b[cc*2+1]*256 + b[cc*2])/32678.0;
} ....
and it seems to work correctly.

That's not the way it looks from here.

If I use the tone.jar to make a sine tone, the traces contain
a distinct 'sawtooth' effect from the small bytes. This made
me think that maybe we should /subtract/ them, but while
that reduced the number of 'sawteeth' per cycle - it did not
remove them.

...Aaaargh!

.....
(moved from above)
...

...I'm afraid the basic problem is still there.
On my the machine program doesn't read output audio, only input (mic, etc). I
can still tell the audio device driver to connect the audio input to its audio
output, but that's the only way to capture audio output.

That is a pity, and a major detriment to the saver.

(I cannot see a way around it, unless the end user is both
prepared and able to reconfigure their sound for the trace to work -
and that is not somethiong -I'd- do for a sound visualisation.)

Still - it will work 'out of the box' for some people, and
potential users will be able to check the PC quickly with
a WebStarted preview.

I will mention it in the program documentation, as well.

Andrew T.

 

[Chris Uppal]

for (int cc = 0; cc < channels; cc++) {
d[cc] = (b[cc*2+1]*256 + b[cc*2])/32678.0;
} ...
and it seems to work correctly.

That's not the way it looks from here.

If I use the tone.jar to make a sine tone, the traces contain
a distinct 'sawtooth' effect from the small bytes.

Ah. My mistake, sorry. Try:

d[cc] = (b[cc*2+1]*256 + (b[cc*2] & 0xFF))/32678.0;

the point is that we have two /unsigned/ values which we then want to normalise
to the range +/- 2**15 (roughly). Just treating the high byte as signed will
do most of the work for us, but we have to convert the second byte to unsigned
before adding it in.

I had seen the "jaggies" but assumed it was just noise -- funny how fixing the
code makes the sound-card work better ;-)

That is a pity, and a major detriment to the saver.

It is unfortunate. I think it's probably all part-and-parcel with the DRM
thing. I now seem to recall that Microsoft don't provide an API for retrieving
sounds as they are played at all (There is a way of getting a deliberately
low-resolution approximation for "visualisations" and the like).

It doubly unfortunate that in the cases where the device driver provides a
loopback, the MS interface to turning it on is about as non-intuitive as can
easily be imagined.

Blame MS, and -- after all, this /is/ Java -- hope that it works better on more
open systems.

-- chris

 

[Andrew T.]

Andrew T. wrote: ....

If I use the tone.jar to make a sine tone, the traces contain
a distinct 'sawtooth' effect from the small bytes.

..Try:

d[cc] = (b[cc*2+1]*256 + (b[cc*2] & 0xFF))/32678.0;

Yes! You are a champion, Chris. Thank you.
.....

I had seen the "jaggies" but assumed it was just noise -- funny how fixing the
code makes the sound-card work better ;-)
LOL!


It is unfortunate. I think it's probably all part-and-parcel with the DRM
thing.

Huh? I had not had much internet time recently and was
tuning out any references to MS, so I was unaware of DRM
till now. Looks horrid - yet another reason to stick with my
older XP until I can find the need/motivation to switch to *nix.

..I now seem to recall that Microsoft don't provide an API for retrieving
sounds as they are played at all (There is a way of getting a deliberately
low-resolution approximation for "visualisations" and the like).

It doubly unfortunate that in the cases where the device driver provides a
loopback, the MS interface to turning it on is about as non-intuitive as can
easily be imagined.

Blame MS, and -- after all, this /is/ Java -- hope that it works better on more
open systems.

If you're right, this saver will become less useful on Win. systems
as time goes by and people upgrade. That would be a distinct pity.
:-(

Andrew T.

 

[Andrew T.]

Chris Uppal wrote:

(no sound signal detected in Java based saver)

It is unfortunate. I think it's probably all part-and-parcel with the DRM
thing. I now seem to recall that Microsoft don't provide an API for retrieving
sounds as they are played at all (There is a way of getting a deliberately
low-resolution approximation for "visualisations" and the like).

I am writing the documentation for the saver, and want to
inform the user why it fails in as few words as possible*.
Would you check if 'DRM' is found in your Windows help?

"* If you only see a flat line or dot for each channel,
it might be due to running a Unix/Linux system
with no audio loop-back installed,
or a Windows system with DRM."

Andrew T.

 

[Chris Uppal]

(no sound signal detected in Java based saver)
[...]
I am writing the documentation for the saver, and want to
inform the user why it fails in as few words as possible*.
Would you check if 'DRM' is found in your Windows help?

Nothing. (What would you expect ;-)

"* If you only see a flat line or dot for each channel,
it might be due to running a Unix/Linux system
with no audio loop-back installed,
or a Windows system with DRM."

I'm not sure it's a good idea to mention DRM -- it isn't that some DRM system
is actively /preventing/ audio monitoring, it's just that (as I understand it)
MS don't see a valid need to insist that audio kit /provides/ the feature (or,
unfortunately, a convenient way to turn it on when available).


-- chris

 

[Andrew T.]

(no sound signal detected in Java based saver)
[...]
I am writing the documentation for the saver, and want to
inform the user why it fails in as few words as possible*.
Would you check if 'DRM' is found in your Windows help?

Nothing. (What would you expect ;-)

"* If you only see a flat line or dot for each channel,
it might be due to running a Unix/Linux system
with no audio loop-back installed,
or a Windows system with DRM."

I'm not sure it's a good idea to mention DRM -- it isn't that some DRM system
is actively /preventing/ audio monitoring,

My research at microsoft suggested they provided sound
only to 'registered and correct' plug-ins that themselves fed the
sound only to other such 'r&c' plug-ins. Failure of a plug-in
supplier to adequately protect the audio bytes could result in
their license (to access the sound system) being revoked.

I very much doubt that Java in general would be recognised
a trusted sink for sound-bytes given that philosophy.

..it's just that (as I understand it)
MS don't see a valid need to insist that audio kit /provides/ the feature (or,
unfortunately, a convenient way to turn it on when available).

But you did find a way to ..re-route the sound from within
the (DRM) OS? No external software or hardware required?

( Note that adding software or hardware to circumvent the
effects of a DRM system might be considered illegal, in
some places. )

Andrew T.

 

[Chris Uppal]

Andrew T. wrote:

[me:]

My research at microsoft suggested they provided sound
only to 'registered and correct' plug-ins that themselves fed the
sound only to other such 'r&c' plug-ins. Failure of a plug-in
supplier to adequately protect the audio bytes could result in
their license (to access the sound system) being revoked.

Hmm, then it's gone further than I realised :-(

But you did find a way to ..re-route the sound from within
the (DRM) OS? No external software or hardware required?

Not exactly. The OS and Windows GUI provide the configuration screens to allow
me to tell the hardware (which came with the machine) to set up a loopback.
Presumably the OS asks the device what inputs it has available, and it
configures the "select an input" window from that. If the device includes a
loopback then that is listed along with all the rest.

-- chris

 

[Andrew T.]

Andrew T. wrote: ....

Not exactly. The OS and Windows GUI ..

Isn't the 'Windows GUI' simply a GUI for configuring 'The OS'?
I do not understand the distinction.

...provide the configuration screens to allow
me to tell the hardware (which came with the machine) to set up a loopback.
Presumably the OS asks the device what inputs it has available, and it
configures the "select an input" window from that. If the device includes a
loopback then that is listed along with all the rest.

OK - I might revise the instructions based on that.

...but did the help find 'DRM'? How does a Win. user
find out if DRM is enabled on their PC?

Andrew T.

 

[Oliver Wong]

Isn't the 'Windows GUI' simply a GUI for configuring 'The OS'?
I do not understand the distinction.

I'm guessing the tool that Chris was using is called "Volume Control".
It comes with every installation of Windows (AFAIK), but I wouldn't say it's
"core" to the OS. You could probably delete that program and no harm would
come to your system (except that you might have difficulty controlling the
volume from then on, unless you find an alternative, equivalent program).

OK - I might revise the instructions based on that.

..but did the help find 'DRM'? How does a Win. user
find out if DRM is enabled on their PC?

Windows XP SP2 help searches locally, and in the Mirosoft Knowledge
Base. Locally, there are zero results, and in the MSKB, there are 15 results
at the time of writing. The topics in the MSKB mostly deal with Windows
Media Player, which makes sense, because that's the only Microsoft
application on WinXP SP2 which actually uses DRM, AFAIK.

DRM is a technology, like XML. It isn't "enabled on a PC" [*].
Applications can be written to be aware of DRM, or they might not be. If
they weren't written to be DRM aware, then they won't be able to decode DRM
protected files (just like a program which doesn't know what XML is probably
won't be able to parse them).

Windows XP itself is not DRM aware, but Vista is planned to be. Media
Player *IS* DRM aware. So that means it's possible to construct media files
that can only be played for a period of 24 hours after it is first
downloaded, for example (to simulate some sort of online movie rental
store). Again WinXP isn't DRM aware, so it won't automagically delete the
file after 24 hours. It's just that Media Player will refuse to play the
file after that period.

- Oliver

[*] You might be thinking of Trusted Computing or TC, which is sort of to
hardware what DRM is to software. Intel is considering making a line of
Trusted CPUs (tm), but they seem to be holding off, since the public opinion
on TC has been mostly negative so far. TC is present in the XBox 360, and is
planned to be in the Playstation 3.