Spectral Editing Idea

G

Guest

Guest
Archived from groups: rec.audio.pro (More info?)

After doing some mixing and remixing I've come to the conclusion that
"spectral editing" - that is, graphically editing a frequency vs time
representation of audio - will be a big part of future audio
management. Yet, there are relatively few programs that allow for this,
and they are all cumbersome and inefficient. I may know more about
mathematics than code, but I know enough to know about how hard
implementation of various things is. So, here's my view of an optimal
spectral editing program.

When a user selects one or more tracks and selects an option, each of
the tracks are run through a FFT routine. This produces, for each track
selected, a 2d image. The top of the image represents higher
frequencies, and the rightmost part of the image represents the end of
the track. The intensity of each frequency at each time is represented
by the color of the pixel, zero intensity being represented by blue,
and then going through the spectrum to green and red according to the
log of the intensity.

When tracks that have had this done are selected, a new window displays
the sum of the images of all the selected tracks that have had this
processing done.

Since the FFT takes a while, I suggest that all images are saved in a
new file in the same place as the original audio, with the same name
and a new extension, with some fast compression on it, possibly just a
Haar wavelet transform and LZW compression.

3 more windows are also created - 'controls', 'current brush', and
'sandbox'.

On "controls" there is:
a master intensity control, controlled by a slider and a numeric field,
that adjusts the intensity of the brush - this one can go negative;
a master size control, controlled by a slider and a numeric field, that
adjusts the size of the brush;
a brush/select listbox, controlled by a slider and a numeric field,
that adjusts the size of the brush;
a add/set/scale/distort listbox, where one of these modes would be
selected by clicking on it;
a single/continuous listbox, like above, except with 2 options, and
doing something different.

All modifications to images are done with a brush. This brush is a 2d
image, in the same form as the main image, although in greyscale rather
than color to make a superposition of the brush and the image easier to
make out and make importing easier. It is shown in the 'current brush'
window. When the brush is selected in "controls" and the curser is over
the image, the brush is superimposed as a "ghost" on the image.

The "sandbox" would be a mostly-blank window that is simply there for
making brush shapes with other brushes and storing bits of images. It
would start out with only a circle on it, maybe a square...

If "select" is selected in "controls" then dragging the cursor on the
main image window or the sandbox creates a selected box area. This can
then be copied. That's all it's good for.

If "brush" is selected on "controls"
If "add" is selected in the controls window, using the brush on an
image adds the intensity of each pixel together (in linear space, not
logarithmic), with the intensity of each brush pixel, scaled by the
value of the "master intensity control" on the "controls" window.
If "set" is selected, using the brush on an image sets the value of
each pixel of the image to the value of the pixel of the brush above
it.
If "scale" is selected, using the brush on an image multiplies the
intensity of each pixel of the edited image (in linear space) by the
intensity of each brush pixel, scaled by the value of the "master
intensity control."
If "distort" is selected, then large values in the brush "pull"
intensity in the image towards them. In other worlds, each pixel in the
edited image has (constant)*(value of brush pixel over it)*(sum of
values of pixels touching the pixel in question) added to it and
(constant)*(value of pixel in question in edited image)*(sum of values
of brush pixels over the pixels touching the pixel in question)
subtracted from it.

If "single" is selected on "controls" when the brush is being used,
then the brush is applied to the original image fairly often, and the
change for each pixel on on original image is the greatest change of
all the applications of the brush. So if you click once, if you hold
for half a second or for 4 seconds, you stamp the brush where you put
it just the same, but holding and dragging a small circle would still
make a line. If "set" is selected, however, the application is done
once and then nothing happens until the *next* mouse click.

If "continuous" is selected when the brush is being used, then the
brush is applied fairly often, but with a fraction of the effect each
time of when "single" is selected; however, each change applies to the
values of the image as changed by the last application. So holding a
light shape for a few seconds stamps that shape more darkly on the
image. I'm thinking about a half-second is good for being equal to one
application with "single" but it doesn't really matter if the intensity
can be adjusted. If "set" is selected, however, the application is done
once and then nothing happens until the *next* mouse click.

"Copy" would be setting the brush to whatever was selected at the time
- converting it to greyscale according to intensity if it was from the
image, or keeping it unchanged if it was from the sandbox. Copy should
automatically set the intensity and size settings to the center, so
that the brush reproduces exactly what is copied before changing those
settings. To revert the brush, one could record it in the sandbox and
copy that afterwards, but a brush history would nevertheless be more
convenient. It may be necessary for this to be a separate command from
the system's standard copy command.

To import images from graphics programs, either a mechanism for
importing the clipboard contents or a mechanism for importing an image
file would be needed. If an image is color, it would be converted to
greyscale as per the equivalence within the program.

Paste would not be necessary, as it would be equivalent to a click with
brush/set/single selected.

Whenever a change is made to an image, the individual change is
recorded in a log for undoes, and the change is converted to a waveform
by sine wave superposition, and added to the waveform, with the
waveform difference being saved in RAM with the FFT space difference.
If a change is made to an image representing multiple tracks, it is
then saved in a separate log for each track, and the waveform
difference for the change in FFT space for each track is converted by
sine wave superposition and added to each track separately.

This would probably be based off Audacity. It could probably pull some
code from Ceres3, and could possibly even pull some code from GIMP.
 
G

Guest

Guest
Archived from groups: rec.audio.pro (More info?)

<bhauth@gmail.com> wrote:
>After doing some mixing and remixing I've come to the conclusion that
>"spectral editing" - that is, graphically editing a frequency vs time
>representation of audio - will be a big part of future audio
>management. Yet, there are relatively few programs that allow for this,
>and they are all cumbersome and inefficient. I may know more about
>mathematics than code, but I know enough to know about how hard
>implementation of various things is. So, here's my view of an optimal
>spectral editing program.

What you describe sounds to me just like an automated parametric. Skip
the FFT stuff and just use conventional filters, then tie it in with the
rest of the console automation system. Voila.

What you want is a user interface that feels like the rest of the automation,
so users don't have to learn a complete new thing just for EQ.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."
 
G

Guest

Guest
Archived from groups: rec.audio.pro (More info?)

Are you using the Audition Spectral Editing? Take a look at Sequoia with
the reNOVAtor software by Algorithmix- very fast and easy to use... Much of
what you have described is implemented, but a bit more elegantly. It is
expensive, but already on the market.

Algorithmix's reNOVAtor takes spectral editing to an incredible level with
the whole "photoshop of sound" principle. Comparisonic colors in the
waveform will give you frequency information at a glance... The FFT filters
are either realtime or offline (which has greater accuracy)

-Ben

--
Benjamin Maas
Fifth Circle Audio
Los Angeles, CA
http://www.fifthcircle.com

Please remove "Nospam" from address for replies

<bhauth@gmail.com> wrote in message ...
> After doing some mixing and remixing I've come to the conclusion that
> "spectral editing" - that is, graphically editing a frequency vs time
> representation of audio - will be a big part of future audio
> management. Yet, there are relatively few programs that allow for this,
> and they are all cumbersome and inefficient. I may know more about
> mathematics than code, but I know enough to know about how hard
> implementation of various things is. So, here's my view of an optimal
> spectral editing program.
>
> When a user selects one or more tracks and selects an option, each of
> the tracks are run through a FFT routine. This produces, for each track
> selected, a 2d image. The top of the image represents higher
> frequencies, and the rightmost part of the image represents the end of
> the track. The intensity of each frequency at each time is represented
> by the color of the pixel, zero intensity being represented by blue,
> and then going through the spectrum to green and red according to the
> log of the intensity.
>
> When tracks that have had this done are selected, a new window displays
> the sum of the images of all the selected tracks that have had this
> processing done.
>
> Since the FFT takes a while, I suggest that all images are saved in a
> new file in the same place as the original audio, with the same name
> and a new extension, with some fast compression on it, possibly just a
> Haar wavelet transform and LZW compression.
>
> 3 more windows are also created - 'controls', 'current brush', and
> 'sandbox'.
>
> On "controls" there is:
> a master intensity control, controlled by a slider and a numeric field,
> that adjusts the intensity of the brush - this one can go negative;
> a master size control, controlled by a slider and a numeric field, that
> adjusts the size of the brush;
> a brush/select listbox, controlled by a slider and a numeric field,
> that adjusts the size of the brush;
> a add/set/scale/distort listbox, where one of these modes would be
> selected by clicking on it;
> a single/continuous listbox, like above, except with 2 options, and
> doing something different.
>
> All modifications to images are done with a brush. This brush is a 2d
> image, in the same form as the main image, although in greyscale rather
> than color to make a superposition of the brush and the image easier to
> make out and make importing easier. It is shown in the 'current brush'
> window. When the brush is selected in "controls" and the curser is over
> the image, the brush is superimposed as a "ghost" on the image.
>
> The "sandbox" would be a mostly-blank window that is simply there for
> making brush shapes with other brushes and storing bits of images. It
> would start out with only a circle on it, maybe a square...
>
> If "select" is selected in "controls" then dragging the cursor on the
> main image window or the sandbox creates a selected box area. This can
> then be copied. That's all it's good for.
>
> If "brush" is selected on "controls"
> If "add" is selected in the controls window, using the brush on an
> image adds the intensity of each pixel together (in linear space, not
> logarithmic), with the intensity of each brush pixel, scaled by the
> value of the "master intensity control" on the "controls" window.
> If "set" is selected, using the brush on an image sets the value of
> each pixel of the image to the value of the pixel of the brush above
> it.
> If "scale" is selected, using the brush on an image multiplies the
> intensity of each pixel of the edited image (in linear space) by the
> intensity of each brush pixel, scaled by the value of the "master
> intensity control."
> If "distort" is selected, then large values in the brush "pull"
> intensity in the image towards them. In other worlds, each pixel in the
> edited image has (constant)*(value of brush pixel over it)*(sum of
> values of pixels touching the pixel in question) added to it and
> (constant)*(value of pixel in question in edited image)*(sum of values
> of brush pixels over the pixels touching the pixel in question)
> subtracted from it.
>
> If "single" is selected on "controls" when the brush is being used,
> then the brush is applied to the original image fairly often, and the
> change for each pixel on on original image is the greatest change of
> all the applications of the brush. So if you click once, if you hold
> for half a second or for 4 seconds, you stamp the brush where you put
> it just the same, but holding and dragging a small circle would still
> make a line. If "set" is selected, however, the application is done
> once and then nothing happens until the *next* mouse click.
>
> If "continuous" is selected when the brush is being used, then the
> brush is applied fairly often, but with a fraction of the effect each
> time of when "single" is selected; however, each change applies to the
> values of the image as changed by the last application. So holding a
> light shape for a few seconds stamps that shape more darkly on the
> image. I'm thinking about a half-second is good for being equal to one
> application with "single" but it doesn't really matter if the intensity
> can be adjusted. If "set" is selected, however, the application is done
> once and then nothing happens until the *next* mouse click.
>
> "Copy" would be setting the brush to whatever was selected at the time
> - converting it to greyscale according to intensity if it was from the
> image, or keeping it unchanged if it was from the sandbox. Copy should
> automatically set the intensity and size settings to the center, so
> that the brush reproduces exactly what is copied before changing those
> settings. To revert the brush, one could record it in the sandbox and
> copy that afterwards, but a brush history would nevertheless be more
> convenient. It may be necessary for this to be a separate command from
> the system's standard copy command.
>
> To import images from graphics programs, either a mechanism for
> importing the clipboard contents or a mechanism for importing an image
> file would be needed. If an image is color, it would be converted to
> greyscale as per the equivalence within the program.
>
> Paste would not be necessary, as it would be equivalent to a click with
> brush/set/single selected.
>
> Whenever a change is made to an image, the individual change is
> recorded in a log for undoes, and the change is converted to a waveform
> by sine wave superposition, and added to the waveform, with the
> waveform difference being saved in RAM with the FFT space difference.
> If a change is made to an image representing multiple tracks, it is
> then saved in a separate log for each track, and the waveform
> difference for the change in FFT space for each track is converted by
> sine wave superposition and added to each track separately.
>
> This would probably be based off Audacity. It could probably pull some
> code from Ceres3, and could possibly even pull some code from GIMP.
>
 
G

Guest

Guest
Archived from groups: rec.audio.pro (More info?)

> so users don't have to learn a complete new thing just for EQ.


What do you mean "just for EQ"? You can create any sound and add it in
however you want! You can use the distortion in the sandbox to make a
bird song sound robotic! You can compress or expand the spectral
profile! You can strip out individual bass notes and change individual
drum hits to sound like hits of a different drum! You can do virtually
anything! It would be less efficient at MIDI composing and shuffling
song pieces around, but there are PLENTY of programs doing that.
 
G

Guest

Guest
Archived from groups: rec.audio.pro (More info?)

> http://www.uisoftware.com/MetaSynth/index.html

That is sort of a graphical EQ with limited functionality. For more of
an idea of about what the /representation/ would look like, try
Frequency. http://home3.swipnet.se/~w-34826/ Of course, my proposal
would be infinitely more powerful.
 
G

Guest

Guest
Archived from groups: rec.audio.pro (More info?)

> Are you using the Audition Spectral Editing?

For what I mean, it has no such thing.

> Take a look at Sequoia with
> the reNOVAtor software by Algorithmix- very fast and easy to use...
Much of

Seems to be Frequency with copy/paste and better algorithms, for lots
of money instead of for free. =/ So it would also be nice for some
specific tasks, but it would hardly be useful as a large part of
composition as the proposed program would be.
 
G

Guest

Guest
Archived from groups: rec.audio.pro (More info?)

John wrote:
> Have you played with KYMA?

Sorry, I generally only spend $3500 on things that I'm pretty sure will
be pretty useful.

"A synthesizer, sampler, digital sound processor, digital
recorder/controller, sound analysis tool, and a re-synthesizer all in
one box?"

Well, this program integrated with, say, Audacity, would do all that
and more.

Maybe you can elaborate on exactly what it is and what it does spectral
editing-wise?
 
G

Guest

Guest
Archived from groups: rec.audio.pro (More info?)

bhauth@gmail.com wrote:
> > http://www.uisoftware.com/MetaSynth/index.html
>
> That is sort of a graphical EQ with limited functionality. For more
of
> an idea of about what the /representation/ would look like, try
> Frequency. http://home3.swipnet.se/~w-34826/ Of course, my proposal
> would be infinitely more powerful.

Have you tried the demo of it? It's a little more than a graphical EQ
with limited functionality. Right now the designers have put more time
into the latest update than getting the site up to date with the latest
features. It's a powerful app. The program can do some of the things
you want to do in its own way but of course not all...

I follow where you are going with your ideas. I have some similar ideas
in my head. Until computers can do these things I'll just need to keep
listening with my ears and seeing with my mind.

Peter
 
G

Guest

Guest
Archived from groups: rec.audio.pro (More info?)

> Everything else you described can be done with
> knobs, whether real or virtual.


Bzzzt. 100% wrong.


> You should look into a 'spray can' function, where you can spray the
image
> on the clipboard onto your main image, while controlling it's size
and
> intensity, and also a 'blend' tool, a 'thumb' or whatever. Then you'd
> theoretically be able to drop in characteritics of different sounds.


Try "reading the post." Then you would see that every single function
is of this type, except things are done a lot better than how you say
they should be.
 
G

Guest

Guest
Archived from groups: rec.audio.pro (More info?)

<bhauth@gmail.com> wrote in message
news:1109496033.511981.317550@g14g2000cwa.googlegroups.com...

> This would probably be based off Audacity. It could probably pull some
> code from Ceres3, and could possibly even pull some code from GIMP.


Sounds pretty neat, you should do it. The interesting function is importing
pieces of another image. Everything else you described can be done with
knobs, whether real or virtual.

You should look into a 'spray can' function, where you can spray the image
on the clipboard onto your main image, while controlling it's size and
intensity, and also a 'blend' tool, a 'thumb' or whatever. Then you'd
theoretically be able to drop in characteritics of different sounds.

I see it being more useful for synthesis than for pro audio.

jb
 
G

Guest

Guest
Archived from groups: rec.audio.pro (More info?)

Well, I guess I'll look at the demo.

What do you mean "until computers can do those things" though? The math
is all there. The computational power required is available in the
typical PC. Computers *can* do these things.
 
G

Guest

Guest
Archived from groups: rec.audio.pro (More info?)

<bhauth@gmail.com> wrote in message
news:1109572090.878989.275130@f14g2000cwb.googlegroups.com...
> > Everything else you described can be done with
> > knobs, whether real or virtual.
>
>
> Bzzzt. 100% wrong.
>

No functions that I can see from the post. So be specific on this point.
Then I can say if it can be done with knobs or not.

>
> > You should look into a 'spray can' function, where you can spray the
> image
> > on the clipboard onto your main image, while controlling it's size
> and
> > intensity, and also a 'blend' tool, a 'thumb' or whatever. Then you'd
> > theoretically be able to drop in characteritics of different sounds.
>
>
> Try "reading the post." Then you would see that every single function
> is of this type, except things are done a lot better than how you say
> they should be.

Fine, whatever, go learn some programming. I'll want the tool for synthesis
if I can spray and blend.

jb