Archived from groups: rec.audio.pro (More info?)
After doing some mixing and remixing I've come to the conclusion that
"spectral editing" - that is, graphically editing a frequency vs time
representation of audio - will be a big part of future audio
management. Yet, there are relatively few programs that allow for this,
and they are all cumbersome and inefficient. I may know more about
mathematics than code, but I know enough to know about how hard
implementation of various things is. So, here's my view of an optimal
spectral editing program.
When a user selects one or more tracks and selects an option, each of
the tracks are run through a FFT routine. This produces, for each track
selected, a 2d image. The top of the image represents higher
frequencies, and the rightmost part of the image represents the end of
the track. The intensity of each frequency at each time is represented
by the color of the pixel, zero intensity being represented by blue,
and then going through the spectrum to green and red according to the
log of the intensity.
When tracks that have had this done are selected, a new window displays
the sum of the images of all the selected tracks that have had this
processing done.
Since the FFT takes a while, I suggest that all images are saved in a
new file in the same place as the original audio, with the same name
and a new extension, with some fast compression on it, possibly just a
Haar wavelet transform and LZW compression.
3 more windows are also created - 'controls', 'current brush', and
'sandbox'.
On "controls" there is:
a master intensity control, controlled by a slider and a numeric field,
that adjusts the intensity of the brush - this one can go negative;
a master size control, controlled by a slider and a numeric field, that
adjusts the size of the brush;
a brush/select listbox, controlled by a slider and a numeric field,
that adjusts the size of the brush;
a add/set/scale/distort listbox, where one of these modes would be
selected by clicking on it;
a single/continuous listbox, like above, except with 2 options, and
doing something different.
All modifications to images are done with a brush. This brush is a 2d
image, in the same form as the main image, although in greyscale rather
than color to make a superposition of the brush and the image easier to
make out and make importing easier. It is shown in the 'current brush'
window. When the brush is selected in "controls" and the curser is over
the image, the brush is superimposed as a "ghost" on the image.
The "sandbox" would be a mostly-blank window that is simply there for
making brush shapes with other brushes and storing bits of images. It
would start out with only a circle on it, maybe a square...
If "select" is selected in "controls" then dragging the cursor on the
main image window or the sandbox creates a selected box area. This can
then be copied. That's all it's good for.
If "brush" is selected on "controls"
If "add" is selected in the controls window, using the brush on an
image adds the intensity of each pixel together (in linear space, not
logarithmic), with the intensity of each brush pixel, scaled by the
value of the "master intensity control" on the "controls" window.
If "set" is selected, using the brush on an image sets the value of
each pixel of the image to the value of the pixel of the brush above
it.
If "scale" is selected, using the brush on an image multiplies the
intensity of each pixel of the edited image (in linear space) by the
intensity of each brush pixel, scaled by the value of the "master
intensity control."
If "distort" is selected, then large values in the brush "pull"
intensity in the image towards them. In other worlds, each pixel in the
edited image has (constant)*(value of brush pixel over it)*(sum of
values of pixels touching the pixel in question) added to it and
(constant)*(value of pixel in question in edited image)*(sum of values
of brush pixels over the pixels touching the pixel in question)
subtracted from it.
If "single" is selected on "controls" when the brush is being used,
then the brush is applied to the original image fairly often, and the
change for each pixel on on original image is the greatest change of
all the applications of the brush. So if you click once, if you hold
for half a second or for 4 seconds, you stamp the brush where you put
it just the same, but holding and dragging a small circle would still
make a line. If "set" is selected, however, the application is done
once and then nothing happens until the *next* mouse click.
If "continuous" is selected when the brush is being used, then the
brush is applied fairly often, but with a fraction of the effect each
time of when "single" is selected; however, each change applies to the
values of the image as changed by the last application. So holding a
light shape for a few seconds stamps that shape more darkly on the
image. I'm thinking about a half-second is good for being equal to one
application with "single" but it doesn't really matter if the intensity
can be adjusted. If "set" is selected, however, the application is done
once and then nothing happens until the *next* mouse click.
"Copy" would be setting the brush to whatever was selected at the time
- converting it to greyscale according to intensity if it was from the
image, or keeping it unchanged if it was from the sandbox. Copy should
automatically set the intensity and size settings to the center, so
that the brush reproduces exactly what is copied before changing those
settings. To revert the brush, one could record it in the sandbox and
copy that afterwards, but a brush history would nevertheless be more
convenient. It may be necessary for this to be a separate command from
the system's standard copy command.
To import images from graphics programs, either a mechanism for
importing the clipboard contents or a mechanism for importing an image
file would be needed. If an image is color, it would be converted to
greyscale as per the equivalence within the program.
Paste would not be necessary, as it would be equivalent to a click with
brush/set/single selected.
Whenever a change is made to an image, the individual change is
recorded in a log for undoes, and the change is converted to a waveform
by sine wave superposition, and added to the waveform, with the
waveform difference being saved in RAM with the FFT space difference.
If a change is made to an image representing multiple tracks, it is
then saved in a separate log for each track, and the waveform
difference for the change in FFT space for each track is converted by
sine wave superposition and added to each track separately.
This would probably be based off Audacity. It could probably pull some
code from Ceres3, and could possibly even pull some code from GIMP.
After doing some mixing and remixing I've come to the conclusion that
"spectral editing" - that is, graphically editing a frequency vs time
representation of audio - will be a big part of future audio
management. Yet, there are relatively few programs that allow for this,
and they are all cumbersome and inefficient. I may know more about
mathematics than code, but I know enough to know about how hard
implementation of various things is. So, here's my view of an optimal
spectral editing program.
When a user selects one or more tracks and selects an option, each of
the tracks are run through a FFT routine. This produces, for each track
selected, a 2d image. The top of the image represents higher
frequencies, and the rightmost part of the image represents the end of
the track. The intensity of each frequency at each time is represented
by the color of the pixel, zero intensity being represented by blue,
and then going through the spectrum to green and red according to the
log of the intensity.
When tracks that have had this done are selected, a new window displays
the sum of the images of all the selected tracks that have had this
processing done.
Since the FFT takes a while, I suggest that all images are saved in a
new file in the same place as the original audio, with the same name
and a new extension, with some fast compression on it, possibly just a
Haar wavelet transform and LZW compression.
3 more windows are also created - 'controls', 'current brush', and
'sandbox'.
On "controls" there is:
a master intensity control, controlled by a slider and a numeric field,
that adjusts the intensity of the brush - this one can go negative;
a master size control, controlled by a slider and a numeric field, that
adjusts the size of the brush;
a brush/select listbox, controlled by a slider and a numeric field,
that adjusts the size of the brush;
a add/set/scale/distort listbox, where one of these modes would be
selected by clicking on it;
a single/continuous listbox, like above, except with 2 options, and
doing something different.
All modifications to images are done with a brush. This brush is a 2d
image, in the same form as the main image, although in greyscale rather
than color to make a superposition of the brush and the image easier to
make out and make importing easier. It is shown in the 'current brush'
window. When the brush is selected in "controls" and the curser is over
the image, the brush is superimposed as a "ghost" on the image.
The "sandbox" would be a mostly-blank window that is simply there for
making brush shapes with other brushes and storing bits of images. It
would start out with only a circle on it, maybe a square...
If "select" is selected in "controls" then dragging the cursor on the
main image window or the sandbox creates a selected box area. This can
then be copied. That's all it's good for.
If "brush" is selected on "controls"
If "add" is selected in the controls window, using the brush on an
image adds the intensity of each pixel together (in linear space, not
logarithmic), with the intensity of each brush pixel, scaled by the
value of the "master intensity control" on the "controls" window.
If "set" is selected, using the brush on an image sets the value of
each pixel of the image to the value of the pixel of the brush above
it.
If "scale" is selected, using the brush on an image multiplies the
intensity of each pixel of the edited image (in linear space) by the
intensity of each brush pixel, scaled by the value of the "master
intensity control."
If "distort" is selected, then large values in the brush "pull"
intensity in the image towards them. In other worlds, each pixel in the
edited image has (constant)*(value of brush pixel over it)*(sum of
values of pixels touching the pixel in question) added to it and
(constant)*(value of pixel in question in edited image)*(sum of values
of brush pixels over the pixels touching the pixel in question)
subtracted from it.
If "single" is selected on "controls" when the brush is being used,
then the brush is applied to the original image fairly often, and the
change for each pixel on on original image is the greatest change of
all the applications of the brush. So if you click once, if you hold
for half a second or for 4 seconds, you stamp the brush where you put
it just the same, but holding and dragging a small circle would still
make a line. If "set" is selected, however, the application is done
once and then nothing happens until the *next* mouse click.
If "continuous" is selected when the brush is being used, then the
brush is applied fairly often, but with a fraction of the effect each
time of when "single" is selected; however, each change applies to the
values of the image as changed by the last application. So holding a
light shape for a few seconds stamps that shape more darkly on the
image. I'm thinking about a half-second is good for being equal to one
application with "single" but it doesn't really matter if the intensity
can be adjusted. If "set" is selected, however, the application is done
once and then nothing happens until the *next* mouse click.
"Copy" would be setting the brush to whatever was selected at the time
- converting it to greyscale according to intensity if it was from the
image, or keeping it unchanged if it was from the sandbox. Copy should
automatically set the intensity and size settings to the center, so
that the brush reproduces exactly what is copied before changing those
settings. To revert the brush, one could record it in the sandbox and
copy that afterwards, but a brush history would nevertheless be more
convenient. It may be necessary for this to be a separate command from
the system's standard copy command.
To import images from graphics programs, either a mechanism for
importing the clipboard contents or a mechanism for importing an image
file would be needed. If an image is color, it would be converted to
greyscale as per the equivalence within the program.
Paste would not be necessary, as it would be equivalent to a click with
brush/set/single selected.
Whenever a change is made to an image, the individual change is
recorded in a log for undoes, and the change is converted to a waveform
by sine wave superposition, and added to the waveform, with the
waveform difference being saved in RAM with the FFT space difference.
If a change is made to an image representing multiple tracks, it is
then saved in a separate log for each track, and the waveform
difference for the change in FFT space for each track is converted by
sine wave superposition and added to each track separately.
This would probably be based off Audacity. It could probably pull some
code from Ceres3, and could possibly even pull some code from GIMP.
/www.fifthcircle.com
Facebook
Twitter