Quantcast
FLAC is our current best hope for the future of fidelity - Page 7 - Gearspace.com
The No.1 Website for Pro Audio
FLAC is our current best hope for the future of fidelity
Old 1st December 2010 | Show parent
  #181
Lives for gear
 
807Recordings's Avatar
 
Verified Member
🎧 10 years
Quote:
Originally Posted by soulstudios ➡️
Absolute rubbish.
WAV has no error-checking or CRC involved, both wavpack and flac have plenty.
Archiving as straight wavs is the worst archive format possible for data protection.
With a wavpack backup you can have the same file twice on the same medium (compared to wav) and have a reliable, efficient way of detecting data integrity. The same goes for individual files zipped (though the compression ratio is typically worse).

You should not be relying on the audio format for the archiving if your pro but rather a series of properly implemented solutions. If your wave is saved correctly and your are using a good back up solution then there is no need to have another.

But then what would I know about this stuff outside of the 15 years of IT consulting. :D


On a side note I just don't see Flac being a consumer level medium. If it was going to be it would have been by now. Personally I have no issues with the format though and can't hear the difference when properly encoded of that and the original wave.
Old 1st December 2010 | Show parent
  #182
soulstudios
Guest
Quote:
Originally Posted by 807Recordings ➡️
You should not be relying on the audio format for the archiving if your pro but rather a series of properly implemented solutions. If your wave is saved correctly and your are using a good back up solution then there is no need to have another.

But then what would I know about this stuff outside of the 15 years of IT consulting. :D
About the same as me then

No, that's an okay attitude for filetypes where corruption is obvious, but not for raw data, which WAV is, with a header.
For data security you need two or more backups on different media, not one, and if it isn't easy to detect corruption from the data itself (which it isn't for audio without wasting years of time) then you need cyclic redundancy checking within the format. And that's what flac and wavpack (and zip) give. To use raw data for backup is ridiculous.
Doing a comparison between two raw backup copies is a waste of time. If one of them is corrupt, without listening/viewing each it's impossible to know which is the corrupt one and which one isn't.
If you're advocating using WAV to your clients for backup I'd take a swing at you, honestly..
Old 1st December 2010 | Show parent
  #183
Lives for gear
 
Cellotron's Avatar
 
Verified Member
3 Reviews written
🎧 15 years
Quote:
Originally Posted by soulstudios ➡️
If you're advocating using WAV to your clients I'd take a swing at you, honestly..
fwiw - the NARAS committee dedicated to the subject a few years ago, after their study in fact recommended bwf wav:

http://www2.grammy.com/PDFs/Recordin...liveryRecs.pdf

Best regards,
Steve Berson
Old 1st December 2010 | Show parent
  #184
soulstudios
Guest
Quote:
Originally Posted by Cellotron ➡️
fwiw - the NARAS committee dedicated to the subject a few years ago, after their study in fact recommended bwf wav:

http://www2.grammy.com/PDFs/Recordin...liveryRecs.pdf

Best regards,
Steve Berson
That's for delivery format, not archiving.
Old 1st December 2010 | Show parent
  #185
Lives for gear
 
dcollins's Avatar
 
Verified Member
🎧 15 years
Quote:
Originally Posted by soulstudios ➡️
About the same as me then

No, that's an okay attitude for filetypes where corruption is obvious, but not for raw data, which WAV is, with a header.
For data security you need two or more backups on different media, not one, and if it isn't easy to detect corruption from the data itself (which it isn't for audio without wasting years of time) then you need cyclic redundancy checking within the format. And that's what flac and wavpack (and zip) give. To use raw data for backup is ridiculous.
Doing a comparison between two raw backup copies is a waste of time. If one of them is corrupt, without listening/viewing each it's impossible to know which is the corrupt one and which one isn't.
If you're advocating using WAV to your clients I'd take a swing at you, honestly..
There are many layers of ECC in our storage and transmission mediums, they are just transparent to the user.

The recommendation to use BWAV is a good one and is supported by a lot of archivists. If you want, you can put a checksum or .zip encode.


DC
Old 1st December 2010 | Show parent
  #186
Lives for gear
 
Cellotron's Avatar
 
Verified Member
3 Reviews written
🎧 15 years
Quote:
Originally Posted by soulstudios ➡️
That's for delivery format, not archiving.
I suggest you read the pdf in full once more, as these are in fact also archiving recommendations. (i.e. the recommendation for inclusion of one "flattened" track without processing or automation, and one with processing/automation - when archiving the individual tracks of multitrack recordings)

Best regards,
Steve Berson
Old 1st December 2010 | Show parent
  #187
soulstudios
Guest
Quote:
Originally Posted by dcollins ➡️
There are many layers of ECC in our storage and transmission mediums, they are just transparent to the user.

The recommendation to use BWAV is a good one
No, it's not.
It doesn't matter how many layers of ECC you put in, if you get data corruption, you get data corruption (which happens with or without ECC, and has happened to me on multiple mediums - HDD, CD, DVD) and you need a method of detecting it. I've already outlined very plainly why CRC built into the format is desirable for a raw datatype, and I'm surprised you can't see it. I also mentioned zip earlier.
Old 1st December 2010 | Show parent
  #188
soulstudios
Guest
Quote:
Originally Posted by Cellotron ➡️
I suggest you read the pdf in full once more, as these are in fact also archiving recommendations. (i.e. the recommendation for inclusion of one "flattened" track without processing or automation, and one with processing/automation - when archiving the individual tracks of multitrack recordings)

Best regards,
Steve Berson
I did read it, and they're embarrassingly out of the loop.
I'll go with my IT solution knowledge, thanks, rather than an embarrassingly out of date research project by a non-IT group.
As already noted, you can't determine a corrupt file between two copies of a raw data format. WAV or BWAV is an entirely inadequate solution for archival material. The fact that some industry body says otherwise (and gives really bad reasons for doing so btw) shouldn't get in the way of using your head...
Old 1st December 2010 | Show parent
  #189
Lives for gear
 
Cellotron's Avatar
 
Verified Member
3 Reviews written
🎧 15 years
Quote:
Originally Posted by soulstudios ➡️
I did read it, and they're embarrassingly out of the loop.
Yup, guess your right - all those folks like that Massenburg and Ainlay guys (whoever they are) probably have never really been in the main stream or done any real audio engineering work.

Quote:
I'll go with my IT solution knowledge, thanks, rather than an embarrassingly out of date research project by a non-IT group.
As already noted, you can't determine a corrupt file between two copies of a raw data format. WAV or BWAV is an entirely inadequate solution for archival material. The fact that some industry body says otherwise (and gives really bad reasons for doing so btw) shouldn't get in the way of using your head...
All kidding aside - obviously you should do whatever you like. They are your files - so handle them any way you want. But if you hand a hard drive of wavpack files to a client I'd say you are much more likely to receive an email saying "hey, I can't play these" than if they were instead saved as wav.

I also was just trying to point out that there was in fact a fairly large summit of audio engineers and IT consultants that extended over 5 years that did in fact put forth some recommendations that were also endorsed by the AES as suggested "standards" so that in the future when restoring a digital audio session archive - instead of us likely having to try to open dozens of different digital file formats saved in a variety of ways - we instead have a single one.

Best regards,
Steve Berson
Old 2nd December 2010 | Show parent
  #190
soulstudios
Guest
Quote:
Originally Posted by Cellotron ➡️
Yup, guess your right - all those folks like that Massenburg and Ainlay guys (whoever they are) probably have never really been in the main stream or done any real audio engineering work.
You can save the dancing emoticon. Having read the document, it is very clear that while they may have some audio experience, their IT knowledge is terribly archaic. I can pull it apart - can you?



Quote:
Originally Posted by Cellotron ➡️
All kidding aside - obviously you should do whatever you like. They are your files - so handle them any way you want. But if you hand a hard drive of wavpack files to a client I'd say you are much more likely to receive an email saying "hey, I can't play these" than if they were instead saved as wav.
Funny, I thought I was talking about flac/wavpack as an archive format - but suit yourself. If I was dumb enough to not decode them for a client on delivery, well, I guess I'd be as smart as those people who wrote the above pdf.


Quote:
Originally Posted by Cellotron ➡️
I also was just trying to point out that there was in fact a fairly large summit of audio engineers and IT consultants that extended over 5 years that did in fact put forth some recommendations that were also endorsed by the AES as suggested "standards" so that in the future when restoring a digital audio session archive - instead of us likely having to try to open dozens of different digital file formats saved in a variety of ways - we instead have a single one.
And that makes sense - for delivery - and even in the context of the document where they talk about 'archival', they are still talking about archival for delivery formats, not archival for studio sessions. It's a really stupid document, and even someone of limited IT skill can easily see how limited and pathetic the scope of their study is.

As noted, you need to use your head, read the logic I've put forth above, and explain to yourself why wav is a bad idea for archiving. Unless of course you want to postulate an uncorruptable medium, which is always fun.
Old 2nd December 2010 | Show parent
  #191
Lives for gear
 
807Recordings's Avatar
 
Verified Member
🎧 10 years
Quote:
Originally Posted by soulstudios ➡️
I did read it, and they're embarrassingly out of the loop.
I'll go with my IT solution knowledge, thanks, rather than an embarrassingly out of date research project by a non-IT group.
As already noted, you can't determine a corrupt file between two copies of a raw data format. WAV or BWAV is an entirely inadequate solution for archival material. The fact that some industry body says otherwise (and gives really bad reasons for doing so btw) shouldn't get in the way of using your head...

I am not sure your back-ground but by your logic then 90% of critical data would be lost now of days.

IF the Wave file plays correctly then its most likely ready to archive.

There is a vast array of system to make sure data is not corrupted when you go to archive. With music it should start with a playback.

When I worked in IT as director of technologies which included many clients who rely on real time transactions in mission critical and data secure systems it was always taken care of at the application layer. Not exclusively to that layer but in a great part. From ECM or Network solutions to geographic backups much of the files had no error or CRC capabilities. There is no need for it and a proper back up solution will not archive something corrupted.

I am off the solution to not alter the original data in any matter but to be able to reproduce it exactly as it was created. If it was in Wave, AIFF, RAW or whatever else then that is what should come out again. Exactly the same as it went in.
Old 9th December 2010 | Show parent
  #192
Gear Maniac
 
🎧 10 years
Quote:
Originally Posted by soulstudios ➡️
Funny, I thought I was talking about flac/wavpack as an archive format - but suit yourself. If I was dumb enough to not decode them for a client on delivery, well, I guess I'd be as smart as those people who wrote the above pdf.

I think of "archive" as something that could be around for generations - if it requires decoding, it's not a good archive. (I don't think analog tape w/ any sort of noise reduction makes a good archive format either, for example.) Maybe in IT circles, archives are meant to only be accessed by IT folks, but when it comes to something like music, maybe the rules are different? I think the simpler to access in the future, the better.
Old 9th December 2010 | Show parent
  #193
Lives for gear
 
🎧 10 years
The way I see it, if you really want an archive that will last decades, you better put it on analog tape.

Hard Drives just don't last that long no matter what the file format is. And who knows if a computer 40 years in the future will have an operating system that even recognizes any of today's files? Or hell, if they will be able to interface with any of our hard drives, optical discs, or flash drives?
Old 9th December 2010 | Show parent
  #194
Lives for gear
 
Susceptor's Avatar
 
🎧 10 years
FLAC is a FORMAT that is OPEN and well DOCUMENTED.
There are enough hardware options to maintain an archive over time. Maybe the hardware interface won't allow it, but legacy interfaces still exist and when new, better interfaces appear, the older ones aren't ruled out instantly, so people can adapt to the new thing.
Old 9th December 2010 | Show parent
  #195
Gear Maniac
 
SixAndChange's Avatar
 
🎧 10 years
Quote:
Originally Posted by Cheebs Goat ➡️
The way I see it, if you really want an archive that will last decades, you better put it on analog tape.

Hard Drives just don't last that long no matter what the file format is. And who knows if a computer 40 years in the future will have an operating system that even recognizes any of today's files? Or hell, if they will be able to interface with any of our hard drives, optical discs, or flash drives?

Reasonable. There is really no good way to ensure the longevity of anything without being subject to the potential of corruption, damage, incompatibility or just losing the information altogether.

At one point the documentation/collector side of me determined that I was going to archive everything I recorded and keep it tucked away on hard drives. After a year of remote recordings and a massive pile of assorted hardrives and storage devices I realized that hooking up all the firewire 400, 800, USB 1, 2, flash, discs, tape, adat, yadda yadda, was just as much work as rolling the tape machine out... Storing, sorting, checking and transferring all this information is beyond my reasonable expectations for what I can continue to do on the storage front.

Sometimes, I just gotta hope what I'm recording is gonna be stored on shiny new discs at everyone else's house haha.
Old 9th December 2010 | Show parent
  #196
Lives for gear
 
Verified Member
🎧 10 years
Quote:
Originally Posted by Cheebs Goat ➡️
The way I see it, if you really want an archive that will last decades, you better put it on analog tape.

Hard Drives just don't last that long no matter what the file format is. And who knows if a computer 40 years in the future will have an operating system that even recognizes any of today's files? Or hell, if they will be able to interface with any of our hard drives, optical discs, or flash drives?
Just upload it to a decent online data storage facility, they'll keep backups and update their hardware as the years go by. There are far more important (in terms of financial, political or social value) things than your songs being stored in digital form and you can bet the will and the way to keep them safe exists.

One of the beauties of digital is you can benefit from the technology and techniques related to the storage of billions of dollars worth of data, it's not just about the needs and the financial backing of audio professionals.
Old 9th December 2010 | Show parent
  #197
Lives for gear
 
🎧 10 years
Quote:
Originally Posted by Jon Hodgson ➡️
One of the beauties of digital is you can benefit from the technology and techniques related to the storage of billions of dollars worth of data, it's not just about the needs and the financial backing of audio professionals.
You guys might want to do some reading. Digital archiving is not as safe as it seems.

The Digital Ice Age - Popular Mechanics
CLIR Films
Old 9th December 2010 | Show parent
  #198
Gear Guru
 
UnderTow's Avatar
 
Verified Member
🎧 15 years
Quote:
Originally Posted by soulstudios ➡️
If I was dumb enough to not decode them for a client on delivery, well, I guess I'd be as smart as those people who wrote the above pdf.
If the archive is dependant on you being around to be translated from some less common format for the client to be able to read it, it is not a good method. So your method would not be acceptable by my IT standards.

Quote:
And that makes sense - for delivery - and even in the context of the document where they talk about 'archival', they are still talking about archival for delivery formats, not archival for studio sessions. It's a really stupid document, and even someone of limited IT skill can easily see how limited and pathetic the scope of their study is.
Wow. Strong words! Actually anyone with half a clue knows that you have to archive to a format that is likely to be easily readable in the future by anyone that might need to. (That last part is not predictable so using the most common format makes a lot of sense). Hence the recommendation to use wave files. There are other ways to insure data integrity than including it in the audio file format.

Quote:
As noted, you need to use your head, read the logic I've put forth above, and explain to yourself why wav is a bad idea for archiving.
BWF is a good format for archiving as any good archiving method has an archive and a backup. (It is never a backup if there is only a single copy). Having a backup, preferably off-site for anything important, you don't have to worry about data integrity as much. Personally I would use media that includes CRC like archiving to LTO tapes (or whatever) if the data was really important.

Alistair
Old 9th December 2010 | Show parent
  #199
Lives for gear
 
Verified Member
🎧 10 years
Quote:
Originally Posted by Cheebs Goat ➡️
You guys might want to do some reading. Digital archiving is not as safe as it seems.

The Digital Ice Age - Popular Mechanics
CLIR Films
Digital Archiving can be made as safe as you want it to be (well no system can ever be 100%, but you can choose how close you get), it's a case of understanding the failure rates and probabilities and designing your systems and procedures accordingly.

Which is why I suggested using an archiving service rather than doing it yourself, though there's nothing to prevent you from adding one or more safety layers of error detection and correction over and above what they have.

What isn't safe is just putting it onto a disk/tape whatever and putting that on the shelf for ten years.
Old 9th December 2010 | Show parent
  #200
Gear Guru
 
UnderTow's Avatar
 
Verified Member
🎧 15 years
Quote:
Originally Posted by Cheebs Goat ➡️
The way I see it, if you really want an archive that will last decades, you better put it on analog tape.
That isn't an archive as it is not identical to the original.

Quote:
And who knows if a computer 40 years in the future will have an operating system that even recognizes any of today's files? Or hell, if they will be able to interface with any of our hard drives, optical discs, or flash drives?
It is 100% guaranteed that there will be solutions for that. There is simply too much data from too many people and companies on all these current formats. There will always be companies offering services to migrate your data just as you can now have the 8mm or 16mm film or VHS video of your parent's wedding copied to DVD. Just make sure you use a common enough format, make several off-site copies and you should be OK.

There are literally billions of computers out there where you can plug a USB drive into but how many properly maintained and calibrated tape machines with the correct format are there still around today? Not many... How many will there be, including the people to operate them, in 40 years? That doesn't seem like a very safe bet at all.

Alistair
Old 12th December 2010 | Show parent
  #201
Lives for gear
 
Jerry Tubb's Avatar
 
Verified Member
🎧 15 years
Quote:
Originally Posted by Cheebs Goat ➡️
The way I see it, if you really want an archive that will last decades, you better put it on analog tape.*

Hard Drives just don't last that long no matter what the file format is.
Quote:
Originally Posted by Cellotron ➡️
I'd say with hard drive space so inexpensive these days (plus the availability of large capacity optical disc formats like BD-R) that it actually compromises the forward robustness of an archive by compressing the files into a format that requires further decoding. *For archives I think it's absolutely best to leave things in an uncompressed PCM file format. * AES recommendations in fact strictly proscribe bwf wav format for forward archiving.
I totally agree with Steve's post about hard drives and leaving audio uncompressed for archiving.
At roughly 100 USD / terrabyte it's a bargain.

I recently had an archive hard drive from 2005 die.
Luckily had all the data/audio backed up on DVD-R, which was easily retrieved & copied to a new HDD.

We've also had a few optical discs erode over the years, usually the cheaper stuff.

So redundancy in a variety of formats are the keys to survival.

We regularly see analog tapes that are (up to) 50+ years old.
Most easily play, except for those with aggravated sticky shed syndrome, usually classic Ampex 456.
So I'll say that it is a good non-digital format for archiving music.
Will tape machines still be around in 40 years(?), probably, as there are plenty of functional Victrolas around.

Cheers, JT
Old 13th December 2010 | Show parent
  #202
soulstudios
Guest
As stated, and proven, unless the storage medium you have has strong CRC built in (hard drive error-checking is not this), WAV and BWAV by themselves are completely inadequate.

And if you don't have strong CRC on the storage, you need it on the data format, otherwise you cannot determine data integrity, even with two or more separate stored copies.
I've gone into the logic on this, I won't **** around with it any more. Either you're smart and you get it, or you don't.

BTW, making future predictions and assuming that WAV will be readable in 20 years time whereas FLAC won't be is actually just wrong. You don't know what formats will be prevalent and readable with up-to-date software in 20 years time. Even the headers for WAV files have changed substantially in the last 20 years.
What you can make sure of is that YOU have a machine for decoding what you have - probably to RAW (as that's the only uncompressed format without headers).
In terms of long-term formats with CRC, zipping WAVs/BWAVS/RAW will work just fine. Zip's been around since the beginning, and is unlikely to disappear in our lifetimes, as it's still the default format for many different kinds of compressed media, not just audio - and that's despite better formats being available for 15 years.
(I suppose you could apply a similar logic to WAV, but in fact Zip has a better chance of longevity given it's status as a ubiquitous format for all data, and therefore changes or incompatibilities in the format are intolerable in the long run - WAV has a much more limited scope).

As for long-term (decades) storage of digital audio without constant replacement of hard drives in a raid config, the best (and cheapest) solution is digital tape (30 years keeptime) - the following explains:
Tape backup vs disk backup - AV5
Old 13th December 2010 | Show parent
  #203
Lives for gear
 
dcollins's Avatar
 
Verified Member
🎧 15 years
Quote:
Originally Posted by soulstudios ➡️
As stated, and proven, unless the storage medium you have has strong CRC built in (hard drive error-checking is not this), WAV and BWAV by themselves are completely inadequate.
Is this correct? I thought HDD had very robust CRC built right in, or today's super-dense data would not work. Isn't a typical error-rate like 1 in 10^15?

DC
Old 13th December 2010 | Show parent
  #204
soulstudios
Guest
Quote:
Originally Posted by dcollins ➡️
Is this correct? I thought HDD had very robust CRC built right in, or today's super-dense data would not work.
DC
Error detection and correction - Wikipedia, the free encyclopedia
Hard drives use block-level ECC codes (a weaker form of CRC) - this is not the same as file-based CRC checksums - it's not always storage, it's also errors creeping in when copying files (due to software, cabling, networking, ram or even the MB). I've seen data get munched due to a slightly old router - with no warning other than the files not working later. The best way is to have file-based CRC as well. That way, you know whether or not the file is alright.
m
Old 13th December 2010 | Show parent
  #205
Lives for gear
 
Verified Member
🎧 10 years
Quote:
Originally Posted by soulstudios ➡️
Error detection and correction - Wikipedia, the free encyclopedia
Hard drives use block-level ECC codes (a weaker form of CRC) - this is not the same as file-based CRC checksums - it's not always storage, it's also errors creeping in when copying files (due to software, cabling, networking, ram or even the MB). I've seen data get munched due to a slightly old router - with no warning other than the files not working later. The best way is to have file-based CRC as well. That way, you know whether or not the file is alright.
m
The error detection and correction systems, notably the Reed Solomon codes used in most hard disks are not "a weaker form of CRC". CRC is purely a checking system, not a correction one, and both can be made as robust as desired.

You can of course add a higher level of error detection or correction to your archiving format, so you could use an additional CRC to check for errors that got through the disk's ECC system, but then what would you do when you found there was a problem? You wouldn't know where it was in the file, or what to do about it, just that the file in the archive was now corrupt. If the aim is to make the storage more reliable then you should be adding another level of error correction, not just a check.

As for your comment on flac and zip being as good as WAV for archiving, you're wrong. The key to data robustness is to make any damage that happens to the stored data have the least effect on the resulting data. So if you have a stream of raw samples, and one bit is corrupted, you then get one sample which is corrupted, which can be heard and dealt with quite reasonably by interpolating.

On the other hand one bit wrong in a compressed format could result in hundreds or thousands of samples being wrong when it is decompressed, which you can't fix.

As for WAV having less guaranteed longevity compared to Zip, WAV is simply raw sample data with a header describing it, it's pretty much the simplest format for audio storage you could have (excluding just the raw sample data), you can describe it in a few lines, so I can't see the ability to read it vanishing (and even without the instructions it wouldn't take a smart person all that long to work it out, so long as he knew he was looking at something with samples in it). And what is it you store in that Zip? a WAV (or similar) file of course... so why even make the comparison.
Old 13th December 2010 | Show parent
  #206
Gear Guru
 
UnderTow's Avatar
 
Verified Member
🎧 15 years
Quote:
Originally Posted by Jon Hodgson ➡️
On the other hand one bit wrong in a compressed format could result in hundreds or thousands of samples being wrong when it is decompressed, which you can't fix.
Assuming it will decompressed at all...

Alistair
Old 17th December 2010 | Show parent
  #207
soulstudios
Guest
Quote:
Originally Posted by Jon Hodgson ➡️
The error detection and correction systems, notably the Reed Solomon codes used in most hard disks are not "a weaker form of CRC". CRC is purely a checking system, not a correction one, and both can be made as robust as desired.
A weaker form of CRC.
Because it only has block-based checksums. As noted.


Quote:
As for your comment on flac and zip being as good as WAV for archiving, you're wrong.
Nope. Not what I said. Better.


Quote:
The key to data robustness is to make any damage that happens to the stored data have the least effect on the resulting data. So if you have a stream of raw samples, and one bit is corrupted, you then get one sample which is corrupted, which can be heard and dealt with quite reasonably by interpolating.
The key to data robustness is to be able to detect errors IMHO.
I've already gone over this. You can't determine data accuracy between two backups without CRC in the format.
Look - if you want a partial-recovery scenario, by all means go with a pair of backups with RAW files, data description in the filename and separate calculated-checksum files. That would certainly satisfy both the goals of meaningful error detection and partial recovery without reliance on out-datable formats.


Quote:
As for WAV having less guaranteed longevity compared to Zip, WAV is simply raw sample data with a header describing it, it's pretty much the simplest format for audio storage you could have (excluding just the raw sample data), you can describe it in a few lines, so I can't see the ability to read it vanishing (and even without the instructions it wouldn't take a smart person all that long to work it out, so long as he knew he was looking at something with samples in it). And what is it you store in that Zip? a WAV (or similar) file of course... so why even make the comparison.
Because Zip has file-based checksums as opposed to block-based. Already mentioned RAW.
Suggest you go back and read what's been written. Cheers.
[email protected]
Old 18th December 2010 | Show parent
  #208
Lives for gear
 
Verified Member
🎧 10 years
Quote:
Originally Posted by soulstudios ➡️
A weaker form of CRC.
Because it only has block-based checksums. As noted.
It is not a simple detection, as CRC is, it is an error correction system. You're comparing apples to oranges.

Or perhaps more like comparing citrus sponge desserts with oranges

Additionally it's not weaker, it's stronger. The ratio of error detection bits to data bits on the hard disk or CD or other format is considerably higher than a file level CRC, it can therefore detect a far greater combination of errors. Yes the file level CRC will pick up things that have passed through the various block level detection and correction systems, but that's not because it's stronger, but rather because its strength is added to that of the systems preceding it.
Quote:
Nope. Not what I said. Better.
You're saying the compressed format is a better archival format? Well then I would say the opposite is the case, UNLESS the data space saved by using compression allows you to add extra redundancy which reduces the probability of data errors by an amount which counters the effect of those errors (in other words the worsened effect of errors that do occur due to the one to many effect of decompression is more than offset by the reduction in the number of errors achieved by adding extra data redundancy for error correction). In other words the compressed format on its own is a worse archival format, however the compressed form PLUS additional levels of error correction may be a better archival format in terms of robustness versus disk space.

You have a simple WAV file and a FLAC file on a hard disk, when you come to read them years later each has a one bit error. The FLAC reader tells you there's an error, but that doesn't fix the problem, you have hundreds or thousands of wrong samples, possibly no readable samples at all. The WAV file doesn't tell you there's an error, but only one sample is wrong and that's probably a notable click so you can find it and interpolate over it.

Which is the better archival format?
Quote:
The key to data robustness is to be able to detect errors IMHO.
detecting errors doesn't give data robustness, it just tells you that things have gone wrong. Once you know you have an error you have to be able to do something about it, that means you either have to correct it, or conceal it, the former requires data redundancy (whether that is achieved through having duplicates stored, or through adding error correction codes to the stored data), the latter requires errors in stored data to have the least effect in reproduced data.

Quote:
I've already gone over this. You can't determine data accuracy between two backups without CRC in the format.
You appear overestimate the power of CRCs. Statistically speaking the chance of two random combinations of data errors in two files being identical is considerably less than the chance of two random combinations of data errors resulting in the same CRC error, it's a simple mathematical probability thing, the data has 2^N possible states, where N is the number of data bits, the CRC has only 2^n possible states, where n is the number of CRC bits. N is almost invariably far greater than n.

That doesn't mean that CRCs aren't a good idea, just that you're more likely to detect errors by comparing two files than by checking the CRC on one file, of course since the two systems are independent combining the two is the most robust.
Quote:
Look - if you want a partial-recovery scenario, by all means go with a pair of backups with RAW files, data description in the filename and separate calculated-checksum files. That would certainly satisfy both the goals of meaningful error detection and partial recovery without reliance on out-datable formats.


Because Zip has file-based checksums as opposed to block-based. Already mentioned RAW.
Suggest you go back and read what's been written. Cheers.
[email protected]
Checksums do not an archival format make. As I pointed out in this post and my previous one, they let you know a problem has occured, they don't help you in fixing that problem.

If we want a distribution system, then a compressed format with a decent length CRC makes a lot of sense, both from data size and integrity points of view (you'll almost certainly detect any errors that have occured and you can rerequest the file.

However, if we're talking about archival, which we are, and so want to maximize the amount of correct data (sound in this case) we can read back in 100 years time, then error detection isn't good enough, we want data robustness, which means redundancy, probably keeping it in a raw sample format and adding error correction data to it in addition.

(Incidentally, you are aware that WAV files are to all intents and purposes a RAW sample format aren't you?)
📝 Reply

Similar Threads

Thread / Thread Starter Replies / Views Last Post
replies: 118 views: 21119
Avatar for davidicus
davidicus 7th February 2009
replies: 0 views: 2045
Avatar for _alex
_alex 3rd December 2009
replies: 2619 views: 475179
Avatar for S21
S21 22nd March 2021
replies: 79 views: 5514
Avatar for TobyB
TobyB 5 days ago
Post Reply

Welcome to the Gearspace Pro Audio Community!

Registration benefits include:
  • The ability to reply to and create new discussions
  • Access to members-only giveaways & competitions
  • Interact with VIP industry experts in our guest Q&As
  • Access to members-only sub forum discussions
  • Access to members-only Chat Room
  • Get INSTANT ACCESS to the world's best private pro audio Classifieds for only USD $20/year
  • Promote your eBay auctions and Reverb.com listings for free
  • Remove this message!
You need an account to post a reply. Create a username and password below and an account will be created and your post entered.


 
 
Slide to join now Processing…

Forum Jump
Forum Jump