A Novel Way To Compact Data For More Efficient Storage And Transmission
A Novel Way To Compact Data For More Efficient Storage And Transmission
IT companies have spent billions of dollars creating ways to move data more efficiently and cheaply in an increasingly distributed world of on-premises
Last July, Atombeam unveiled its first product – Neurpac – which is central to the company’s data-as-codewords strategy of shrinking the size of data being transferred by an average of 75 percent in near real-time, resulting in an average four-times increase in effective bandwidth.
Neurpac is a cloud-based platform that includes an encoder, decoder, and trainer that can integrate into an organization’s cloud infrastructure. According to Atombeam, the trainer, using AI and machine learning, creates a set of small codewords – a codebook – with each codeword about three to ten bits in length. The codewords correspond to larger patterns found in a data sample – usually 64, 128, or 200 bits long. The codebook is installed on both ends of the communication link – at the sending and the receiving ends, connected by a satellite.
Yeomans says it is unlike data compression, which re-encodes information using fewer bits than the original. Because it’s on an “individual-only, one-at-a-time basis you can’t really use the compressed files for anything other than storing. If you said, ‘I’ve got a datalake full of this data and I want to do some research on it – I want to find out why this part is burning out on this tractor – just finding the data from those particular line of tractors is really hard, so 78 percent of the time the analysts spend is in finding and cleaning up the data; very little is actually analyzing it,” he says.
You're viewing a single thread.
But you definitely could do data mining and other things on compressed files.