Tag Archives: storage

Why not store numbers as diff of previous number?

This morning had a thought. We store numbers in a fixed sized memory space. So if we use four bytes to store numbers we would need eight bytes to store the numbers 5 and 6. But, what if we store 5 in four bytes and then 6 in two bit, as a delta? The bits can indicate +1, 0, -1., here an increment of the 5 by one. Larger increments would use more bits of course. Thus, we naturally get compression.

Subject areas: mobile computing, Internet of Things.

True, this wouldn’t work as a ‘live’ memory storage, the I/O would be complex. Or would it? In constrained devices such as Wearable Computing, for example, a smart watch, or in an Internet Of Things remote device, memory limitations may require compressed storage.

As usual, this is not a new idea. It is related to “Delta Encoding” or “Data differencing”. An interesting article on how delta coding could be used for compression is “Effective compression using frame-of-reference and delta coding”.

I have not seen this delta encoding memory approach mentioned anywhere yet.


Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Hierarchical Storage Management, from drive to cloud

Years ago I came upon the HSM concept. Is this now applicable in today’s networked world for personal use? I propose HSM can be extended to encompass the Cloud.

HSM is an enterprise data storage technique, a tiered storage. Data is automatically moved from expensive but fast storage systems like hard disk arrays to cheaper but slower systems like optical or tape drive.

Conceptually, HSM is analogous to the cache found in most computer CPUs, where small amounts of expensive SRAM memory running at very high speeds is used to store frequently used data, but the least recently used data is evicted to the slower but much larger main DRAM memory when new data has to be loaded. — http://en.wikipedia.org/wiki/Hierarchical_storage_management

In the consumer world we have in our PCs fast hard drives and on mobile devices fast flash memory. A simple two tiered system would migrate least recently used files from local media to the cloud. In the home or SOHO environment, a three-tiered system is possible. Solid-State Drives (SSD) of modest size could be the 1st tier, SATA disk the 2nd, and finally, Cloud services can provide the 3rd tier.

For example, you have a PDF on your system that is a great resource, but you haven’t used it in a few weeks. The HSM manager would take that file and move it to the cloud (secure, private, encrypted, …., of course). In its place, to allow access by the user, is a link to the HSM managed storage location. Next time you use the file it will be migrated back to the local storage (but now also backed up in the cloud).

This is really an application of “file virtualization“.

Note that the HSM in enterprise systems is not simply based on “files” but on the underlying storage mumbo jumbo (frames, and all that).

This approach could make the potential future Windows 8 ‘Storage Spaces’ be even more useful. On *nix OS this is possible to implement now. It probably already is.

Here is a conceptual demo. We’ll use a known cloud storage service provider like Dropbox. As far as I know, Dropbox does not offer HSM.

On your PC you set a property on various folders that makes them eligible for HSM monitoring. This could be accomplished using a GUI and drag&drop. The HSM will immediately copy the folders to the SSD on your system or the main hard drive, if the files are not already on the fastest subsystem. In the original location of the folders, a link to the new locations will be created (soft links?). The end user will not see any difference. Kind of like “web folders” or WebDAV protocol.

After a period of time, the local HSM monitor will record which files have not been used and invoke the Dropbox local service to stream the files to the cloud. All that remains on the file system are links to the remote files; storage space is reclaimed.

Feb 26, 2012: Another company will be competing with DropBox. As above it allows the user to designate specific folders to participate in cloud storage. Since remote files will be slower to access, this company will attempt to “predict” which files would be used more often. See this article.

Further Reading

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Windows 8 Storage Spaces?

Finally Windows may get a file system that adds features that have been around for years in Linux and Unix.

The state of the art in file system design could arguably be ZFS which is used in Solaris and a few other places.

Storage systems seem like very complex engineering systems. I wonder if Microsoft would have just been better off not reinventing the wheel and just licensing ZFS technology or something else? Just search for user horror stories regarding any of the storage improvements on Windows, from drive extender, encryption, WHS, etc.

And what ever happened to the object file system, the DB based file system?

Storage Spaces is somewhat like ZFS, although it has no deduplication and lacks other ZFS features. However, it is a start – and Microsoft will probably add features such as snapshots, replication, deduplication, and, maybe, compression. El Reg also thinks that there could be a Hyper-V virtualisation angle to this – and more is to come. — http://www.theregister.co.uk/2012/01/07/windows_8_storage_spaces/page2.html

Jan 17, 2012: Ah, so Microsoft is working Resilient File System (ReFS).

Microsoft mentioned all the important keywords when promoting ReFS with words like resilience, availability and the use of storage pools, something made popular by Sun’s – and now Oracle’s – ZFS.
Source: The Inquirer (http://s.tt/15gRk)

Further Reading

VOWS “Winter’s Grave” OFFICIAL video

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.