Alois Cochard's Blog: Networking: Ensure data integrity in a decentralized storage system

Posted by alois on January 27, 2009 at 12:57 AM | Comments (1)

Introduction
I'm actually developing on my free time a decentralized computing platform in java, I'm in the early stage, defining concepts and building the foundation.
This platform is target to be available to anyone connected to internet, so data integrity must be granted to avoid corruption of data from someone else, I gonna describe the concept I created to avoid any corruption of data in a high decentralized storage system.

Definition
The decentralized platform consist of (highly simplified):

Network service to handle data request (from remote peer or from local clients)
Storage service to store data and data checksum
Communication service to handle request/respond between peers

The source identifier, is the peer identifier of the peer who provided the data.

Integrity checking scenario
Network service receive data request:

ask storage if data exist (using data id)

if not, request data from other peers
save data inside storage with source identifier

compute data checksum
request data checksum from other peers

if data cheksum come from same peer as data, request data cheksum from other peer

compare both data checksums

if checksums are different:

increment integrity failure counter

if integrity failure counter exceed defineed limit

delete data and data checksum from storage

exit scenario without returning data

if checksums are equal:

reset integrity failure counter

save data checksum with source identifier inside storage
return data

Conclusion
Please feel free to comment it, especially if you know better solution or any software who have implemented a such concept efficiently.

UniNet experimental project was close, you can find informations about my new project here.

Bookmark blog post:

del.icio.us

Digg

DZone

Furl

Comments

Comments are listed in date ascending order (oldest first) | Post Comment

Have a look at Freenet.

They have something that seems quite similar to what you are describing in terms of a distributed storage system.

Posted by: here_he_is_javanet on January 28, 2009 at 03:48 PM

Powered by
Movable Type 3.01D

java.net RSS Feeds