Enhancing the Reliability of Cloud Data through Identifying Data Inconsistency between Cloud Systems

The technologies of cloud and Internet of Things (IoT) have been extensively applied to many areas in our lives. The large amount of data collected from cloud applications and IoT devices is often maintained and processed by cloud systems. It is imperative to ensure the reliability of data stored on cloud systems when hardware failure occurs. The Apache Hadoop is one of the most popular cloud systems in academia and industry. To deal with hardware failure, its default file system, Hadoop Distributed File System (HDFS), keeps multiple duplicates of every data block it stores. Nevertheless, in the case of a total loss of a cloud, preserving multiple copies of data blocks still cannot assure their reliability. Hadoop provides a tool, distcp (distributed copy), to copy data from cloud to cloud to address this issue. Practically, data inconsistency caused by human mistakes or accidents may exist between data duplicated across clouds, which could significantly reduce the data reliability. Unfortunately, Hadoop does not provide efficient ways to locate discrepancies of duplicated data between clouds. Without data reliability, all corresponding data processing and decision making could be greatly affected. Therefore, it is crucial to detect the inconsistency between duplicated data stored on different clouds to uphold the data reliability. We designed and implemented a new scheme in Hadoop to efficiently identify data inconsistency between clouds. As a result, the reliability of data kept on clouds could be greatly enhanced.