Projeto e análise de desempenho de um algoritmo iterativo para grandes grafos em um ambiente distribuído

Abstract

Currently, large volumes of data are generated and collected through sensors, devices, and social networks. The ability to handle large masses of data has become an important factor for the success of many organizations, increasingly requiring the use of parallel and distributed processing. To help developers design distributed programs, there are a number of tools (frameworks), such as Apache Hadoop and Spark. These frameworks provide various configuration parameters (for example, Hadoop has more than 200) and assigning optimized values to all of them is no trivial task. This work investigates the influence of these parameters on Apache Hadoop performance, using the HEDA algorithm, an iterative algorithm that calculates centrality metrics in large graphs. The execution of HEDA in a complex network is extremely important because there are several measures of centrality that determine the importance of a vertex within the graph. It was observed that in some cases the improvement in execution time reached approximately 80% applying the values proposed by this work to the Hadoop configuration parameters. In addition, it was possible to increase processor utilization by five times and greatly improve scalability. The work also presents the methods applied to prepare, execute and analyze the experiments, which may aid in further studies.