Archivo de la etiqueta: HDFS

The replication factor is a property that can be set in the HDFS configuration file that will allow you to adjust the global replication factor for the entire cluster. For each block stored in HDFS, there will be n – … Seguir leyendo →

– Copy fromLocal/ToLocal from/to S3: $ bin/hadoop fs -copyToLocal s3://my-bucket/myfile.rb /home/hadoop/myfile.rb $ bin/hadoop fs -copyFromLocal job5.avro s3://my-bucket/input – Merge all the files from one folder into one single file: $ hadoop jar ~/lib/emr-s3distcp-1.0.jar –src s3://my-bucket/my-folder/ –dest s3://my-bucket/logs/all-the-files-merged.log –groupBy ‘.*(*)’ –outputCodec … Seguir leyendo →