Archivo mensual: junio 2015

yarn: execute a script on all the nodes in the cluster


This is more Linux script related, but, sometimes we have a Hadoop (YARN) cluster running and we need to run a post install script or activity that executes on all the nodes in the cluster: for i in `yarn node … Seguir leyendo

Publicado en Uncategorized | Etiquetado , , , , | Deja un comentario

HDFS: changing the replication factor


The replication factor is a property that can be set in the HDFS configuration file that will allow you to adjust the global replication factor for the entire cluster. For each block stored in HDFS, there will be n – … Seguir leyendo

Publicado en Uncategorized | Etiquetado , | Deja un comentario

get the size of an Amazon S3 bucket folder / obtener el tamaño de una carpeta en S3


aws s3 ls s3://my-bucket/folder –recursive | awk ‘BEGIN {total=0}{total+=$3}END{print total/1024/1024″ MB”}’

Publicado en Uncategorized | Etiquetado , | Deja un comentario