Archivo de la etiqueta: Linux

Create a really big file / Crear un archivo realmente grande

This is sometimes useful when playing with bigdata. Instead of a dd command and wait the file being created block by clock, we can run: $ fallocate -l 200G /mnt/reallyBigFile.csv It essentially “allocates” all of the space you’re seeking, but … Sigue leyendo

Publicado en Uncategorized | Etiquetado , , | 2 comentarios

Instalando Maven en instancia Amazon EC2

Maven es una herramienta de software para la gestión y construcción de proyectos Java Obtenemos maven: $ wget http://apache.saix.net/maven/maven-3/3.2.3/binaries/apache-maven-3.2.3-bin.tar.gz Descomprimimos: $ tar -xzvf apache-maven-3.2.3-bin.tar.gz Movemos la carpeta a un directorio de instalación permanente: $ sudo mv /home/ec2-user/apache-maven-3.2.3 /usr/local/maven Creamos link … Sigue leyendo

Publicado en Uncategorized | Etiquetado , , , | Deja un comentario

Adding a JAR path to Hadoop classpath

This is simple, but it is a frequent question: If we need to add some specific path pointing to a thirdparty library we can run a command like the following: $ export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/hadoop/.versions/Cascading-2.5-SDK/binary/cascading/*:/home/hadoop/.versions/Cascading-2.5-SDK/binary/cascading/lib/cascading-core/* Here I am adding two directories to … Sigue leyendo

Publicado en Uncategorized | Etiquetado , , | Deja un comentario

Mandus Momberg’s Blog ! – the beauty of BASH

I would like to share with you a new awesome blog from an awesome professional: http://blog.mandusmomberg.com/ And… as a first post, a nice one, about the beauty of BASH: http://blog.mandusmomberg.com/blog/2014/12/01/o-what-a-beautiful-bashing/ Enjoy !

Publicado en Uncategorized | Etiquetado , | Deja un comentario

vi sudo save with root permissions / grabar cambios con permisos de root

Just: :w !sudo tee % % is current file. !sudo tee calls tee with administrator privileges and writes to current file.  But not vi buffered file. That’s why you will see a warning like this when using the command: W12: … Sigue leyendo

Publicado en Uncategorized | Etiquetado , | Deja un comentario

MapReduce: Compression and Input Splits

This is something that always rise doubts: When considering compressed data that will be processed by MapReduce, it is important to check if the compression format supports splitting. If not, the number of map tasks may not be the expected. … Sigue leyendo

Publicado en Uncategorized | Etiquetado , , , | Deja un comentario

skb rides the rocket

[21068723.434629] xen_netfront: xennet: skb rides the rocket: 19 slots skb rides the rocket bug, this issue affects often Hadoop clusters. Each time I face it, I remember this excelent blog post from Brendan Gregg: http://www.brendangregg.com/blog/2014-09-11/perf-kernel-line-tracing.html Enjoy

Publicado en Uncategorized | Etiquetado , , | Deja un comentario