Archivo de la categoría: Uncategorized

monitoring HTTP requests on the fly


Install httpry: sudo yum install httpry or $ sudo yum install gcc make git libpcap-devel $ git clone https://github.com/jbittel/httpry.git $ cd httpry $ make $ sudo make install then run: sudo httpry -i eth0 Output will be like: httpry version … Seguir leyendo

Publicado en Uncategorized | Etiquetado , | Deja un comentario

Control Characters on vi Linux editor


Show hidden/control characters on vi: :set list   Hide hidden/control characters on vi: :set nolist   Replace hidden/control characters on vi: :%s/^M//g :%s/.$//g  

Publicado en Uncategorized | Etiquetado , | Deja un comentario

FileInputFormat vs. CombineFileInputFormat


When you put a file into HDFS, it is converted to blocks of 128 MB. (Default value for HDFS on EMR) Consider a file big enough to consume 10 blocks. When you read that file from HDFS as an input … Seguir leyendo

Publicado en Uncategorized | Etiquetado , , , | Deja un comentario

Start Hive in Debug Mode


Never go out without it: hive –hiveconf hive.root.logger=DEBUG,console

Publicado en Uncategorized | Etiquetado , | Deja un comentario

mac: installing homebrew


To download install Homebrew run the install script on the command line: ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)”

Publicado en Uncategorized | Etiquetado | Deja un comentario

get the driver’s IP in spark yarn-cluster mode


In cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application. In client mode, the driver runs in the client process, … Seguir leyendo

Publicado en Uncategorized | Etiquetado , | Deja un comentario

Consider boosting spark.yarn.executor.memoryOverhead


This is a very specific error related to the Spark Executor and the YARN container coexistence. You will typically see errors like this one on the application container logs: 15/03/12 18:53:46 WARN YarnAllocator: Container killed by YARN for exceeding memory … Seguir leyendo

Publicado en Uncategorized | Etiquetado , , | Deja un comentario