Archivo de la categoría: Uncategorized

monitoring HTTP requests on the fly


Install httpry: sudo yum install httpry or $ sudo yum install gcc make git libpcap-devel $ git clone https://github.com/jbittel/httpry.git $ cd httpry $ make $ sudo make install then run: sudo httpry -i eth0 Output will be like: httpry version … Seguir leyendo

Publicado en Uncategorized | Etiquetado , | Deja un comentario

FileInputFormat vs. CombineFileInputFormat


When you put a file into HDFS, it is converted to blocks of 128 MB. (Default value for HDFS on EMR) Consider a file big enough to consume 10 blocks. When you read that file from HDFS as an input … Seguir leyendo

Publicado en Uncategorized | Etiquetado , , , | Deja un comentario

Start Hive in Debug Mode


Never go out without it: hive –hiveconf hive.root.logger=DEBUG,console

Publicado en Uncategorized | Etiquetado , | Deja un comentario

mac: installing homebrew


To download install Homebrew run the install script on the command line: ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)”

Publicado en Uncategorized | Etiquetado | Deja un comentario

get the driver’s IP in spark yarn-cluster mode


In cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application. In client mode, the driver runs in the client process, … Seguir leyendo

Publicado en Uncategorized | Etiquetado , | Deja un comentario

Consider boosting spark.yarn.executor.memoryOverhead


This is a very specific error related to the Spark Executor and the YARN container coexistence. You will typically see errors like this one on the application container logs: 15/03/12 18:53:46 WARN YarnAllocator: Container killed by YARN for exceeding memory … Seguir leyendo

Publicado en Uncategorized | Etiquetado , , | Deja un comentario

Compile Scala program with sbt


Install sbt: curl https://bintray.com/sbt/rpm/rpm | sudo tee /etc/yum.repos.d/bintray-sbt-rpm.repo sudo yum install sbt Compile & Build Place build.sbt and the .scala program in the same directory and run: sbt package    

Publicado en Uncategorized | Etiquetado | Deja un comentario