Archivo del Autor: hvivani

Acerca de hvivani

sysadmin, developer, RHCSA

Hadoop 1 vs Hadoop 2 – How many slots do I have per node ?

This is a topic that always rise a discussion… In Hadoop 1, the number of tasks launched per node was specified via the settings mapred.map.tasks.maximum and mapred.reduce.tasks.maximum. But this is ignored when set on Hadoop 2. In Hadoop 2 with … Sigue leyendo

Publicado en Uncategorized | Etiquetado , , | Deja un comentario

Hadoop useful commands

- Copy fromLocal/ToLocal from/to S3: $ bin/hadoop fs -copyToLocal s3://my-bucket/myfile.rb /home/hadoop/myfile.rb $ bin/hadoop fs -copyFromLocal job5.avro s3://my-bucket/input – Merge all the files from one folder into one single file: $ hadoop jar ~/lib/emr-s3distcp-1.0.jar –src s3://my-bucket/my-folder/ –dest s3://my-bucket/logs/all-the-files-merged.log –groupBy ‘.*(*)’ –outputCodec … Sigue leyendo

Publicado en Uncategorized | Etiquetado , | Deja un comentario

Generar clave publica desde clave privada

Necesito tener esto a mano: ssh-keygen -y -f ~/.ssh/test-key.pem > ~/.ssh/test-key.pem.pub Chequear previamente que los permisos en test-key.pem sean 600.

Publicado en Uncategorized | Etiquetado , | Deja un comentario

Hadoop: HDFS find / recover corrupt blocks

1) Search for files on corrupt files: A command like ‘hadoop fsck /’ will show the status of the filesystem and any corrupt files. This command will ignore lines with nothing but dots and lines talking about replication: hadoop fsck … Sigue leyendo

Publicado en Uncategorized | Etiquetado , , | Deja un comentario

Simple Java Telnet Port Scanner

It can be improved in many ways, but.. import java.io.*;  import java.net.*;  import java.util.*;  import java.util.TimerTask;  //import org.apache.commons.*;//import org.apache.commons.net.telnet.TelnetClient;  class Connectivity extends TimerTask  {      public static void main(String args[])      {          try          {              System.out.println(“Please enter ip … Sigue leyendo

Publicado en Uncategorized | Etiquetado | Deja un comentario

Testing Java Cryptography Extension (JCE) is installed

If JCE is already installed, you should see on that the jar files ‘local_policy.jar’ and ‘US_export_policy.jar’ are on $JAVA_HOME/jre/lib/security/ But, we can test it: import javax.crypto.Cipher; import java.security.*; import javax.crypto.*; class TestJCE { public static void main(String[] args) { boolean … Sigue leyendo

Publicado en Uncategorized | Etiquetado , | Deja un comentario

HDFS: Cluster to cluster copy with distcp

Este es el formato del comando distcp para copiar de hdfs a hdfs considerando cluster origen y destino en Amazon AWS: hadoop distcp “hdfs://ec2-54-86-202-252.compute-1.amazonaws.comec2-2:9000/tmp/test.txt” “hdfs://ec2-54-86-229-249.compute-1.amazonaws.comec2-2:9000/tmp/test1.txt” Mas informacion sobre distcp: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_7_2.html http://hadoop.apache.org/docs/r1.2.1/distcp2.html  

Publicado en Uncategorized | Etiquetado , , , | Deja un comentario