Create multiple files at once with ‘touch’


Sometimes we might need to create thousands or millions of files at once.

This command will create the number specified in the range using touch:

touch bspl{00001..70000}.c
Anuncios
Publicado en Uncategorized | Etiquetado , | Deja un comentario

Debugging Java Threads


Which Java process is using most of the CPU:

 $ ps u -C java

Generate the Java thread dump:

$ jstack -l PId > PId-threads.txt

From the Java threads I can count:

$ awk '/State: / { print }' < PId-threads.txt  | sort | uniq -c
 450    java.lang.Thread.State: BLOCKED (on object monitor)
 240    java.lang.Thread.State: RUNNABLE
  47    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 294    java.lang.Thread.State: TIMED_WAITING (parking)
  31    java.lang.Thread.State: TIMED_WAITING (sleeping)
  42    java.lang.Thread.State: WAITING (on object monitor)
  62    java.lang.Thread.State: WAITING (parking)

From this, we search on the ones that are “waiting to lock”:

$ awk '/waiting to lock / { print }' < PId-threads.txt  | sort | uniq -c
   1     - waiting to lock <0x0000000600a027d8> (a org.apache.log4j.spi.RootLogger)
 294     - waiting to lock <0x0000000600f2e770> (a java.util.Hashtable)
  19     - waiting to lock <0x0000000600f36fc8> (a java.lang.Object)
   1     - waiting to lock <0x000000072f6e6708> (a org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler)

Main thread blocking here is: <0x0000000600f2e770>.

From the BLOCKED threads, we have many processes waiting for getConnection:

"Thread-132985" prio=10 tid=0x00007fec40784800 nid=0x662d waiting for monitor entry [0x00007fec18cd4000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1449)
    - waiting to lock <0x0000000600f2e770> (a java.util.Hashtable)

And many others waiting for Connection.close:

"IPC Client (738091550) connection to /10.66.2.38:9022 from hadoop" daemon prio=10 tid=0x00007fec41c1f800 nid=0x2dcc waiting for monitor entry [0x00007fec15da6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.hadoop.ipc.Client$Connection.close(Client.java:1135)
    - waiting to lock <0x0000000600f2e770> (a java.util.Hashtable)

All of them are BLOCKED by <0x0000000600f2e770>.

Opening other PId’s we can find the Java thread that has the lock on this resource. The culprit will look like:

"Thread-133346" prio=10 tid=0x00007fec40ac7800 nid=0x747e runnable [0x00007fec17cc4000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.Thread.<init>(Thread.java:234)
    at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:396)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1452)
    - locked <0x0000000600f2e770> (a java.util.Hashtable)
    at org.apache.hadoop.ipc.Client.call(Client.java:1381)
    at org.apache.hadoop.ipc.Client.call(Client.java:1363)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:215)
    at com.sun.proxy.$Proxy42.getApplicationReport(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:163)
    at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
    at com.sun.proxy.$Proxy43.getApplicationReport(Unknown Source)
    at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:291)
    at org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:294)
    at org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:152)
    at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:319)
    - locked <0x0000000733033970> (a org.apache.hadoop.mapred.ClientServiceDelegate)
    at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
    at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
    at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
    at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:311)
    - locked <0x0000000733032e20> (a org.apache.hadoop.mapreduce.Job)
    at org.apache.hadoop.mapreduce.Job.getJobState(Job.java:347)
    at org.apache.hadoop.mapred.JobClient$NetworkedJob.getJobState(JobClient.java:295)
    - locked <0x0000000733032e10> (a org.apache.hadoop.mapred.JobClient$NetworkedJob)
    at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:244)
    at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:547)
    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:426)
    at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)

This kind of Blocked Status is not technically a Deadlock, but just a thread blocking other threads: locked status for a resource while many other objects start queuing waiting for the same resource.

Publicado en Uncategorized | Etiquetado , | Deja un comentario

Masa para Pionono casera


Ingredientes:

Azucar 40 g (1/3 taza)
Huevos 4 Unidades
Esencia de vainilla 1 cda
Harina Leudante 40 g (1/3 taza)
miel 1 cda
Sal fina 1/2 cda

Preparacion:

Batimos los huevos con el  azúcar y  la miel hasta  lograr un punto letra (unos 10 min). Luego, agregamos  la sal y  la esencia de vainilla.

Tamizamos la harina e integrar al batido, dentro de la batidora funcionando a baja velocidad.

Enmantecar una placa de aproximadamente 50 por 30 cm y colocar papel para horno o manteca tambien enmantecado sobre la misma.

pionono00
Colocar la masa sobre el papel enmantecado en la placa.

pionono01
Llevamos al horno a una temperatura de 190-200 °C  durante 8 a 10 minutos.

Retiramos del horno e inmediatamente desmoldamos sobre una rejilla  y cubrimos con la placa. Cuando enfríe retiramos el papel.

pionono02

Publicado en Cooking, Uncategorized | Deja un comentario

La Catanga


Corria el año 1946 cuando mis abuelos Armando Vivani (paterno) y Eduardo Divito (materno), se juntaban despues de trabajar en la Gomeria “Merlo” en la ciudad de 9 de Julio – Buenos Aires.

Alli se dedicaban a preparar La Catanga, un auto de carreras que corria Armando Vivani los fines de semana en circuitos locales de tierra.

la_catanga_00

La Catanga tenia motor de Ford “T” y se pasaban gran parte de la tarde y noche rebajando la tapa de cilindros y multiples de escape y de admision “a lima”.

la_catanga_01

Ese viejo modelo “T” llegaba a levantar 100/120 Km/h en las rectas !

la_catanga_02

Publicado en Uncategorized | Etiquetado , | 1 Comentario

Creating Bigtop patches


To contribute to Bigtop project, we need to submit a patch.

We should follow this process for managing our proposed contributions:

  1. Create a Jira ticket with the description of the problem. (Note: the ticket should be Minor priority for most things, and only Major if it is fixing a bug that prevents something from working as expected; also the component will be “deployment” for new charms or bundles, as well as for changes to the puppet manifests.)
  2. Create branch in /hvivani/bigtop fork named BIGTOP-XXXX, where XXXX is the Jira ticket number, e.g. BIGTOP-2417.
  3. Commit the change with a commit message of the form BIGTOP-XXXX: message, where message must match the title of your JIRA ticket. Then push it to the branch.
  4. Open a pull request from your branch to the upstream Apache Bigtop repository, e.g. PR 138. Again, ensure the PR title exactly matches the title of your Jira ticket prefixed by the BIGTOP-XXXX ticket number.
  5. Refresh your Jira ticket. You should see your GitHub PR linked to the ticket. You should also see a comment by ASF GitHub Bot with information about the PR which includes a link to the PR’s patch file, which is the PR URL with .patch appended to it. For example, BIGTOP-2417 contains a link to https://github.com/apache/bigtop/pull/138.patch.
  6. Once you do it, click the Submit Patch button in the ticket to inform committers that this ticket has a Patch Available.
Publicado en Uncategorized | Etiquetado , | Deja un comentario

Building and Deploying Apache Bigtop Applications


If you want to explore how to build an application for Apache Bigtop and then deploy it using EMR, have a look at this blog post I wrote for Amazon AWS:

bigtop_1_1

http://blogs.aws.amazon.com/bigdata/post/TxNJ6YS4X6S59U/Building-and-Deploying-Custom-Applications-with-Apache-Bigtop-and-Amazon-EMR

Publicado en Mis Publicaciones, Uncategorized | Etiquetado , , , | Deja un comentario

Java change default version / cambiar la version Java por defecto


If we have more than one Java version installed on your Linux server (Redhat flavor) you can change defaults using ‘alternatives’ command:

[hadoop@ip-172-31-36-252 ~]$ sudo /usr/sbin/alternatives --config java

There are 2 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
   2           /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java

Enter to keep the current selection[+], or type selection number: 2
[hadoop@ip-172-31-36-252 ~]$ sudo /usr/sbin/alternatives --config java

There are 2 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
*  1           /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
 + 2           /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java

Enter to keep the current selection[+], or type selection number:
[hadoop@ip-172-31-36-252 ~]$

 

Publicado en Uncategorized | Etiquetado , , | Deja un comentario