Archivo de la etiqueta: S3

AWS S3 API – Throughput Notes


Some notes on settings to maximize throughput and increase parallelism while using S3 API: aws configure set default.s3.max_concurrent_requests 20 aws configure set default.s3.max_queue_size 10000 aws configure set default.s3.multipart_threshold 64MB aws configure set default.s3.multipart_chunksize 16MB aws configure set default.s3.max_bandwidth 50MB/s aws … Seguir leyendo

Publicado en Uncategorized | Etiquetado , | Deja un comentario

s3:// vs s3n:// vs s3a:// vs EMRFS


s3:// Apache Hadoop implementation of a block-based filesystem backed by S3. Apache Hadoop has deprecated use of this filesystem as of May 2016. s3n:// A native filesystem for reading and writing regular files on S3. S3N allows Hadoop to access … Seguir leyendo

Publicado en Uncategorized | Etiquetado , , , , | Deja un comentario