Getting S3A working correctly on Spark can be a frustrating experience; using S3 as a cost effective semi-solution for HDFS pretty much requires it because of various performance [speed] improvements. There are bits and pieces of what you need to know scattered across the Internet. This is what I’ve distilled, as well as the stack […]
Category: Blog
![Option[A] interpolator in Scala](https://deploymentzone.com/wp-content/uploads/2015/08/images.png)
Option[A] interpolator in Scala
Basically I have a slew of optionals coming in as JSON, because as we all know even the most strict schema has edge case missing values. When trying to turn Option[A] to string with the standard interpolator it looks like: None,Some(“us”),None,Some(14) So here’s a better way that uses an interpolator for Option[A] which yields code […]
Anorm, Redshift and IDENTITY columns
At the time of this writing Redshift does not support the PostgreSQL RETURNING fields syntax nor JDBC Statement.RETURN_GENERATED_KEYS. If you attempt something like: SQL”””INSERT INTO …”””.executeInsert(long(“id”)) you’ll be rewarded with: java.sql.SQLFeatureNotSupportedException: [Amazon][JDBC](10220) Driver not capable. To get around this you need to wrap things inside a transaction and get the maximum ID value from the […]
Amazon Redshift, PostgreSQL and JDBC driver woes
Amazon Redshift is PostgreSQL 8.x compatible though Amazon also offers JDBC drivers specific to Redshift which may result in better error messages? (I’ve dealt with enough cryptic errors from Redshift JDBC drivers that I just don’t believe this.) In any case, attempting to use PostgreSQL drivers in the same JVM as Redshift JDBC drivers may […]
Spark, serialization and case classes
In Spark if you try to do anything cute [read: inheritance from an abstract class] with case classes, i.e., abstract class PairReducable[A <: Keyable, B, P <: PairReducable[A, B, P]] (val item: A, val amount: B) extends Product2[A, B] with Amount[B] { override val _1 = item override val _2 = amount def reduce(other: PairReducable[A, […]
nodejs on Amazon ElasticBeanstalk failed installation problems
We have a nodejs application where I work which all of a sudden had problems getting provisioned using Amazon’s ElasticBeanstalk. Certainly something must have changed and after digging a bit we linked the problem to a node-gyp issue that doesn’t appear to be slated for a fix. Examining our /opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh file on a Beanstalked instance […]
Spark, EC2 and easy spark-shell deployment
If you have a Scala Spark project and you need to deliver the resulting über-jar to an Amazon EC2 Spark master and then execute spark-shell with that jar on the classpath then I have a bash script for you. #!/usr/bin/env bash set -o nounset set -o errexit readonly default_env=”${0#*-}” readonly ENV=${1:-$default_env} readonly JAR_NAME=”your-analytics.jar” readonly UPLOAD_JAR=`dirname […]
Spark and merged CSV files
Spark is like Hadoop – uses Hadoop, in fact – for performing actions like outputting data to HDFS. You’ll know what I mean the first time you try to save “all-the-data.csv” and are surprised to find a directory named all-the-data.csv/ containing a 0 byte _SUCCESS file and then several part-0000n files for each partition that […]
Whitescreen when upgrading OSX to Yosemite
During the upgrade to OSX Yosemite on my MacPro I was reaching a white screen with a movable cursor but no other activity. (This screen would appear partway through the progress meter under the Apple logo, after a screen flash.) This is not the same as “About a minute remaining…” for hours (see below). Here’s […]
Akka fatal crash after updating
Since this is the second time I’ve ran into this problem – because its the second time I’ve done a nightmarish Play minor version upgrade – I’m posting this so it won’t bite me again. When I start seeing crashes during my tests that look something like this: java.lang.AbstractMethodError: null at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210) ~[akka-actor_2.10-2.3.4.jar:na] at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172) […]
netty and the Datastax Cassandra driver
I recently had: java.lang.NoSuchMethodError: org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.<init>(IIIIIZ)V happening to me when I attempted to container:start (from xsbt-web-plugin) within an sbt session on a Spray/spray-servlet application. I was going to great lengths to search my $CLASSPATH and dependency’s transitive dependencies to determine what was relying on an older version of Netty causing the Datastax driver to fail. I […]
JRuby, Rails, Tomcat, ElasticBeanstalk and environment variables
I’ve been refactoring a Rails 3 application with a spaghetti configuration mess into something more in line with The Twelve Factor App where I’m using the dotenv gem for configuration. (One day we may want to use a service like Consul for configuration. That day is not yet today.) I also wanted to see what […]
Java + bash: search a bunch of jars for a class
There’s a lot of really bad examples floating around out there to do this in search results. This shell function recursively searches starting at the current directory using the regular expression specified as the only argument for search criteria. It does restrict the search to only those files with names ending with .class (to prevent […]
Beating a Poison Ivy Reaction
“Why are there so many of these vines climbing my trees this year? Well, at least they are very easy to pull down. This shouldn’t take too long to clear.” Brilliant idea, especially a couple hours in when I remembered “isn’t there something about leaves of three – leave it be?” I am undergoing my […]
recursively sed on OSX
When using sed to perform an in-place edit on a directory I seem to always forget that zero length arguments require a space between the argument (an empty string, that is ” or “”) and the flag. So this is incorrect: $ grep -rl Trafficland . | xargs sed -i”” -e “s/Trafficland/TrafficLand/g” You end up […]
ASP.NET vNext
The original, which I blatantly stole from PA. The article which makes ASP.NET vNext look great. The links that prove its a trap.
Uploading large files over public wifi
Anyone who has worked out of a coffee shop and has had to upload large files to a remote system has probably experienced the initial burst speed of the upload followed by a disheartening throttling down of bandwidth to a crawl. As I write a lot of software that uses the JVM I produce some […]
AWS instances and java.net.UnknownHostException
This problem can be solved by adding the host name to /etc/hosts or overriding the hostname when the DHCP client gets a lease. On AWS if you’re autoscaling neither solution is palatable. Instead you can specify a script like the following as User Data (under “Advanced” if you’re using the AWS console to launch your […]
spray-routing and getFromResourceDirectory
In order to serve static files from a jar, chances are you’ll want to use the pathPrefix directive and not the path directive when using getFromResourceDirectory in spray-routing. Here I’ve created a directory structure: src/main/resources/admin ├── angular.js ├── camera.png └── index.html 0 directories, 3 files Now I want all requests for “/admin” to be answered […]
Mavericks: Day 2 with MacVim, Python, and crashing
I require MacVim on OSX to work productively. I also use a couple of plugins that require Python – YouCompleteMe and powerline. After upgrading to Mavericks I was having problems since my brew’d Python was replaced with the OSX system python. It felt like a clean install of MacVim was in order anyway. I experienced […]
Verifying unexpected messages in an Akka Actor
I have an actor that uses become to change from an inactive to active state, and handles messages differently based on such state changes. I want to be able to test that the actor behaves correctly in each state by ignoring messages that are invalid for a given state. Initially I incorrectly believed that UnhandledMessage […]
Correcting “One or more requested classes are not Suites” in IntelliJ/Scala
I’ve run into a couple problems in IntelliJ 12.x when working on Scala projects with ScalaTest. The test runner normally works wonderfully, but rarely I’ll get into a state where, when attempting to run a test configuration for a Suite I get an error message: One or more requested classes are not Suites This occurs […]
I wrote a book on RSpec and TDD
I recently had a book published through Packt Publishing titled Instant RSpec Test-Driven Development How-to. The book is targeted at novice to intermediate software developers who want to quickly become productive with test-driven development methodology (TDD). I don’t dwell much in theory, instead delivering compact how-tos and practical examples, though admittedly the book’s length – […]
Typesafe Config and maps in Scala
While not apparent from the quick start samples in the README its pretty painless to derive a map of values from HOCON. In the domain I’m modeling there are some devices identified by a name that represent a UDP socket address. I end up with a resources/reference.conf that contains the following: decoders = [ { […]