Pig Latin Compiler for Apache Spark / Flink
The goal of this project is to build a compiler for the Pig Latin dataflow language on modern data analytics platforms such as Apache Spark and Apache Flink. The project is not intented as a replacement or competitor of the official Pig compiler for Hadoop or its extensions such as PigSpork. Instead we have the following goals:
- We want to build a compiler from scratch that compiles natively to the Scala-based Spark/Flink API and avoids all the stuff needed for MapReduce/Hadoop.
- Though, we are aiming at being compatible to the original Pig compiler we plan to integrate extensiblity features allowing to define and use user-defined operators (not only UDFs) and in this way being able to integrate extensions for graph processing or machine learning.
Clone & Update
Simply clone the git project.
To build the project, in the project directory invoke
This will build the (main) Pig compiler project as well as the shipped backends.
There are several test cases included which should be passed: unit
tests can be executed by
sbt test, integration tests which compile
and execute Pig scripts on Spark or Flink are executed by
Note that building the compiler requires the most recent Spark and Flink jars, but they will be downloaded by sbt automatically.
If you want to use the compiler with the frontend scripts (see below), you have to build an assembly:
We provide a simple wrapper script for processing Pig scripts. Just call it with
piglet --master local --backend spark your_script.pig
To run this script you have to specify the full path to the platform distribution jar in the environment
SPARK_JAR for Spark (e.g.
spark-assembly-1.5.2-hadoop2.6.0.jar) and in
flink-dist_2.11-1.0.0.jar) for Flink.
For Flink you also have to provide the path to the
conf directory in
An example for Spark could look like the following:
export SPARK_JAR=/opt/spark-1.6.0/assembly/target/scala-2.11/spark-assembly-1.6.0-hadoop2.6.0.jar piglet --master local --backend spark your_script.pig
The equivalent for Flink would be:
export FLINK_JAR=/opt/flink-1.0.0/build-target/lib/flink-dist_2.11-1.0.0.jar export FLINK_CONF_DIR=/opt/flink-1.0.0/build-target/conf piglet --master local --backend flink your_script.pig
Note, that both for Spark and Flink you need a version built for Scala 2.11 (see e.g.
Spark doc and Flink doc)
and the same version used for building must also be used for execution. For Flink you have to run the start script found in the bin directory (e.g.
/opt/flink-1.0.0/build-target/bin/start-local.sh) before executing scripts.
The following options are supported:
--master mspecifies the master (local, yarn-client, yarn)
--compilecompile and build jar, but do not execute
--outdir dirspecifies the output directory for the generated code
--backend bspecifies the backend to execute the script. Currently, we support
spark(Apache Spark in batch mode)
sparks: Apache Spark Streaming
flink: Apache Flink in batch mode
flinks: Apache Flink Streaming
mapreduce: Apache Hadoop (by simply passing the script to the original Pig compiler)
--params key=value, ...
--show-planPrint the resulting dataflow plan
--show-statsShow execution runtimes for (some) Piglet methods
--keepKeep generated files
--sequentialIf more than one input script is provided, do not merge them but execute them sequentially
--backend-args key=value, ...
In addition, you can start an interactive Pig shell similar to Grunt:
piglet --interactive --backend spark
where Pig statements can be entered at the prompt and are executed as soon as
STORE statement is entered. Furthermore, the schema can be printed using
Piglet can also be run as a Docker container. However, the image is not yet on DockerHub, so it has to be built manually:
sbt clean package assembly docker build -t dbis/piglet .
Currently, the Docker image supports the Spark backend only.
To start the container, run:
docker run -it --rm --name piglet dbis/piglet
This uses the container's entrypoint which runs piglet. The above command will print the help message.
You can start the interactive mode, using
-i option and enter your script.
docker run -it --rm --name piglet dbis/piglet -b spark -i
Alternatively, you can add your existing files into the container by mounting volumes and run the script in batch mode:
docker run -it --rm --name piglet -v /tmp/test.pig:/test.pig dbis/piglet -b spark /test.pig
As mentioned before, the container provides an entrypoint that executes piglet. In case you need a bash for that container, you need to overwrite the entrypoint:
docker run -it --rm --name piglet --entrypoint /bin/bash dbis/piglet
To configure the program, we ship a configuration file. When starting the program for the first time, we will create our program home directory in your home directory and also copy the configuration file into this directory.
More specifically, we will create a folder
~/.piglet (on *nix like systems) and copy the configuration file
application.conf to this location.
If you update Piglet to a new version and the configuration file still exists from a previous version, a configuration exception might occur because we cannot find new configuration keys introduced by the new Piglet version in the existing config file. In such cases, you can start piglet with the
--update-config) option. This will force the override of your old configuration (make sure you have a backup if needed). Alternatively, you can simply remove the existing
~/.piglet/application.conf. This will also trigger the copy routine.
We use the Typesafe Config library.
As stated before, we support various backends that are used to execute the scripts. You can add your own backend by creating a jar file that contains the necessary configuration information and
classes and adding it to the classpath (e.g. using the
More detailed information on how to create backends can be found in backends.md
- Details on the supported language features (statements, functions, etc.) are described here.
- Documentation on how to setup integration with Zeppelin.
- We use the Scala testing framework as well as the scoverage tool
for test coverage. You can produce a coverage report by running
sbt clean coverage test. The results can be found in