2 Getting Started - Reference Documentation
Authors: The Whole GPars Gang
Version: 1.0-SNAPSHOT
Table of Contents
2 Getting Started
Let's set out a few assumptions before we get started:- You know and use Groovy and Java: otherwise you'd not be investing your valuable time studying a concurrency and parallelism library for Groovy and Java.
- You definitely want to write your codes employing concurrency and parallelism using Groovy and Java.
- If you are not using Groovy for your code, you are prepared to pay the inevitable verbosity tax of using Java.
- You target multi-core hardware with your code.
- You appreciate that in concurrent and parallel code things can happen at any time, in any order, and more likely with than one thing happening at once.
- Code-level helpers Constructs that can be applied to small parts of the code-base such as individual algorithms or data structures without any major changes in the overall project architecture
- Parallel Collections
- Asynchronous Processing
- Fork/Join (Divide/Conquer)
- Architecture-level concepts Constructs that need to be taken into account when designing the project structure
- Actors
- Communicating Sequential Processes (CSP)
- Dataflow
- Data Parallelism
- Shared Mutable State Protection Although about 95% of current use of shared mutable state can be avoided using proper abstractions, good abstractions are still necessary for the remaining 5% use cases, when shared mutable state cannot be avoided
- Agents
- Software Transactional Memory (not fully implemented in GPars as yet)
2.1 Downloading and Installing
GPars is now distributed as standard with Groovy. So if you have a Groovy installation, you should have GPars already. The exact version of GPars you have will, of course, depend of which version of Groovy. If you don't already have GPars, and you do have Groovy, then perhaps you should upgrade your Groovy!If you do not have a Groovy installation, but get Groovy by using dependencies or just having the groovy-all artifact, then you will need to get GPars. Also if you want to use a version of GPars different from the one with Groovy, or have an old GPars-less Groovy you cannot upgrade, you will need to get GPars. The ways of getting GPars are:- Download the artifact from a repository and add it and all the transitive dependencies manually.
- Specify a dependency in Gradle, Maven, or Ivy (or Gant, or Ant) build files.
- Use Grapes (especially useful for Groovy scripts).
The GPars Artifact
As noted above GPars is now distributed as standard with Groovy. If however, you have to manage this dependency manually, the GPars artifact is in the main Maven repository and in the Codehaus main and snapshots repositories. The current release version (0.12) is in the Maven and Codehaus main repositories, the current development version (1.0-SNAPSHOT) is in the Codehaus snapshots repository. To use from Gradle or Grapes use the specification:"org.codehaus.gpars:gpars:0.12"
"org.codehaus.gpars:gpars:1.0-SNAPSHOT"
<dependency> <groupId>org.codehaus.gpars</groupId> <artifactId>gpars</artifactId> <version>0.12</version> </dependency>
Transitive Dependencies
GPars requires that two dependencies, namely jsr166y and extra166y (artifacts from the JSR-166 Project ), be on the classpath for GPars using programs to compile and execute. Release versions of these artifacts are in the main Maven and Codehaus repositories. Development versions of the artifacts are available in the Codehaus snapshots repository. Using Gradle or Grapes you would use dependency specifications:"org.codehaus.jsr166-mirror:jsr166y:1.7.0" "org.codehaus.jsr166-mirror:extra166y:1.7.0"
<dependency> <groupId>org.codehaus.jsr166-mirror</groupId> <artifactId>jsr166y</artifactId> <version>1.7.0</version> </dependency> <dependency> <groupId>org.codehaus.jsr166-mirror</groupId> <artifactId>extra166y</artifactId> <version>1.7.0</version> </dependency>
2.2 A Hello World Example
Once you are setup, try the following Groovy script to test that your setup is functioning as it should.import static groovyx.gpars.actor.Actors.actor/** * A demo showing two cooperating actors. The decryptor decrypts received messages * and replies them back. The console actor sends a message to decrypt, prints out * the reply and terminates both actors. The main thread waits on both actors to * finish using the join() method to prevent premature exit, since both actors use * the default actor group, which uses a daemon thread pool. * @author Dierk Koenig, Vaclav Pech */def decryptor = actor { loop { react { message -> if (message instanceof String) reply message.reverse() else stop() } } }def console = actor { decryptor.send 'lellarap si yvoorG' react { println 'Decrypted message: ' + it decryptor.send false } }[decryptor, console]*.join()
GPars has been designed primarily for use with the Groovy programming language. Of course all Java and Groovy programs are just bytecodes running on the JVM, so GPars can be used with Java source. Despite being aimed at Groovy code use, the solid technical foundation, plus the good performance characteristics, of GPars make it an excellent library for Java programs. In fact most of GPars is written in Java, so there is no performance penalty for Java applications using GPars.For details please refer to the Java API section.To quick-test using GPars via the Java API, you can compile and run the following Java code:
import groovyx.gpars.MessagingRunnable; import groovyx.gpars.actor.DynamicDispatchActor;public class StatelessActorDemo { public static void main(String[] args) throws InterruptedException { final MyStatelessActor actor = new MyStatelessActor(); actor.start(); actor.send("Hello"); actor.sendAndWait(10); actor.sendAndContinue(10.0, new MessagingRunnable<String>() { @Override protected void doRun(final String s) { System.out.println("Received a reply " + s); } }); } }class MyStatelessActor extends DynamicDispatchActor { public void onMessage(final String msg) { System.out.println("Received " + msg); replyIfExists("Thank you"); } public void onMessage(final Integer msg) { System.out.println("Received a number " + msg); replyIfExists("Thank you"); } public void onMessage(final Object msg) { System.out.println("Received an object " + msg); replyIfExists("Thank you"); } }
2.3 Code conventions
We follow certain conventions in the code samples. Understanding these may help you read and comprehend GPars code samples better.- The leftShift operator << has been overloaded on actors, agents and dataflow expressions (both variables and streams) to mean send a message or assign a value.
myActor << 'message'myAgent << {account -> account.add('5 USD')}myDataflowVariable << 120332
- On actors and agents the default call() method has been also overloaded to mean send . So sending a message to an actor or agent may look like a regular method call.
myActor "message"myAgent {house -> house.repair()}
- The rightShift operator >> in GPars has the when bound meaning. So
myDataflowVariable >> {value -> doSomethingWith(value)}
- GParsPool.withPool()
- GParsPool.withExistingPool()
- GParsExecutorsPool.withPool()
- GParsExecutorsPool.withExistingPool()
- Actors.actor()
- Actors.reactor()
- Actors.fairReactor()
- Actors.messageHandler()
- Actors.fairMessageHandler()
- Agent.agent()
- Agent.fairAgent()
- Dataflow.task()
- Dataflow.operator()
2.4 Getting Set Up in an IDE
Adding the GPars jar files to your project or defining the appropriate dependencies in pom.xml should be enough to get you started with GPars in your IDE.GPars DSL recognition
IntelliJ IDEA in both the free Community Edition and the commercial Ultimate Edition will recognize the GPars domain specific languages, complete methods like eachParallel() , reduce() or callAsync() and validate them. GPars uses the GroovyDSL mechanism, which teaches IntelliJ IDEA the DSLs as soon as the GPars jar file is added to the project.2.5 Applicability of Concepts
GPars provides a lot of concepts to pick from. We're continuously building and updating a page that tries to help user choose the right abstraction for their tasks at hands. Please, refer to the Concepts compared page for details.To briefly summarize the suggestions, below you can find basic guide-lines distilled from the page:- You're looking at a collection, which needs to be iterated or processed using one of the many beautiful Groovy collections method, like each() , collect() , find() and such. Proposing that processing each element of the collection is independent of the other items, using GPars parallel collections can be recommended.
- If you have a long-lasting calculation , which may safely run in the background, use the asynchronous invocation support in GPars. You can also benefit, if your long-calculating closures need to be passed around and yet you'd like them not to block the main application thread.
- You need to parallelize an algorithm at hand. You can identify sub-tasks and you're happy to explicitly express the options for parallelization. You create internally sequential tasks, each of which can run concurrently with the others, providing they all have a way to exchange data at some well-defined moments through communication channels with safe semantics. Use GPars dataflow tasks, variables and streams.
- You can't avoid shared mutable state. Multiple threads will be accessing shared data and (some of them) modifying the data. Traditional locking and synchronized approach feels too risky or unfamiliar. Go for agents, which will wrap your data and serialize all access to it.
- You're building a system with high concurrency demands. Tweaking a data structure here or task there won't cut it. You need to build the architecture from the ground up with concurrency in mind. Message-passing might be the way to go.
- Groovy CSP will give you highly deterministic and composable model for concurrent processes.
- If you're trying to solve a complex data-processing problem, consider GPars dataflow operator to build a data flow network.
- Actors will shine if you need to build a general-purpose, highly concurrent and scalable architecture.
2.6 What's New
Again, the new release, this time GPars 0.12, introduces a lot of gradual enhancements and improvements on top of the previous release.Check out the JIRA release notesProject changes
See the Breaking Changes listing for the list of breaking changes.
Asynchronous functions
- Performance tuning to the asynchronous closure invocation mechanism
Parallel collections
- Added a couple of new parallel collection processing methods to keep up with the innovation pace in Groovy
Fork / Join
Actors
- StaticDispatchActor has been added to provide easier to create and better performing alternative to DynamicDispatchActor
- A new method sendAndPromise has been added to actors to send a message and get a promise for the future actor's reply
Dataflow
- Operator and selector speed-up
- Kanban-style dataflow operator management has been added
- Chaining of Promises using the new then() method
- Added a DSL for easy operator pipe-lining
- Polished the way operators can be stopped
- Added support for custom error handlers
- Added synchronous dataflow variables and channels
- Read channels can report their length
Agent
Stm
Other
- Removed deprecated classes and methods
- Added numerous code examples and demos
- Enhanced project documentation
- Re-styled the user guide
Renaming hints
- The makeTransparent() method that forces concurrent semantics to iteration methods (each, collect, find, etc.) has been removed
- The stop() method on dataflow operators and selectors has been renamed to terminate() to match naming used for actor
- The reportError() method on dataflow operators and selectors has been replaced with the addErrorHandler() method
- The RightShift (>>) operator of DataflowVariables and channels now calls then() instead of whenBound() and so can be chained
2.7 Java API - Using GPars from Java
Using GPars is very addictive, I guarantee. Once you get hooked you won't be able to code without it. May the world force you to write code in Java, you will still be able to benefit from most of GPars features.Java API specifics
Some parts of GPars are irrelevant in Java and it is better to use the underlying Java libraries directly:- Parallel Collection - use jsr-166y library's Parallel Array directly
- Fork/Join - use jsr-166y library's Fork/Join support directly
- Asynchronous functions - use Java executor services directly
GPars Closures in Java API
To overcome the lack of closures as a language element in Java and to avoid forcing users to use Groovy closures directly through the Java API, a few handy wrapper classes have been provided to help you define callbacks, actor body or dataflow tasks.- groovyx.gpars.MessagingRunnable - used for single-argument callbacks or actor body
- groovyx.gpars.ReactorMessagingRunnable - used for ReactiveActor body
- groovyx.gpars.DataflowMessagingRunnable - used for dataflow operators' body
Actors
The DynamicDispatchActor as well as the ReactiveActor classes can be used just like in Groovy:import groovyx.gpars.MessagingRunnable; import groovyx.gpars.actor.DynamicDispatchActor; public class StatelessActorDemo { public static void main(String[] args) throws InterruptedException { final MyStatelessActor actor = new MyStatelessActor(); actor.start(); actor.send("Hello"); actor.sendAndWait(10); actor.sendAndContinue(10.0, new MessagingRunnable<String>() { @Override protected void doRun(final String s) { System.out.println("Received a reply " + s); } }); } } class MyStatelessActor extends DynamicDispatchActor { public void onMessage(final String msg) { System.out.println("Received " + msg); replyIfExists("Thank you"); } public void onMessage(final Integer msg) { System.out.println("Received a number " + msg); replyIfExists("Thank you"); } public void onMessage(final Object msg) { System.out.println("Received an object " + msg); replyIfExists("Thank you"); } }
import groovy.lang.Closure; import groovyx.gpars.ReactorMessagingRunnable; import groovyx.gpars.actor.Actor; import groovyx.gpars.actor.ReactiveActor;public class ReactorDemo { public static void main(final String[] args) throws InterruptedException { final Closure handler = new ReactorMessagingRunnable<Integer, Integer>() { @Override protected Integer doRun(final Integer integer) { return integer * 2; } }; final Actor actor = new ReactiveActor(handler); actor.start(); System.out.println("Result: " + actor.sendAndWait(1)); System.out.println("Result: " + actor.sendAndWait(2)); System.out.println("Result: " + actor.sendAndWait(3)); } }
Convenience factory methods
Obviously, all the essential factory methods to build actors quickly are available where you'd expect them.import groovy.lang.Closure; import groovyx.gpars.ReactorMessagingRunnable; import groovyx.gpars.actor.Actor; import groovyx.gpars.actor.Actors;public class ReactorDemo { public static void main(final String[] args) throws InterruptedException { final Closure handler = new ReactorMessagingRunnable<Integer, Integer>() { @Override protected Integer doRun(final Integer integer) { return integer * 2; } }; final Actor actor = Actors.reactor(handler); System.out.println("Result: " + actor.sendAndWait(1)); System.out.println("Result: " + actor.sendAndWait(2)); System.out.println("Result: " + actor.sendAndWait(3)); } }
Agents
import groovyx.gpars.MessagingRunnable; import groovyx.gpars.agent.Agent; public class AgentDemo { public static void main(final String[] args) throws InterruptedException { final Agent counter = new Agent<Integer>(0); counter.send(10); System.out.println("Current value: " + counter.getVal()); counter.send(new MessagingRunnable<Integer>() { @Override protected void doRun(final Integer integer) { counter.updateValue(integer + 1); } }); System.out.println("Current value: " + counter.getVal()); } }
Dataflow Concurrency
Both DataflowVariables and DataflowQueues can be used from Java without any hiccups. Just avoid the handy overloaded operators and go straight to the methods, like bind , whenBound , getVal and other. You may also continue using dataflow tasks passing to them instances of Runnable or Callable just like groovy Closure .import groovyx.gpars.MessagingRunnable; import groovyx.gpars.dataflow.DataflowVariable; import groovyx.gpars.group.DefaultPGroup;import java.util.concurrent.Callable;public class DataflowTaskDemo { public static void main(final String[] args) throws InterruptedException { final DefaultPGroup group = new DefaultPGroup(10); final DataflowVariable a = new DataflowVariable(); group.task(new Runnable() { public void run() { a.bind(10); } }); final DataflowVariable result = group.task(new Callable() { public Object call() throws Exception { return (Integer)a.getVal() + 10; } }); result.whenBound(new MessagingRunnable<Integer>() { @Override protected void doRun(final Integer integer) { System.out.println("arguments = " + integer); } }); System.out.println("result = " + result.getVal()); } }
Dataflow operators
The sample below should illustrate the main differences between Groovy and Java API for dataflow operators.- Use the convenience factory methods accepting list of channels to create operators or selectors
- Use DataflowMessagingRunnable to specify the operator body
- Call getOwningProcessor() to get hold of the operator from within the body in order to e.g. bind output values
import groovyx.gpars.DataflowMessagingRunnable; import groovyx.gpars.dataflow.Dataflow; import groovyx.gpars.dataflow.DataflowQueue; import groovyx.gpars.dataflow.operator.DataflowProcessor;import java.util.Arrays; import java.util.List;public class DataflowOperatorDemo { public static void main(final String[] args) throws InterruptedException { final DataflowQueue stream1 = new DataflowQueue(); final DataflowQueue stream2 = new DataflowQueue(); final DataflowQueue stream3 = new DataflowQueue(); final DataflowQueue stream4 = new DataflowQueue(); final DataflowProcessor op1 = Dataflow.selector(Arrays.asList(stream1), Arrays.asList(stream2), new DataflowMessagingRunnable(1) { @Override protected void doRun(final Object[] objects) { getOwningProcessor().bindOutput(2*(Integer)objects[0]); } }); final List secondOperatorInput = Arrays.asList(stream2, stream3); final DataflowProcessor op2 = Dataflow.operator(secondOperatorInput, Arrays.asList(stream4), new DataflowMessagingRunnable(2) { @Override protected void doRun(final Object[] objects) { getOwningProcessor().bindOutput((Integer) objects[0] + (Integer) objects[1]); } }); stream1.bind(1); stream1.bind(2); stream1.bind(3); stream3.bind(100); stream3.bind(100); stream3.bind(100); System.out.println("Result: " + stream4.getVal()); System.out.println("Result: " + stream4.getVal()); System.out.println("Result: " + stream4.getVal()); op1.stop(); op2.stop(); } }