Server API

Let's demonstrate the TurkServer API with a simple example, which allows chat interaction between a group of users:

@ExperimentServer("chat")
public class ChatExperiment {
	HITWorkerGroup group;
	ExperimentLog log;
	ExperimentController controller;
	
	@Inject
	public ChatExperiment(
			HITWorkerGroup group,
			ExperimentLog log,
			ExperimentController controller) {
		this.group = group;
		this.log = log;
		this.controller = controller;
	}
	
	@StartExperiment
	void start() throws MessageException {
		log.printf("Starting chat with %d people", group.groupSize());
		Map<String, Object> data = ImmutableMap.of(
						"msg", (Object) "Please start chatting!");
		controller.sendExperimentBroadcast(data);		
	}
	
	@BroadcastMessage
	boolean chatMessage(HITWorker worker, Map<String, Object> data) {
		log.printf("Worker %s said: %s", worker.getUsername(), data.get("msg"));		
		return true;
	}
	
	@TimeLimit
	void timedOut() throws MessageException {
		Map<String, Object> data = ImmutableMap.of(
				"msg", (Object) "No more chatting for you!");
		controller.sendExperimentBroadcast(data);		
	}
}

This simple example uses some TurkServer classes (HITWorkerGroup, ExperimentLog, and ExperimentController) to send messages and control the experiment, and specifies some additional messages using Java annotations that will be called asynchronously when certain events happen. In this case, we have a very simple 'experiment' that just passes messages between a group of users who are chatting with each other.

In general, using the API involves accessing TurkServer classes designed to facilitate logging, messaging, and experiment control, and specifying callback methods to handle events of interest in your experiment. This divides the API up into two parts: the injectables, which are objects created on demand for each experiment, and the callbacks, which are called by other threads when certain things happen.

Injectables

The injectables, so called because of the dependency injection paradigm, are objects that are created for each instance of experiment. They are allow the experiment logic to do certain things such as send messages to users, configure and start rounds, and log data. We use Google Guice to set up experiments; feel free to read more about how that works on their site.

In a nutshell, there are two ways to specify objects of interest in your experiment, and these are created automatically when the @Inject annotation is found. You can create an object with a no-argument constructor, and inject the fields directly:

public class Something {
    @Inject HITWorkerGroup group;
    @Inject ExperimentLog log;
    @Inject ExperimentController controller;
}

Or, you can annotate a constructor with @Inject, which cleanly specifies all the dependencies. This way is more verbose, but it's preferred by us as well as the Guice programmers since it is more safe and testable. With TurkServer, you can actually instantiate your experiments using test classes and simulate them, making sure everything works before spending real money and getting real people.

public class Something {
    HITWorkerGroup group;
    ExperimentLog log;
    ExperimentController controller;
    @Inject
    public Something(
        HITWorkerGroup group,
        ExperimentLog log,
        ExperimentController controller
    ) {
        this.group = group;
        this.log = log;
        this.controller = controller;
    }
}

The currently injectable classes are the following:

• HITWorker - a single worker, injected for single-user interactions.
• HITWorkerGroup - a group of workers, injected for multi-user interactions
• ExperimentController - allows control of when the experiment should finish, as well as controlling an experiment that takes place over multiple rounds
• ExperimentLog - allows for logging of data from the experiment

Callbacks

Callbacks are specified by annotations as well, and are triggered by various things that happen, such as a user sending a message or the experiment reaching the conditions to start. An important note: all these events typically happen in different threads, so experiment classes must be thread-safe. For easy ways to do this, check out our examples and how they use the java.util.concurrent package.

@StartExperiment
void startExp() {
	controller.startRounds();
}

A method annotated with @StartExperiment is called after your experiment has been constructed and has satisfied the preconditions (usually having an acceptable set of users) to start. In this example, we call the controller's startRounds method, which tells TurkServer that our experiment will collect data over multiple rounds and configures the log accordingly.

@StartRound
void startRound(int n) {

}

The @StartRound annotation specifies a method to be called at the beginning of each round - when rounds are first configured, or after the end of other rounds. Call finishRound or finishExperiment on the controller object to signal that a round or the entire experiment should be finished, respectively.

@TimeLimit
void timeLimit() {
	
}

The @TimeLimit annotation results in methods being called when the pre-configured time limit, if any, for an experiment is hit. Usually, you will want to trigger the end of your experiment and notify users in this method.

@WorkerConnect
void connect(HITWorker worker) {
	
}

Methods with the @WorkerConnect annotation are called when a worker connects to an experiment after it starts, usually from being disconnected. You should define this to re-send the state of your experiment to a reconnected worker.

@WorkerDisconnect
void disconnect(HITWorker worker) {

}

A @WorkerDisconnect method is called when workers drop connection to your experiment. It is not always called before a @WorkerConnect method, especially when connections time out. However, it can be useful to keep track of when people have left a synchronous experiment.

@BroadcastMessage
boolean broadcast(HITWorker worker, Map<String, Object> msg) {

}

A @BroadcastMessage method is called when a user sends a message to the entire group of users in the experiment. The msg map is a JSON object that has been deserialized to Java. Your method should return true if the broadcast should be forwarded to all users in the group and false if not. The annotation supports the forms @BroadcastMessage(key='somekey') or @BroadcastMessage(key='somekey',value='somevalue') to be triggered only when such a key or a key with the corresponding value exists in the map. The annotation can be overloaded with many such methods to simplify the message processing on the server side.

@ServiceMessage
void service(HITWorker worker, Map<String, Object> msg) {

}

A @ServiceMessage has the same method as a broadcast message but is meant for just client-to-server communication, and there is no messaging to the entire group. It also supports the custom forms above.