Java Code Examples for org.apache.flink.api.common.Plan#getMaximumParallelism()

The following examples show how to use org.apache.flink.api.common.Plan#getMaximumParallelism() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: LocalExecutor.java    From Flink-CEPplus with Apache License 2.0 4 votes vote down vote up 
/**
 * Executes the given program on a local runtime and waits for the job to finish.
 *
 * <p>If the executor has not been started before, this starts the executor and shuts it down
 * after the job finished. If the job runs in session mode, the executor is kept alive until
 * no more references to the executor exist.</p>
 *
 * @param plan The plan of the program to execute.
 * @return The net runtime of the program, in milliseconds.
 *
 * @throws Exception Thrown, if either the startup of the local execution context, or the execution
 *                   caused an exception.
 */
@Override
public JobExecutionResult executePlan(Plan plan) throws Exception {
	if (plan == null) {
		throw new IllegalArgumentException("The plan may not be null.");
	}

	synchronized (this.lock) {

		// check if we start a session dedicated for this execution
		final boolean shutDownAtEnd;

		if (jobExecutorService == null) {
			shutDownAtEnd = true;

			// configure the number of local slots equal to the parallelism of the local plan
			if (this.taskManagerNumSlots == DEFAULT_TASK_MANAGER_NUM_SLOTS) {
				int maxParallelism = plan.getMaximumParallelism();
				if (maxParallelism > 0) {
					this.taskManagerNumSlots = maxParallelism;
				}
			}

			// start the cluster for us
			start();
		}
		else {
			// we use the existing session
			shutDownAtEnd = false;
		}

		try {
			// TODO: Set job's default parallelism to max number of slots
			final int slotsPerTaskManager = jobExecutorServiceConfiguration.getInteger(TaskManagerOptions.NUM_TASK_SLOTS, taskManagerNumSlots);
			final int numTaskManagers = jobExecutorServiceConfiguration.getInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, 1);
			plan.setDefaultParallelism(slotsPerTaskManager * numTaskManagers);

			Optimizer pc = new Optimizer(new DataStatistics(), jobExecutorServiceConfiguration);
			OptimizedPlan op = pc.compile(plan);

			JobGraphGenerator jgg = new JobGraphGenerator(jobExecutorServiceConfiguration);
			JobGraph jobGraph = jgg.compileJobGraph(op, plan.getJobId());

			return jobExecutorService.executeJobBlocking(jobGraph);
		}
		finally {
			if (shutDownAtEnd) {
				stop();
			}
		}
	}
}
Example 2
Source File: LocalExecutor.java    From flink with Apache License 2.0 4 votes vote down vote up 
/**
 * Executes the given program on a local runtime and waits for the job to finish.
 *
 * <p>If the executor has not been started before, this starts the executor and shuts it down
 * after the job finished. If the job runs in session mode, the executor is kept alive until
 * no more references to the executor exist.</p>
 *
 * @param plan The plan of the program to execute.
 * @return The net runtime of the program, in milliseconds.
 *
 * @throws Exception Thrown, if either the startup of the local execution context, or the execution
 *                   caused an exception.
 */
@Override
public JobExecutionResult executePlan(Plan plan) throws Exception {
	if (plan == null) {
		throw new IllegalArgumentException("The plan may not be null.");
	}

	synchronized (this.lock) {

		// check if we start a session dedicated for this execution
		final boolean shutDownAtEnd;

		if (jobExecutorService == null) {
			shutDownAtEnd = true;

			// configure the number of local slots equal to the parallelism of the local plan
			if (this.taskManagerNumSlots == DEFAULT_TASK_MANAGER_NUM_SLOTS) {
				int maxParallelism = plan.getMaximumParallelism();
				if (maxParallelism > 0) {
					this.taskManagerNumSlots = maxParallelism;
				}
			}

			// start the cluster for us
			start();
		}
		else {
			// we use the existing session
			shutDownAtEnd = false;
		}

		try {
			// TODO: Set job's default parallelism to max number of slots
			final int slotsPerTaskManager = jobExecutorServiceConfiguration.getInteger(TaskManagerOptions.NUM_TASK_SLOTS, taskManagerNumSlots);
			final int numTaskManagers = jobExecutorServiceConfiguration.getInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER, 1);
			plan.setDefaultParallelism(slotsPerTaskManager * numTaskManagers);

			Optimizer pc = new Optimizer(new DataStatistics(), jobExecutorServiceConfiguration);
			OptimizedPlan op = pc.compile(plan);

			JobGraphGenerator jgg = new JobGraphGenerator(jobExecutorServiceConfiguration);
			JobGraph jobGraph = jgg.compileJobGraph(op, plan.getJobId());

			return jobExecutorService.executeJobBlocking(jobGraph);
		}
		finally {
			if (shutDownAtEnd) {
				stop();
			}
		}
	}
}