Java Code Examples for org.apache.flink.configuration.MemorySize#add()

The following examples show how to use org.apache.flink.configuration.MemorySize#add() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.

Example 1

Source File: ProcessMemoryUtils.java From flink with Apache License 2.0

6 votes

private MemorySize deriveJvmOverheadFromTotalFlinkMemoryAndOtherComponents(
		Configuration config,
		MemorySize totalFlinkMemorySize) {
	MemorySize totalProcessMemorySize = getMemorySizeFromConfig(config, options.getTotalProcessMemoryOption());
	MemorySize jvmMetaspaceSize = getMemorySizeFromConfig(config, options.getJvmOptions().getJvmMetaspaceOption());
	MemorySize totalFlinkAndJvmMetaspaceSize = totalFlinkMemorySize.add(jvmMetaspaceSize);
	if (totalProcessMemorySize.getBytes() < totalFlinkAndJvmMetaspaceSize.getBytes()) {
		throw new IllegalConfigurationException(
			"The configured Total Process Memory size (%s) is less than the sum of the derived " +
				"Total Flink Memory size (%s) and the configured or default JVM Metaspace size  (%s).",
			totalProcessMemorySize.toHumanReadableString(),
			totalFlinkMemorySize.toHumanReadableString(),
			jvmMetaspaceSize.toHumanReadableString());
	}
	MemorySize jvmOverheadSize = totalProcessMemorySize.subtract(totalFlinkAndJvmMetaspaceSize);
	sanityCheckJvmOverhead(config, jvmOverheadSize, totalProcessMemorySize);
	return jvmOverheadSize;
}

Example 2

Source File: JobManagerProcessUtilsTest.java From flink with Apache License 2.0

6 votes

@Test
public void testOffHeapMemoryDerivedFromJvmHeapAndTotalFlinkMemory() {
	MemorySize jvmHeap = MemorySize.ofMebiBytes(150);
	MemorySize defaultOffHeap = JobManagerOptions.OFF_HEAP_MEMORY.defaultValue();
	MemorySize expectedOffHeap = MemorySize.ofMebiBytes(100).add(defaultOffHeap);
	MemorySize totalFlinkMemory = jvmHeap.add(expectedOffHeap);

	Configuration conf = new Configuration();
	conf.set(JobManagerOptions.TOTAL_FLINK_MEMORY, totalFlinkMemory);
	conf.set(JobManagerOptions.JVM_HEAP_MEMORY, jvmHeap);

	JobManagerProcessSpec jobManagerProcessSpec = JobManagerProcessUtils.processSpecFromConfig(conf);
	assertThat(jobManagerProcessSpec.getJvmDirectMemorySize(), is(expectedOffHeap));
	MatcherAssert.assertThat(
		testLoggerResource.getMessages(),
		hasItem(containsString(String.format(
			"The Off-Heap Memory size (%s) is derived the configured Total Flink Memory size (%s) minus " +
				"the configured JVM Heap Memory size (%s). The default Off-Heap Memory size (%s) is ignored.",
			expectedOffHeap.toHumanReadableString(),
			totalFlinkMemory.toHumanReadableString(),
			jvmHeap.toHumanReadableString(),
			defaultOffHeap.toHumanReadableString()))));
}

Example 3

Source File: JobManagerFlinkMemoryUtils.java From flink with Apache License 2.0

5 votes

private static void sanityCheckTotalFlinkMemory(
		MemorySize totalFlinkMemorySize,
		MemorySize jvmHeapMemorySize,
		MemorySize offHeapMemorySize) {
	MemorySize derivedTotalFlinkMemorySize = jvmHeapMemorySize.add(offHeapMemorySize);
	if (derivedTotalFlinkMemorySize.getBytes() != totalFlinkMemorySize.getBytes()) {
		throw new IllegalConfigurationException(String.format(
			"Sum of the configured JVM Heap Memory (%s) and the configured Off-heap Memory (%s) " +
				"does not match the configured Total Flink Memory (%s). Please, make the configuration consistent " +
				"or configure only one option: either JVM Heap or Total Flink Memory.",
			jvmHeapMemorySize.toHumanReadableString(),
			offHeapMemorySize.toHumanReadableString(),
			totalFlinkMemorySize.toHumanReadableString()));
	}
}