org.apache.flink.streaming.api.functions.source.FileMonitoringFunction Java Examples

The following examples show how to use org.apache.flink.streaming.api.functions.source.FileMonitoringFunction. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: StreamingNoop.java    From Flink-CEPplus with Apache License 2.0 6 votes vote down vote up 
public static void main(String[] args) throws Exception {
	ParameterTool params = ParameterTool.fromArgs(args);

	// define the dataflow
	StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
	env.setParallelism(2);
	env.setRestartStrategy(RestartStrategies.fixedDelayRestart(10, 1000));
	env.readFileStream("input/", 60000, FileMonitoringFunction.WatchType.ONLY_NEW_FILES)
		.addSink(new DiscardingSink<String>());

	// generate a job graph
	final JobGraph jobGraph = env.getStreamGraph().getJobGraph();
	File jobGraphFile = new File(params.get("output", "job.graph"));
	try (FileOutputStream output = new FileOutputStream(jobGraphFile);
		ObjectOutputStream obOutput = new ObjectOutputStream(output)){
		obOutput.writeObject(jobGraph);
	}
}
Example #2
Source File: StreamingNoop.java    From flink with Apache License 2.0 6 votes vote down vote up 
public static void main(String[] args) throws Exception {
	ParameterTool params = ParameterTool.fromArgs(args);

	// define the dataflow
	StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
	env.setParallelism(2);
	env.setRestartStrategy(RestartStrategies.fixedDelayRestart(10, 1000));
	env.readFileStream("input/", 60000, FileMonitoringFunction.WatchType.ONLY_NEW_FILES)
		.addSink(new DiscardingSink<String>());

	// generate a job graph
	final JobGraph jobGraph = env.getStreamGraph().getJobGraph();
	File jobGraphFile = new File(params.get("output", "job.graph"));
	try (FileOutputStream output = new FileOutputStream(jobGraphFile);
		ObjectOutputStream obOutput = new ObjectOutputStream(output)){
		obOutput.writeObject(jobGraph);
	}
}
Example #3
Source File: StreamingNoop.java    From flink with Apache License 2.0 6 votes vote down vote up 
public static void main(String[] args) throws Exception {
	ParameterTool params = ParameterTool.fromArgs(args);

	// define the dataflow
	StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
	env.setParallelism(2);
	env.setRestartStrategy(RestartStrategies.fixedDelayRestart(10, 1000));
	env.readFileStream("input/", 60000, FileMonitoringFunction.WatchType.ONLY_NEW_FILES)
		.addSink(new DiscardingSink<String>());

	// generate a job graph
	final JobGraph jobGraph = env.getStreamGraph().getJobGraph();
	File jobGraphFile = new File(params.get("output", "job.graph"));
	try (FileOutputStream output = new FileOutputStream(jobGraphFile);
		ObjectOutputStream obOutput = new ObjectOutputStream(output)){
		obOutput.writeObject(jobGraph);
	}
}
Example #4
Source File: StreamExecutionEnvironment.java    From Flink-CEPplus with Apache License 2.0 3 votes vote down vote up 
/**
 * Creates a data stream that contains the contents of file created while system watches the given path. The file
 * will be read with the system's default character set.
 *
 * @param filePath
 * 		The path of the file, as a URI (e.g., "file:///some/local/file" or "hdfs://host:port/file/path/")
 * @param intervalMillis
 * 		The interval of file watching in milliseconds
 * @param watchType
 * 		The watch type of file stream. When watchType is {@link org.apache.flink.streaming.api.functions.source.FileMonitoringFunction.WatchType#ONLY_NEW_FILES}, the system processes
 * 		only
 * 		new files. {@link org.apache.flink.streaming.api.functions.source.FileMonitoringFunction.WatchType#REPROCESS_WITH_APPENDED} means that the system re-processes all contents of
 * 		appended file. {@link org.apache.flink.streaming.api.functions.source.FileMonitoringFunction.WatchType#PROCESS_ONLY_APPENDED} means that the system processes only appended
 * 		contents
 * 		of files.
 * @return The DataStream containing the given directory.
 *
 * @deprecated Use {@link #readFile(FileInputFormat, String, FileProcessingMode, long)} instead.
 */
@Deprecated
@SuppressWarnings("deprecation")
public DataStream<String> readFileStream(String filePath, long intervalMillis, FileMonitoringFunction.WatchType watchType) {
	DataStream<Tuple3<String, Long, Long>> source = addSource(new FileMonitoringFunction(
			filePath, intervalMillis, watchType), "Read File Stream source");

	return source.flatMap(new FileReadFunction());
}
Example #5
Source File: StreamExecutionEnvironment.java    From flink with Apache License 2.0 3 votes vote down vote up 
/**
 * Creates a data stream that contains the contents of file created while system watches the given path. The file
 * will be read with the system's default character set.
 *
 * @param filePath
 * 		The path of the file, as a URI (e.g., "file:///some/local/file" or "hdfs://host:port/file/path/")
 * @param intervalMillis
 * 		The interval of file watching in milliseconds
 * @param watchType
 * 		The watch type of file stream. When watchType is {@link org.apache.flink.streaming.api.functions.source.FileMonitoringFunction.WatchType#ONLY_NEW_FILES}, the system processes
 * 		only
 * 		new files. {@link org.apache.flink.streaming.api.functions.source.FileMonitoringFunction.WatchType#REPROCESS_WITH_APPENDED} means that the system re-processes all contents of
 * 		appended file. {@link org.apache.flink.streaming.api.functions.source.FileMonitoringFunction.WatchType#PROCESS_ONLY_APPENDED} means that the system processes only appended
 * 		contents
 * 		of files.
 * @return The DataStream containing the given directory.
 *
 * @deprecated Use {@link #readFile(FileInputFormat, String, FileProcessingMode, long)} instead.
 */
@Deprecated
@SuppressWarnings("deprecation")
public DataStream<String> readFileStream(String filePath, long intervalMillis, FileMonitoringFunction.WatchType watchType) {
	DataStream<Tuple3<String, Long, Long>> source = addSource(new FileMonitoringFunction(
			filePath, intervalMillis, watchType), "Read File Stream source");

	return source.flatMap(new FileReadFunction());
}
Example #6
Source File: StreamExecutionEnvironment.java    From flink with Apache License 2.0 3 votes vote down vote up 
/**
 * Creates a data stream that contains the contents of file created while system watches the given path. The file
 * will be read with the system's default character set.
 *
 * @param filePath
 * 		The path of the file, as a URI (e.g., "file:///some/local/file" or "hdfs://host:port/file/path/")
 * @param intervalMillis
 * 		The interval of file watching in milliseconds
 * @param watchType
 * 		The watch type of file stream. When watchType is {@link org.apache.flink.streaming.api.functions.source.FileMonitoringFunction.WatchType#ONLY_NEW_FILES}, the system processes
 * 		only
 * 		new files. {@link org.apache.flink.streaming.api.functions.source.FileMonitoringFunction.WatchType#REPROCESS_WITH_APPENDED} means that the system re-processes all contents of
 * 		appended file. {@link org.apache.flink.streaming.api.functions.source.FileMonitoringFunction.WatchType#PROCESS_ONLY_APPENDED} means that the system processes only appended
 * 		contents
 * 		of files.
 * @return The DataStream containing the given directory.
 *
 * @deprecated Use {@link #readFile(FileInputFormat, String, FileProcessingMode, long)} instead.
 */
@Deprecated
@SuppressWarnings("deprecation")
public DataStream<String> readFileStream(String filePath, long intervalMillis, FileMonitoringFunction.WatchType watchType) {
	DataStream<Tuple3<String, Long, Long>> source = addSource(new FileMonitoringFunction(
			filePath, intervalMillis, watchType), "Read File Stream source");

	return source.flatMap(new FileReadFunction());
}