Java Code Examples for org.apache.beam.runners.core.construction.ParDoTranslation#getAdditionalOutputTags()
The following examples show how to use
org.apache.beam.runners.core.construction.ParDoTranslation#getAdditionalOutputTags() .
You can vote up the ones you like or vote down the ones you don't like,
and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: ParDoMultiOverrideFactory.java From beam with Apache License 2.0 | 5 votes |
@SuppressWarnings("unchecked") private PTransform<PCollection<? extends InputT>, PCollectionTuple> getReplacementForApplication( AppliedPTransform< PCollection<? extends InputT>, PCollectionTuple, PTransform<PCollection<? extends InputT>, PCollectionTuple>> application) throws IOException { DoFn<InputT, OutputT> fn = (DoFn<InputT, OutputT>) ParDoTranslation.getDoFn(application); DoFnSignature signature = DoFnSignatures.getSignature(fn.getClass()); if (signature.processElement().isSplittable()) { return SplittableParDo.forAppliedParDo((AppliedPTransform) application); } else if (signature.stateDeclarations().size() > 0 || signature.timerDeclarations().size() > 0 || signature.timerFamilyDeclarations().size() > 0) { return new GbkThenStatefulParDo( fn, ParDoTranslation.getMainOutputTag(application), ParDoTranslation.getAdditionalOutputTags(application), ParDoTranslation.getSideInputs(application), ParDoTranslation.getSchemaInformation(application), ParDoTranslation.getSideInputMapping(application)); } else { return application.getTransform(); } }
Example 2
Source File: PipelineTranslator.java From incubator-nemo with Apache License 2.0 | 4 votes |
/** * @param ctx provides translation context. * @param beamNode the beam node to be translated. * @param sideInputMap side inputs. * @return the created DoFnTransform. */ private static AbstractDoFnTransform createDoFnTransform(final PipelineTranslationContext ctx, final TransformHierarchy.Node beamNode, final Map<Integer, PCollectionView<?>> sideInputMap) { try { final AppliedPTransform pTransform = beamNode.toAppliedPTransform(ctx.getPipeline()); final DoFn doFn = ParDoTranslation.getDoFn(pTransform); final TupleTag mainOutputTag = ParDoTranslation.getMainOutputTag(pTransform); final TupleTagList additionalOutputTags = ParDoTranslation.getAdditionalOutputTags(pTransform); final PCollection<?> mainInput = (PCollection<?>) Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(pTransform)); final HasDisplayData displayData = (builder) -> builder.add(DisplayData.item("name", beamNode.getFullName())); final DoFnSchemaInformation doFnSchemaInformation = ParDoTranslation.getSchemaInformation(beamNode.toAppliedPTransform(ctx.getPipeline())); if (sideInputMap.isEmpty()) { return new DoFnTransform( doFn, mainInput.getCoder(), getOutputCoders(pTransform), mainOutputTag, additionalOutputTags.getAll(), mainInput.getWindowingStrategy(), ctx.getPipelineOptions(), DisplayData.from(displayData), doFnSchemaInformation, Collections.emptyMap()); } else { return new PushBackDoFnTransform( doFn, mainInput.getCoder(), getOutputCoders(pTransform), mainOutputTag, additionalOutputTags.getAll(), mainInput.getWindowingStrategy(), sideInputMap, ctx.getPipelineOptions(), DisplayData.from(displayData), doFnSchemaInformation, Collections.emptyMap()); } } catch (final IOException e) { throw new RuntimeException(e); } }