Batch or Streaming? Write the Output Deadline First · Azure Data Fundamentals

An arrival pattern is not an answer

A system can receive events continuously and still calculate its only report once a week. The arrival path is not the same as the processing path. Batch processing works on a bounded collected input; stream processing treats input as an ongoing arrival sequence. Both can filter and aggregate, and both need a design matching the required output. Before selecting a pattern, write what question the output must answer, when it must be available, and what constitutes a complete input for that question.

A buffer can capture and retain events for downstream work. It does not automatically calculate a running total or prove that a dashboard is current. Similarly, a historical report can be correct for its reporting boundary without being an adequate substitute for an ongoing display. Naming a service real-time does not settle latency, error handling, reconciliation or data quality. Those are conditions to establish, not guarantees supplied by a convenient label.

Worked case: a community workshop network

Fictional case:a network of craft workshops records material requests as they arrive. Coordinators need an updated view of request counts during the working day. At the end of each week, another team checks the complete approved requests and produces a reconciled distribution report. The two outputs share source events, but they ask different questions about timing and completeness. Start by drawing each output separately rather than making one pipeline label do all the explanatory work.

For the working-day view, the design needs a processing path that consumes arrivals and publishes the requested updates within an explicitly agreed timing budget. Capturing events into a buffer supplies inputs; the consumer still needs the counting logic. For the weekly report, the agreed input can be a bounded approved collection checked after the period closes. A batch process can prepare that report without claiming to be the working-day view. Neither output should silently be described as the other.

Now consider a proposed simplification:keep the buffer and delete the processing consumer because all messages are safely stored. That preserves inputs but removes the stage that makes the running result. A different proposal deletes the retained source collection after displaying one current total. That may leave too little information for the weekly reconciliation. These proposals affect different requirements. State the lost capability in terms of the needed operation or evidence, not merely by saying that one architecture is modern and another is old.

Define counting rules before doing arithmetic

Even a tiny analytical count needs explicit rules. Specify which records qualify, the collection or time boundary, and the operation to apply. In a supplied exercise, a start can be inclusive while the end is exclusive; the record on the end boundary then belongs outside that defined interval. That is an exercise contract, not an assertion of a default across every Microsoft service. A total of all captured inputs may differ from the qualifying count, just as a row count differs from the sum of a measure.

A production system can also have timing and recovery questions not specified in a foundation problem. Do not invent a watermark policy, late-event treatment or exactly-once guarantee to complete an answer. Record what must be clarified before making those claims. A foundation decision can identify the required processing role while leaving implementation details unknown. This preserves a checkable conclusion instead of filling the missing design with assumed behavior.

Place platform components in the right role

Fabric integrates analytics experiences around OneLake, its logical lake. Its Real-Time hub provides a central discovery and management role for supported streaming sources. A warehouse table, a lakehouse collection and a Power BI semantic model serve different data or analytical purposes; none should be identified as the streaming-source discovery hub just because it can be part of the overall solution. The diagram separates the source, buffer, processing and output roles so that the missing stage is visible.

Use the two application checks below before returning to the sequential streaming lesson and formal tasks. Explain a choice by the output requirement and available evidence. If a colleague changes the deadline or changes what counts as a complete collection, revisit the choice. That changed condition can make a different pattern justified without making the original reasoning universally wrong. The result is a bounded planning explanation, not an instruction to deploy resources or a promise of operational performance.