Kinesis Data Streams vs Data Firehose: which to use

Try 3 questions on this

Pick an answer to see why each option is right or wrong.

1. A team must land clickstream records in Amazon S3 and Amazon OpenSearch Service continuously, with no consumer application to write or scale, and is willing to accept delivery in batches rather than instantly. Which solution meets these requirements?

   Right. In the game that’s Renown for your Design High-Performing Architectures building.

   Not that one: B is right. In the game your Design High-Performing Architectures building would lose a little Renown, and this question would come back later.

   Your answerA.Use Amazon SQS with two queues, and write a worker for each queue that batches records into the destinations.Rebuilds delivery on a queue, with two workers to write, deploy and scale, for a stream the managed service already handles.The answerB.Use Amazon Data Firehose, which is fully managed and delivers buffered records to both destinations.The requirement names Firehose's two defining properties. Managed delivery.Your answerC.Use S3 event notifications to copy each record into OpenSearch Service.S3 event notifications fire when an object is created in a bucket. The clickstream records have not reached a bucket at that point.Your answerD.Use Amazon Kinesis Data Streams and write a consumer for each destination that reads from the shards.Kinesis Data Streams gives ordered, replayable shards and leaves the delivery to consumer applications the team has said it does not want to write.

   Show the full explanation and doc quotes

   "Amazon Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon OpenSearch Service... With Amazon Data Firehose, you don't need to write applications or manage resources. You configure your data producers to send data to Amazon Data Firehose, and it automatically delivers the data to the destination that you specified." And batching: "Amazon Data Firehose buffers incoming streaming data to a certain size or for a certain period of time before delivering it to destinations. Buffer Size is in MBs and Buffer Interval is in seconds." The willingness to accept batches is what makes Firehose rather than a stream the right answer.

   Written from docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html

2. An analytics team needs streaming records in Amazon S3 with no consumer code, and a separate fraud team needs to read the same records within milliseconds and replay the last day of them after a model change. A solutions architect must serve both from one ingestion path. Which approach will meet these requirements?

   Right. In the game that’s Renown for your Design High-Performing Architectures building.

   Not that one: D is right. In the game your Design High-Performing Architectures building would lose a little Renown, and this question would come back later.

   Your answerA.Send records to two Firehose streams, one for S3 delivery and one for the fraud team.Duplicates the managed path and still gives the fraud team no low-latency subscriber, because a Firehose stream is a delivery mechanism rather than a readable stream.Your answerB.Send records to a Firehose stream, and have the fraud team read the delivered objects from S3 as they land.Reading the delivered objects means waiting for the buffer to flush and gives no replay of the stream, so the fraud team gets neither millisecond latency nor re-reads.Your answerC.Send records to a Kinesis data stream, and have the analytics team write a consumer batching them into S3.This meets the fraud team's needs and puts the consumer application back on the analytics team, which the requirement explicitly excludes.The answerD.Send records to a Kinesis data stream, and configure a Firehose stream to read from it and deliver to Amazon S3.The two requirements pull in opposite directions and the documented composition satisfies both. Firehose is the managed delivery half: "you don't need to write applications or manage resources"

   Show the full explanation and doc quotes

   And it "buffers incoming streaming data to a certain size or for a certain period of time before delivering it", which is why it cannot serve millisecond reads or replay. The composition is documented on the same page: "You can also configure your Firehose stream to automatically read data from an existing Kinesis data stream, and load it into destinations." That leaves the data stream as the real-time, replayable source for the fraud team's own consumer, with Firehose attached to it for the analytics team's delivery.

   Written from docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html

3. A regulated workload must be able to replay a Kinesis data stream for up to a year, and the team asks whether the stream itself can hold records that long or whether the data has to be copied elsewhere for replay. Which approach will meet these requirements?

   Right. In the game that’s Renown for your Design High-Performing Architectures building.

   Not that one: A is right. In the game your Design High-Performing Architectures building would lose a little Renown, and this question would come back later.

   The answerA.Raise the stream's retention period, which runs from 24 hours by default up to 8,760 hours, that is 365 days.A year is inside what the stream itself offers. "A Kinesis data stream stores records from 24 hours by default, up to 8760 hours (365 days). You can update the retention period via the Kinesis Data Streams console or by using the IncreaseStreamRetentionPeriod and the DecreaseStreamRetentionPeriod operations."Your answerB.Copy the records to Amazon S3, because a Kinesis data stream holds records for at most seven days whatever is configured.The maximum retention period is 8,760 hours, which is 365 days.Your answerC.Raise the retention period to its maximum of 8,760 hours, but note that replay beyond 30 days must still come from a copy.Records remain readable from the stream for the whole configured retention period.Your answerD.Create a second stream and write to both, because retention is fixed at creation.The retention period can be changed on an existing stream.

   Written from docs.aws.amazon.com/streams/latest/dev/kinesis-extended-retention.html