Kinesis is a real time data processing service provided by AWS. It continuously captures and stores large amounts of data that power real time streaming dashboards.
A benefit of Kinesis is that in addition to real-time processing, we can also enable parallel processing which enables multiple Kinesis apps to process the same stream of data at the same time.
Kinesis is scalable as all AWS services are, durable as it replicates data to three availability zones and stores it for 24 hours by default (can be increased to 7 days).
Kinesis can be used for:
- Gaming: taking user input, processing them in real-time and providing live feedback to those inputs
- Real time Analytics on IOT devices
- Monitoring logs in real time and executing events based on their content – for example, monitoring stock information and initiating trades as a result of the data analysis
As can be seen from the above, we could take information from an IOT device, let’s say your fridge as an example. Kinesis will analyse that data for the millions of customers that use the connected fridge & will carry out some analysis. If they decide that you’ve run out of milk, it may invoke an SNS topic to be sent directly to you.
In Kinesis, shards are the processing power that your Kinesis deployment has. SO, one shard has the capability to read 2MB per second and write 1MB per second. The more data you have, the more shards you will need.
Multiple consumers can consume data concurrently from the same stream