Questions[amazon-dynamodb](server)

Currently if I run aws dynamodb create-table --generate-cli-skeleton, it will generate a template for me to fill in column definitions etc to create a new dynamodb table.

I want to be able to dump the table layout from another dynamodb table and use it as a json input to the create-table command.

However when I use aws dynamodb describe-table to generate a json, it is structurally different from the cli skeleton file.

What is the easier way to create such a json file based on an existing table?

In a nutshell how to connect DynamoDB to AWS Quicksight, considering these criteria?

In my existing solution, data arrives at AWS IoT, invokes a rule which sends it to Lambda. Lambda decodes it and puts it into DynamoDB. Each record in DynamoDB actually contains several readings/observations for the resource. So the data is not 'tidy' from a statistician's perspective. My desired outcome is to get the data into Quicksight and to be able to perform the same depth of analysis that I could in R, but with the massive benefits of automation, a pretty dashboard and worldwide distribution. This question is specifically about how to get the data to Quicksight (which has no import available at present for noSQL databases), and where the data cleaning should be done.

I have tried:

1. Streaming the data through Lambda (with a trigger on new records), unrolling in Lambda and forwarding to S3 using Kinesis Firehose. From there, you can connect Athena to the dataset and import that into Quicksight. But I loose existing records. And it seems messy to me.
2. I have read some blogs suggesting the use of AWS Glue and Or Redshift for a DataLake. And also some suggesting that I migrate the data into a Relational database first. Many suggestions that would take weeks to test out.
3. Take a snap shot of the entire database on a weekly basis and just plonk that into S3. This seems wasteful and not inherently scale-able? But I might be underestimating the capability of AWS.
4. Instead of using the services, it is possible to schedule a Lmabda to run weekly and scan the whole table and pull it apart to create a massive csv file and put it in S3. It might be a crazy approach, but it's a simple way to take care of all data cleaning and even cross with other datasets up front. for instance, imagine a customer details data set and a customer purchase history dataset, which will be much larger. It might be nice to merge them together in Lambda to remove the need for complex data set joining. Or it might be a very bad idea. I'm really not sure. Hence the question.

On data cleaning, to complicate matters Im interested in 3 databases. I need to join them. Easy as, in R. But seemingly very difficult in this scenario.

I am exploring using AWS Lambda through Laravel Vapor. Avoiding the use of Redis can save us ~$56/mo (t2.small at ~$24, NAT Gateway at ~$32).

Instead of Redis, I plan to use DynamoDB which is much cheaper. Pretty much same thoughts to this thread https://twitter.com/JackEllis/status/1246500143425146880

My question is, does it make sense to use DynamoDB as a cache layer on top of RDS or is the latency pretty much the same, which means I would be better off just relying on MySQL's internal caching?

For context, in Laravel there's a package called laravel-model-caching that caches query results. With Redis as the cache driver it makes sense, because you get super low latency compared to mysql even with mysql caching.

But now if I get rid of Redis, caching queries using the package might not make sense if the cache layer is another DB (Dynamo) anyway.

Is my thinking correct that DynamoDB and RDS latency is not much different?

I am going to set up a DynamoDB on aws, and their pricing is about .3$/GB a month. If i store 500 GB for an hour, do some work on it, then remove 200GB after an hour. Do i have to pay for the 500 GB used for the whole month or, 500 GB for the hour it was on the DB ?

I am trying to decide which database I should use for an application I'm making. I was leaning toward dynamoDB because of its scalability, but then I read in the documentation which said:

there is a limit of 64 KB on the item size

although it looks like MySQL has a similar restriction documented here

This application will be storing a lot of data in two arrays, which could contain upwards of 10,000-100,000 strings in each. I estimate that these strings will each be somewhere around 20 characters long, so each element of the array will be around 40bytes and each array could be around 4MB.

Given this predicament, what database on amazon AWS would you use; or how would you get around the limit of size per row?