I have a strange question(at least to me!), but I'm not sure how to even approach it.
Background
I have a few batch jobs(python scripts called via cron or python script being called by bash script). For example, 20 ETL jobs and 20 data processing jobs. That occurs at different times.
Current approach
I keep my server on and run cron jobs to run the scripts at their intended time(they do not all run at the same time..one may run at 1pm whereas another runs at 5pm). This is working perfectly right now
Problem I want to save money and the jobs do not have to start exactly at the time I set, they could be a few minutes or an hour or two off so I was thinking of using preemptible instances that start the server on a schedule.
I was thinking of starting the server every hour and then it processes any jobs that are past their schedule that has not been done yet.
I have no idea how to do this! Is there a way I can schedule this within ubuntu? I thought of setting up an external DB and have a script that queries it to see what processes need to start but I felt it was overkill, I suspect this maybe a common problem that has a simpler solution.
Please let me know!
0 Answers