2 messages
Discussion related to Amazon Web Services (AWS)
Archive: https://archive.sweetops.com/aws/
SA11 months ago(edited)
Hi everyone,
I'm working on a data restoration process involving large batches of archived files stored in S3 Glacier Deep Archive. The recent data we store is tarred into single files for efficiency, but the older data we're trying to restore consists of thousands of individual files per batch, which is making the restoration process challenging.
S3 doesn’t support folder restores and requires files to be restored individually. As you can imagine, trying to restore such a large number of individual files using bash scripts is causing timeouts and issues with tracking the progress.
Has anyone experienced a similar issue with restoring large numbers of individual files from S3 Glacier? We’re considering S3 Batch Operations, but I’d love to hear if anyone has other strategies or best practices for efficiently handling this kind of large-scale restore, especially when dealing with a massive number of files.
TIA
I'm working on a data restoration process involving large batches of archived files stored in S3 Glacier Deep Archive. The recent data we store is tarred into single files for efficiency, but the older data we're trying to restore consists of thousands of individual files per batch, which is making the restoration process challenging.
S3 doesn’t support folder restores and requires files to be restored individually. As you can imagine, trying to restore such a large number of individual files using bash scripts is causing timeouts and issues with tracking the progress.
Has anyone experienced a similar issue with restoring large numbers of individual files from S3 Glacier? We’re considering S3 Batch Operations, but I’d love to hear if anyone has other strategies or best practices for efficiently handling this kind of large-scale restore, especially when dealing with a massive number of files.
TIA
Gitmoxi10 months ago
Hey Folks!!
I'm currently exploring a startup idea at the intersection of DevOps and Generative AI. As someone with hands-on experience in DevOps, I’d love to hear your perspective on the current challenges practitioners face and where you think GenAI could meaningfully help.
If you're open to it, I’d really appreciate a quick 30-minute chat. Would love to learn from your experience — and happy to share more about what I’m thinking too! Please like or comment on this message and I can reach out to you directly.
Thanks so much for considering, and hope to connect! 🙏 🙏
I'm currently exploring a startup idea at the intersection of DevOps and Generative AI. As someone with hands-on experience in DevOps, I’d love to hear your perspective on the current challenges practitioners face and where you think GenAI could meaningfully help.
If you're open to it, I’d really appreciate a quick 30-minute chat. Would love to learn from your experience — and happy to share more about what I’m thinking too! Please like or comment on this message and I can reach out to you directly.
Thanks so much for considering, and hope to connect! 🙏 🙏