virtualhome.blog My blog about virtualized infrastructures, backup and disaster recovery topics and the cloud !
Related Articles
Object Storage with Veeam Sizing and Architecting – Part 2
In this blog series, I’m going to talk about plenty of things around the sizing and architecting for using Object Storage with Veeam Backup & Replication. Each part of this blog series covers different topics and things.
The second part of Sizing and Architecting for Object Storage with Veeam covers the following topics:
- On-premises vs cloud object storage
- How does Object Lock & Block Generation work with Veeam
This second part of the blog series should give you some hints, tips, and tricks on selecting the right location for your object storage. Furthermore, I try to explain how Block Generation works and why it is superior on the market right now compared to other data protection solutions.
On-premises vs cloud object storage
When deciding between cloud-based and on-premises object storage, there are many factors to consider. Mainly it’s about scalability, accessibility, costs, security and performance including latency!
Cloud Object Storage Pros
- Scalability won’t be a problem for the most customers as you can easily scale up and down. Based on your needs, scalability in the cloud is very easy to achieve. This is especially suitable for customers with fluctuating storage requirements.
- Ease of Use Data and objects stored in the cloud can easily be accessed from everywere. You only need an internet connection and of course the acces and secret keys.
- Costs: When it comes to cost, you pay for what you use. The cost factor is an OPEX investment and helps you pay while you scale.
- Deployment: Cloud Object Storage is easy to ramp up. You do not need to prepare rackspace, network and everything that requires the object storage system to be used within your datacenter. You simply book “the capacity” and it is very easy to do so
- Scale Out: Furthermore it allows you to “get another datacenter” for storing data in a quick manner. Most of the times leveraging a public cloud allows a customer to ramp up another site / DC without paying for the premises and all the things around it like electricity, space, rent and so on.
Cloud Object Storage Cons
- Security: Some people and customers prefer to not store their data in the cloud. Mainly because it is not managed by themselves. This is especially a thing when it comes to several security certifications or “good practices” or simply because of the attitude and willingness to store data in the cloud
- Internet Dependency: If you store your data in the cloud, it is obvious that you need a reliable internet connection or backup lanes to get access to your data. This might seem obvious, but it definetely needs to be considered when talking about disaster recovery or simply high availability.
- Internet Latency: When it comes to data retrieval there can be a network latency which is not very suitable for Instant Recovery for example. Not everyone is able to spend the money for an Azure Express Route or an AWS Direct Connect.
- Costs: Depending on how much and how big you scale in the cloud, the costs can definetely be high. This of course always depends on the customer.
On-Premises Object Storage Pros
- Control: If you own the object-storage system you obviously have full control over your data and the whole object-storage infrastructure. This means you can apply your security standards and generally treat that system as you would like to.
- Performance: On-Premises object storage can definetely be faster and more performant than cloud object storage as you do not rely on an internet connection. The limit of your performance with on-premises object storage systems is the network speeed and definetely the architecture of that system. Depending on the architecture some object storage vendors rely on Cassandra databases. There are also other metadata storing databases where others are storing metadata alongside the normal data and scale out the performance.
- Customization and Adaption: You can tailor the object-storage system to your specific business needs also from a sizing and performance perspective which makes it easy to integrate it with your already existing infrastructure.
On-Premises Object Storage Cons
- Cost: Just to mention, this is of course an CAPEX investment where you pay the whole system / capacity upfront. Also depending on the vendor you can start with a lower capacity and scale with your needs. Furthermore this is simply something which needs to be determined based on the financial planning and needs of the customer
- Scalability Challenges: Depending on the architecture and system you need to scale with differend nodes, network and additional ressources. This is of course obvious, but achieving a performance and speed comparable to block and file storage, some vendors really need a lot of nodes.
- Performance Challenges: Not all object storage vendors have systems which are capable in delivering awesome performance. This does not necessarily have to do with Write and Read performance, but rather with metadata operations as the objects stored in the database needs to be accessed. In this case I need to quote the Veeam BP guide again: “Veeam backups put a high load on Object storage, so a proper design is required” (Source: Veeam Best Practice Guide – Object Repository)
- Maintenance: You are responsible for maintenance, patching and upgrading that infrastructure. This means you will need to have ressources and IT-staff handy like for every other storage you manage on-premises.
Choosing between cloud and on-premises object storage depends on your specific business needs, budget, and long-term goals. Feel free to comment down below if I didn’t mention someting as there could be more pros and cons around cloud or on-premises storage.
My personal take on it
Due to the fact, that I operate only in Germany and EMEA, my observations are, that customers most likely want to have their backup data on-premises. This also applies to object storage. Of course using a Capacity Tier to copy data to a public cloud provider is an easy to ramp up solution, but with the downside of performance and latency when it comes to restoring. Reflecting on the last 3 years though, I think there is a shift in adopting S3 / object storage on-premises as it makes several things easier. Especially when talking about automation, REST and scalability, S3 / Object Storage gives you the experience of a cloud solution on-premises. This is something customers love, especially when it really works 🙂
How does Object Lock & Block Generation work with Veeam
It is well known, that Veeam Backup & Replication uses the AWS S3 Object Lock & Versioning feature to achieve “Immutability”. If you want to know more about how Object Lock & AWS S3 Versioning, then make sure to check out this page from AWS S3 because it explains it in detail.
Immutability means that data can’t be changed, modified or deleted. In Veeam Backup & Replication, immutability is used to protect a backup chain for a specific period of time. The specific period of time is set by the administrator who created the object-storage repository. As soon as immutability is turned on, Veeam prevents any data in the object storage repository from being deleted until the set time has passed. These immutability settings apply to the entire backup chain and all its restore points. In the end, this setting keeps your backups safe from accidents or ransomware / cyber threats.
Object Lock itself can be used in various constellations within Veeam Backup & Replication. Firstly, within a “Simple Backup Repository” utilizing S3, or S3-compatible object-storage. Secondly, with an object-storage bucket within a Scale-out-Backup-Repository used as a Performance Tier or Capacity Tier.
Setting the immutability in Veeam
Setting the immutability within Veeam is as simple as selecting the checkbox “Make recent backups immutable for” and specify a number of days.
What happens in the background is, that Veeam is setting the mode for that S3 bucket and then sets object lock & versioning accordingly. There are two different modes which can be used within AWS S3 or S3 compatible object-storage.
If you drill down into an S3 bucket and its folders to actual backup data by using S3 Explorer for example, you will see the following:
The screenshot shows, that that specific object is locked with “ACTIVE” object-lock until a specified point in time and that the mode is set to “COMPLIANCE”.
Since Veeam V12.1 you can also use “GOVERNANCE” as the mode for object-lock. If you want to do this, you need to create the following registry key:
S3GovernanceImmutabilityMode
When this setting is set to “1” than this means, that the Governance mode is being used. If it is set to “0” then the Compliance mode is being used.
Make sure to know the impact when using this registry setting, as this means a VBR-wide setting for every S3 bucket attached to that VBR server!
The difference between Compliance and Governance mode is very different, but both add valuable protection to your objects.
Compliance Mode
In Compliance mode, a locked object version is protected from being overwritten or deleted by any user, including the root user. Furthermore, its retention settings can’t be modified. This means, that the retention mode cannot be change and the period of retention can not be shortened. Also, the Compliance Mode makes sure, that a version can’t be overwritten or deleted as long as it is in the set retention period.
Governance Mode
In governance mode, users need special permissions to overwrite, delete, or change the lock settings of an object version. This mode allows most users to be prevented from deleting objects, while still letting certain authorized users modify retention settings or delete objects when you require it. This might also be helpful when you are a hoster or a service provider and have to deal with orphaned / decommissioned customers.
Also see this post on the Veeam Community Hub covering that Governance mode topic pretty well!
In addition, this may be common knowledge, but Object Lock itself only protects the objects inside the bucket. Not the bucket itself ! I repeat, not the bucket and its settings itself. This is very important when you choose an object storage solution. Some vendors do have features besides object lock for protecting the bucket. Furthermore, some vendors do have mechanisms in place to avoid setting retention settings, changing the retention mode, or generally changing anything around the bucket itself. This makes sense when it comes to a complete view on security and immutability. Your “backup repository” needs to be fully protected from all sides, not only from the “inside” with object lock.
Pure Storage FlashBlade Example
On the Pure Storage FlashBlade system for example, we have a feature called Retention Lock. Retention Lock helps enforce object and bucket retention settings that use Object Lock compliance mode. If it is set to “ratcheted (locked)”, manual eradication is blocked to protect objects locked in compliance mode, default retention cannot be reduced when the default retention mode is compliance, and all changes to default retention via S3 are denied.
Furthermore, this setting is only changeable with the Pure Technical Support. At least 2 approvers (up to 6) on the customers site need to validate any changes to the system. This setting really helps ensuring that Object Lock + Retention Lock completely covers the security of that system.
How does Block Generation work with Veeam
Block Generation is crucial when it comes to Immutability. I want to briefly explain it here and give you some example calculation. Block Generation is also very significant when it comes to sizing. Luckily, the new Veeam sizing calculator is aware of a Direct to S3 workload with immutability and reflects the block generation, which adds up capacity to the backup repository.
Block Generation is done to reduce I/O operations and therefore costs. This means, that Veeam will add days to the anticipated immutability expiration date. You do not need to configure this, as it is done automatically by Veeam. Veeam Backup & Replication reduces the frequency of access and modifications to the data blocks, by adding 10 or 30 days (based on the object storage). This also streamlines the retention of all backups pointing to the object-storage repository, as all have the same retention. On the backup market, this is pretty unique when it comes to Object Lock and its overhead because of versioning. There are other vendors out there who need twice the capacity or more than 30% capacity on top to get around with the object lock functionality.
Let’s do a quick example combining theory and the Veeam calculator together. This perfectly reflects the space usage when using S3 and object lock.
Calculation example for Object Storage with Veeam
Let’s dive into the calculator. In this example you can see that the first calculation with 235 TB is based upon 30 days of retention and 30 days of immutability. However, when I uncheck the “Enable Immutability” checkbox, you can clearly see, that it is only 177.1 TB.
If you are using Amazon S3 or IBM Cloud Object Storage, Veeam adds 30 days up to the retention policy. If you are using all other types of S3 / object-storage, for example an on-premises solution, Veeam adds 10 days to the retention.
In this example, Veeam adds up 10 days to the 30 days immutability.
This means 235 TB = 30 days + 10 days Block Generation = 40 days
If you double-check this with a rule of three, you can calculate the following:
235 TB = 40 days, whereby 177.1 roughly is 30.145 days.
This also means, that the Veeam calculator uses the +10 days for block generation. Therefore, make sure to consider this within your sizing, when you leverage Amazon S3 or IBM Cloud Object Storage.
Conclusion on Sizing and Architecting for Object Storage Part 2
I really hope this second part of the blog series around Sizing and Architecting for Object Storage was interesting. Feel free to comment and give some hints and tips which I can add here. There are more articles to come in this series, and I’m going to list them here.
- Sizing and Architecting for Object Storage with Veeam – Part 1
- Sizing and Architecting for Object Storage with Veeam – Part 2
Check out all Veeam related posts on my blog here: Veeam Blog Posts
