Storage Accounts
Append Blobs slowness on reading
Append blobs are optimized for appending operations, making them a perfect fit
for data created continuously over time, like log files, sensors/measurements data, etc.
Internally they are organized as a sequence of blocks of data.
Details here.
Despite their fastness in append operations, reading from them could be quite slow,
particularly when the append-blobs are created with many write operations,
resulting in many internal separated blocks of data. In such a case, the solution is to create
a temporary traditional block-blob from the append-blob we want to download, and download it instead of the original one.
Obviously the “copy” operation must be done online, storage-account on storage-account, using a temporary SAS uri.
Here, a piece of C# code showing:
- creation of different 10 MB append-blobs composed by 1, 10, 1000 and 10000 blocks.
- creation og block blobs from append blobs
- download of append and block blobs
Below the download times. Notice the increasing time required to download append blobs with the growing of the number of internal blocks. On the other hand, the corresponding block blobs, built from append-blobs, are always fast.
Every file is 10 MB.
| File | # internal blocks | Time |
|---|---|---|
| 1_10000000.append.dat | 1 | 1.567 s |
| 10_1000000.append.dat | 10 | 1.432 s |
| 100_100000.append.dat | 100 | 1.618 s |
| 1000_10000.append.dat | 1000 | 5.239 s |
| 10000_1000.append.dat | 100000 | 23.592 s |
| 1_10000000.block.dat | n.a. | 1.544 s |
| 10_1000000.block.dat | n.a. | 1.455 s |
| 100_100000.block.dat | n.a. | 1.375 s |
| 1000_10000.block.dat | n.a. | 1.433 s |
| 10000_1000.block.dat | n.a. | 1.476 s |
Diagnostic settings
After activating Diagnostic Settings on a storage account, it is possible to trace every single API call targeting the account. If you choose to send logging data to a storage account, a json log file is created. It contains many information: source ip, request and response size, user-agent, etc. [1] Among them, the hash of authentication “token” used. If the authentication method was the storage account key, it is exactly the SHA256 of the key used. This piece of C# generated the same hash starting from a key as a string, exactly the one you see and copy/paste from the Azure Portal.
string key = "abcd...==";
Console.WriteLine(Convert.ToHexString(SHA256.Create().ComputeHash(Convert.FromBase64String(key))));
[1] https://docs.microsoft.com/en-us/azure/storage/tables/monitor-table-storage-reference#resource-logs