c# - Why upload to azure blob so slow? -

i have custom stream used perform write operations directly page cloud blob.

public sealed class windowsazurecloudpageblobstream : stream {     // 4 mb top limit page blob write operations     public const int maxpagewritecapacity = 4 * 1024 * 1024;      // every operation on page blob has manipulate value rounded 512 bytes     private const int pageblobpageadjustmentsize = 512;      private cloudpageblob _pageblob;      public override void write(byte[] buffer, int offset, int count)     {         var additionaloffset = 0;         var bytestowritetotal = count;          list<task> list = new list<task>();         while (bytestowritetotal > 0)         {             var bytestowritetotaladjusted = rounduptopageblobsize(bytestowritetotal);              // azure not allow write many bytes want             // max allowed size per write 4mb             var bytestowritenow = math.min((int)bytestowritetotaladjusted, maxpagewritecapacity);             var adjustmentbuffer = new byte[bytestowritenow];             ...             var memorystream = new memorystream(adjustmentbuffer, 0, bytestowritenow, false, false);             var task = _pageblob.writepagesasync(memorystream, position, null);             list.add(task);         }          task.waitall(list.toarray());     }      private static long rounduptopageblobsize(long size)      {          return (size + pageblobpageadjustmentsize - 1) & ~(pageblobpageadjustmentsize - 1);      } 

i have low performance of write(). example:

stopwatch s = new stopwatch(); s.start(); using (var memorystream = new memorystream(adjustmentbuffer, 0, bytestowritenow, false, false)) {       _pageblob.writepages(memorystream, position); }  s.stop(); console.writeline(s.elapsed); => 00:00:01.52 == average speed 2.4 mb/s 

how can improve algorithm? how use parallel.foreach speedup process?

why 2.5 mb/sec, not 60mb/sec in official site or http://blogs.microsoft.co.il/applisec/2012/01/05/windows-azure-benchmarks-part-2-blob-write-throughput/

like you, had lot of performance issues page blobs - though not severe. seems you've done homework, , can see you're doing book.

a few things check:

• ensure vm not swapping (you can check in remote desktop). example extra-small 768mb memory vm's small practical use if you'd ask me.
• set own connection limits, if you're running small vm's. servicepointmanager.defaultconnectionlimit.
• larger pages give more performance.
• write in multiple threads (e.g. use tasks / async / await, if have lot do).

oh , 1 more thing:

• don't use emulator these kinds of things. emulator not representation of real-world azure, wrt. benchmarks.

the main reason you're access times slow because you're doing synchronously. benchmarks @ microsoft access blobs in multiple threads, give more throughput.

now, azure knows performance issue, why they've attempted mitigate problem backing storage local caching. happens here write data local (f.ex. in file), cut tasks pieces , use multiple threads write blob storage. data storage movement library 1 such libraries. however, when using them should keep in mind these have different durability constraints (it's enabling 'write caching' on local pc) , might break way intended setup distributed system (if read & write same storage multiple vm's).

why...

you've asked 'why'. in order understand why blob storage slow, need understand how works. first i'd point out there this presentation microsoft azure explains how azure storage works.

first thing should realize azure storage backed distributed set of (spinning) disks. because of durability , consistency constraints, ensure there's 'majority vote' data written stable storage. performance, several levels of system have caches, read caches (again, due durability constraints).

now, azure team doesn't publish everything. fortunately me, 5 years ago previous company created similar system on smaller scale. had similar performance problems azure, , system quite similar presentation i've linked above. such, think can explain , speculate bit on bottlenecks are. clarity i'll mark sections speculation think appropriate.

if write page blob storage, setup series of tcp/ip connections, store page @ multiple locations, , when majority vote received give 'ok' client. now, there few bottlenecks in system:

1. you have set series of tcp/ip connections throughout infrastructure. setting these cost time.
2. the endpoints of storage have perform disk seek correct location, , perform operation.
3. geo-replication of course take more time local replication.
4. [speculate] found lot of time spent during 'buffering' phase.

number (1), (2) , (3) here quite known. number (4) here result of (1) , (2). note cannot throw infinite number of requests spinning disks; well... can, system come grinding halt. so, in order solve that, disk seeks different clients scheduled in such way seek if know can write (to minimize expensive seeks). however, there's issue here: if want push throughput, need start seeking before have data - , if you're not getting data fast enough, other requests have wait longer. herein lies dilemma: can either optimize (this can hurt per-client throughput , stall else, mixed workloads) or buffer , seek & write @ once (this easier, adds latency everyone). because of vast amount of clients azure serves, suspect chose last approach - adds more latency complete write cycle.

regardless of that, of time spent (1) , (2) though. actual data bursts , data writes quite fast. give rough estimation: here commonly used timings.

so, leaves 1 question: why writing stuff in multiple threads faster?

the reason simple: if write stuff in multiple threads, there's high chance store actual data on different servers. means can shift our bottleneck "seek + network setup latency" "throughput". , long our client vm can handle it, it's infrastructure can handle well.