c++ - Where is a delay in an HTTP POST coming from? -

i developing web service on ubuntu 14.04 in c++ using cpp-netlib in asynchronous mode. service needs respond data sent on http post in message body. observing poor performance when receiving input larger 1k, , want fix that.

if data relatively small, less 1k, server receives data instantly. if data more 1k, there 1 second delay after asynchronous read callback function first invoked before first chunk of actual data presented callback. after initial delay, subsequent chunks arrive instantly.

how can eliminate delay? artifact of using curl post test data? how can test performance of curl posting data?

you can find minimal source code demonstrating problem on github. here's command use post data server:

rcook$ curl -d @asyncdaemon.h http://localhost:8787/foo 

here's example of output (annotated):

rcook$ ./async_daemon  1431387368.321863: asyncdaemon constructor 1431387368.322446: receive thread beginning  *** it's waiting connection here.  1431387371.536191: begin transaction 0 on thread 24050 1431387371.536237: transaction 0 constructor 1431387371.536273: received 1206 byte request /foo 127.0.0.1:49402 1431387371.536312: invoked asynchronous read 1431387371.536321: end transaction handler 1431387371.536335: begin asynchronous read callback on thread 24050 1431387371.536348: read 0 bytes 1431387371.536386: invoked asynchronous read 1431387371.536394: end asynchronous read callback  *** asynchronous read callback invoked quickly, gets no data. *** there pause of on 1 second before asynchronous *** read callback invoked again.  1431387372.537203: begin asynchronous read callback on thread 24050 1431387372.537253: read 1024 bytes 1431387372.537307: invoked asynchronous read 1431387372.537317: end asynchronous read callback  *** there no significant delay when reading next chunk.  1431387372.537429: begin asynchronous read callback on thread 24050 1431387372.537469: read 182 bytes 1431387372.537478: finished reading body 1431387372.537746: wrote response 1431387372.537763: transaction 0 destructor 1431387372.537772: end asynchronous read callback  *** server killed keyboard interrupt.  ^c1431387375.382186: terminating signal 2 1431387375.382231: initiating shutdown 1431387375.382241: stopping server 1431387375.382363: server run finished 1431387375.382423: receive thread ending 1431387375.382522: asyncdaemon destructor 

as can see, after asynchronous read callback invoked first time (and receives 0 bytes of data, btw), requests chunk of input. @ point there pause of on 1 second before input arrives, 1431387371.536394 1431387372.537203 in example. what's going on during time? how can eliminate delay?

i have done research online, , run several experiments (synchronous vs. asynchronous mode cpp-netlib (no effect), curl vs. libcurl (no effect)), have not been able find answer.

update: tcp dump

upon jxh's suggestion, ran tcp dump during sample transaction:

00:28:01.304446 ip6 localhost.52265 > localhost.8787: flags [s], seq 3956487146, win 43690, options [mss 65476,sackok,ts val 395479802 ecr 0,nop,wscale 7], length 0 00:28:01.304461 ip6 localhost.8787 > localhost.52265: flags [r.], seq 0, ack 3956487147, win 0, length 0 00:28:01.305014 ip localhost.49421 > localhost.8787: flags [s], seq 1668603425, win 43690, options [mss 65495,sackok,ts val 395479803 ecr 0,nop,wscale 7], length 0 00:28:01.305039 ip localhost.8787 > localhost.49421: flags [s.], seq 4010788604, ack 1668603426, win 43690, options [mss 65495,sackok,ts val 395479803 ecr 395479803,nop,wscale 7], length 0 00:28:01.305079 ip localhost.49421 > localhost.8787: flags [.], ack 1, win 342, options [nop,nop,ts val 395479803 ecr 395479803], length 0 00:28:01.305185 ip localhost.49421 > localhost.8787: flags [p.], seq 1:176, ack 1, win 342, options [nop,nop,ts val 395479803 ecr 395479803], length 175 00:28:01.305210 ip localhost.8787 > localhost.49421: flags [.], ack 176, win 350, options [nop,nop,ts val 395479803 ecr 395479803], length 0 00:28:02.306555 ip localhost.49421 > localhost.8787: flags [p.], seq 176:1382, ack 1, win 342, options [nop,nop,ts val 395480053 ecr 395479803], length 1206 00:28:02.306620 ip localhost.8787 > localhost.49421: flags [.], ack 1382, win 1373, options [nop,nop,ts val 395480053 ecr 395480053], length 0 00:28:02.307223 ip localhost.8787 > localhost.49421: flags [p.], seq 1:52, ack 1382, win 1373, options [nop,nop,ts val 395480053 ecr 395480053], length 51 00:28:02.307270 ip localhost.49421 > localhost.8787: flags [.], ack 52, win 342, options [nop,nop,ts val 395480053 ecr 395480053], length 0 00:28:02.307494 ip localhost.8787 > localhost.49421: flags [p.], seq 52:66, ack 1382, win 1373, options [nop,nop,ts val 395480053 ecr 395480053], length 14 00:28:02.307522 ip localhost.49421 > localhost.8787: flags [.], ack 66, win 342, options [nop,nop,ts val 395480053 ecr 395480053], length 0 00:28:02.307765 ip localhost.8787 > localhost.49421: flags [f.], seq 66, ack 1382, win 1373, options [nop,nop,ts val 395480053 ecr 395480053], length 0 00:28:02.307867 ip localhost.49421 > localhost.8787: flags [f.], seq 1382, ack 67, win 342, options [nop,nop,ts val 395480053 ecr 395480053], length 0 00:28:02.307917 ip localhost.8787 > localhost.49421: flags [.], ack 1383, win 1373, options [nop,nop,ts val 395480053 ecr 395480053], length 0 

i'm not experienced tcpdump, looks 175 bytes flow server (http headers?), after delay of little on 1 second, 1206 bytes flow server, followed 51 byte chunk minimal delay, followed server response.

this tells me delay being introduced on client side, in curl. know why?

problem solved, debugging , diagnostic techniques suggested @jxh.

adding --trace - --trace-time curl command revealed curl spending mysterious second waiting server return 100 continue response before sent rest of request:

01:31:44.043611 == info: connected localhost (127.0.0.1) port 8787 (#0) 01:31:44.043726 => send header, 175 bytes (0xaf) 0000: 50 4f 53 54 20 2f 66 6f 6f 20 48 54 54 50 2f 31 post /foo http/1 0010: 2e 31 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 .1..user-agent:  0020: 63 75 72 6c 2f 37 2e 33 35 2e 30 0d 0a 48 6f 73 curl/7.35.0..hos 0030: 74 3a 20 6c 6f 63 61 6c 68 6f 73 74 3a 38 37 38 t: localhost:878 0040: 37 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 7..accept: */*.. 0050: 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 content-length:  0060: 31 32 30 36 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 1206..content-ty 0070: 70 65 3a 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f pe: application/ 0080: 78 2d 77 77 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e x-www-form-urlen 0090: 63 6f 64 65 64 0d 0a 45 78 70 65 63 74 3a 20 31 coded..expect: 1 00a0: 30 30 2d 63 6f 6e 74 69 6e 75 65 0d 0a 0d 0a    00-continue.... 01:31:45.045626 == info: done waiting 100-continue 01:31:45.045831 => send data, 1206 bytes (0x4b6) 

it known deficiency in cpp-netlib (at least of version 0.11.0) not support sending 100 continue response curl expecting.

the solution became convincing curl not wait 100 continue response. discovered here, adding -h 'expect:' curl command line trick. that, entire transaction takes millisecond.

since answered own question, won't accept answer few weeks give others chance contribute better.