java - Getting puzzling performance with InputStream and Scanner -

i seem running odd performance issue inputstream , scanner.

i'm using tail -f on log file, live streaming code. when approximately ~7,500,000 lines in 60 seconds(125,000 lines per sec). fair enough.

now since log file being written faster 125,000 lines/s grep log file narrow down on results.

when run updated tail -f commmand in linux shell approximately 40,000 lines in 60 seconds (667 lines/s). beneath 141,000 lines in 60 seconds when ungrepped.

however when run exact same command through code, 16,000 lines in 60s (266 lines/s)! somewhere i'm losing half of lines , i'm scratching head wondering why.

code:

  public boolean call() {   //send command no wait.     final printstream out = new printstream(ecli.getoutputstream());      //notice command terminator "\n" needed here command executed.     out.print(tailcommandbuilder(logdir,filename, filters));     out.print((char)4);     out.print('\n');     final scanner s = new scanner(ecli.getinputstream());     out.flush();     //inputstream in =  new bufferedinputstream(ecli.getinputstream());     stopwatch sw = new stopwatch();     sw.reset();     sw.start();     log.info("stopwatch started.");     double linecounter = 0;     //final bufferedreader buffreader = new bufferedreader(new inputstreamreader(in));     try {         loop:         while (keeprunning.get()) {              if (s.hasnextline()) {                 string st = s.nextline();                     linecounter++;             } else {                 keeprunning.getandset(false);                 try {                     thread.sleep(1);                 } catch (interruptedexception e) {                     // todo auto-generated catch block                     e.printstacktrace();                 }             }         }     } catch (exception e) {         // todo auto-generated catch block         log.error("error in ioread: " + e.getmessage() + e.getstacktrace());      } {         sw.stop();         log.info("total time: " + sw.gettime());             s.close();         log.warn("total lines read: " + linecounter);         ecli.clearbuffer();         ecli.disconnect();     }      return true; } 

edit:

as asked before, ecli instance of 'extended cli' class implements command line interface type instance. uses inputstream/outputstream communicate.

i not use --line-buffering grep.

i think i'm narrowing down on answer here.

i've revamped front-end code have gone 50% 66% log capture percentage when grepped. difference how new implementation pulls inputstream.

the logs seem 'burst' input. i'm thinking somewhere there buffer needs increased handle burst of info , allow logs average out.

either buffer needs increased in bufferedreader, inputstream or shell.