php - Prevent bots from scraping our content / overloading our server -


we planning offer job platform-service firms. have few thousand jobs offer our guest/visitors.

since yesterday noticed our server-load crazy , when checked logs saw had multiple site-request per second different ip addresses. order in pages called indicate same user / bot

we want available public if bots slowing our server massively down or forcing new hardware in trouble.

we displaying our job-content in iframes, encoder like: http://www.tareeinternet.com/scripts/iframe-encoder/

help solve our problem?

or options have? annoying since don't have user-sessions or recurring ip-addreses (i think using proxys switch regulary)

have checked headers recurring data? if they, example, have recurring user-agent can can block those:

  • apache: 

setenvifnocase user-agent "^wget" bad_bot  setenvifnocase user-agent "^emailsiphon" bad_bot  setenvifnocase user-agent "^emailwolf" bad_bot  <directory "/var/www">          order allow,deny          allow          deny env=bad_bot  </directory>

  • code: can check each requests specific header , redirect somewhere else.

Comments

Post a Comment

Popular posts from this blog

android - MPAndroidChart - How to add Annotations or images to the chart -

i working latest mpandroidchart library , looking way add annotations or images clickable. example: once data reaches value event occurs want able display user - there multiple events can occur annotations (not sure if right word) or image added @ point in chart. any pointers welcome. check documentation of yaxis . focus on setspacetop(...) , setaxismaxvalue(...) methods. for background: bar-shadow , disable calling: barchart.setdrawbarshadow(false) or change color: bardataset.setbarshadowcolor(int color)
Read more

javascript - Add class to another page attribute using URL id - Jquery -

i have attached id url. ex: when click link 1 , has #tab-2 id , takes me landing page. in landing page have find element has #tab-2 id have add class .active . i tried no luck. please 1 correct code. thanks js fiddle link html <div class="menu"> <ul> <li><a href="http://jsfiddle.net/gscyoa0j/1/#tab-2">link 1</a></li> <li><a href="http://jsfiddle.net/gscyoa0j/1/#tab-44">link 2</a></li> <li><a href="http://jsfiddle.net/gscyoa0j/1/#tab-74">link 3</a></li> </ul> </div> landing page <div class="tab"> <ul> <li><a id="tab-2" class="active" >link 1 content</a></li> <li><a id="tab-44" >link 2 content</a></li> <li><a id="tab-74" >link 3 content</a></l
Read more

firefox - Where is 'webgl.osmesalib' parameter? -

i'm trying webgl running on firefox (v37.0.2) via software acceleration mesa. tried following steps have been stated in article: http://www.binarytides.com/enable-webgl-firefox-ubuntu/ however, unable find parameter named ' webgl.osmesalib ' in about:config section? has been removed (or renamed)? if yes, how should instruct firefox use mesa's software acceleration webgl? looks support osmesa removed in favour llvmpipe.
Read more