Faster method for migrating AND processing data between MySQL databases -


i trying normalize schema. have 1 mysql database used in cms , data schema problematic. example, in order properties , sub-properties of entity, have query several tables , post-db processing. there couple sub-properties stored in json string in 1 field, , when parse json gives foreign key ids table in db. kind of mess talking db, processing stuff, talking db again rest.

i trying create script run on cron schedule take data existing system, process it, , put different database normalized schema.

i have initally created php script this, 1 category (out of ~800) containing 350 products (out of 100k) script takes 5 minutes complete. need quicker. script uses adodb library interaction mysql.

here options (since format question/answer), , i'm curious if 1 of these might exponentially better in performance:

  • raw php (no libraries)
  • php phalcon
  • perl
  • node.js
  • something else..

i think may want write "translating" procedures in mysql process old data , generate new entries directly. can faster other solutions.

if query performance not issue, can create views (procedures, functions) based on current schema , make further programming easier. materialized views can improve performance.

if have "transform" huge database outside mysql, can write "results" disk files first (text or sql) , import them mysql after "data" transformed.


Comments

Popular posts from this blog

android - MPAndroidChart - How to add Annotations or images to the chart -

javascript - Add class to another page attribute using URL id - Jquery -

firefox - Where is 'webgl.osmesalib' parameter? -