Faster method for migrating AND processing data between MySQL databases -
i trying normalize schema. have 1 mysql database used in cms , data schema problematic. example, in order properties , sub-properties of entity, have query several tables , post-db processing. there couple sub-properties stored in json string in 1 field, , when parse json gives foreign key ids table in db. kind of mess talking db, processing stuff, talking db again rest.
i trying create script run on cron schedule take data existing system, process it, , put different database normalized schema.
i have initally created php script this, 1 category (out of ~800) containing 350 products (out of 100k) script takes 5 minutes complete. need quicker. script uses adodb library interaction mysql.
here options (since format question/answer), , i'm curious if 1 of these might exponentially better in performance:
- raw php (no libraries)
- php phalcon
- perl
- node.js
- something else..
i think may want write "translating" procedures in mysql process old data , generate new entries directly. can faster other solutions.
if query performance not issue, can create views (procedures, functions) based on current schema , make further programming easier. materialized views can improve performance.
if have "transform" huge database outside mysql, can write "results" disk files first (text or sql) , import them mysql after "data" transformed.
Comments
Post a Comment