# "*": "https://raw.githubusercontent.com/wefindx/schema/master/project/oo-item.yaml" # "base:title": "0oo - Postgres synchronization solution" "og:title": "Postgres synchronization solution" "og:description": "This is a challenging computer problem, data synchronization but I have an idea I plan to use a inefficient rolling hash and sorting to solve the problem of synchronization. This is similar to rsync" "og:image": "https://avatars0.githubusercontent.com/u/28134655" "og:url": "/intent/52001/" "base:css": "/static/css/bootstrap.min.9c25540d6272.css" "base:extra-css": "/static/css/base.57997aeac1df.css" "base:favicon": "/static/favicon.acaa334f0136.ico" "base:body_class": "" "layout:logo": "/static/0oo.8d2a8bbef612.svg" "layout:index": "/" "layout:menu": "/menu/" "layout:categories": "/intents/" "layout:ideas": "/methods/" "layout:projects": "/projects/" "layout:users": "/users/" "layout:about": "/about/" "layout:help": "/help/" "layout:bug_report": "https://github.com/wefindx/0oo" "layout:login": "/accounts/login/" "layout:light-off": "/darken/?darken=true" "layout:set-multilingual": "/mulang/?mulang=true" "layout:lang": "Language" "layout:set-language-post-action": "/i18n/setlang/" "layout:csrf-token": "hh3DTbngWO8OEwyjz79B5Kr5ct5mEF6IEOFJWCgg2TxnPFK9nPKNc1PUsR49RHpf" "layout:input-next": "/project/52001/" "layout:languages": [{"code": "ja", "is-active": "false", "name": "日本語"}, {"code": "lt", "is-active": "false", "name": "Lietuviškai"}, {"code": "zh-hans", "is-active": "false", "name": "简体中文"}, {"code": "en", "is-active": "true", "name": "English"}, {"code": "ru", "is-active": "false", "name": "Русский"}, {"code": "oo", "is-active": "false", "name": "O;o,"}] # "item:title": "Postgres synchronization solution" "item:score": -8.0 "item:mission": "I want to take one set of data and synchronize it with another set where there could be changed to columns, new rows and deleted rows" "item:permalink": "/project/52001/?l=en" "item:base-administration": false "item:body": | This is a challenging computer problem, data synchronization but I have an idea I plan to use a inefficient rolling hash and sorting to solve the problem of synchronization. This is similar to rsync "item:destinations": | "item:source-date": "" "item:permalink": "/method/52001/?l=en" "item:owner": "chronological" "item:created": "2022-08-16T13:27:07.939085" "item:ownerlink": "/user/198/chronological" # "item:result:items": - "id": "t-146001" "url": "https://github.com/samsquire/postgres-sync" "text": | I created a vagrant setup with 3 postgres installations and created a cronjob to run the sync script "cost": -5.0 "currency": "HUR" "assets": | My vagrant setup uses persistent disks and uses ansible to deploy the cronjob and sync script. It is configured by YAML file. I've also installed psycopg2 and I found documentation on how to retrieve the tables in a database in Postgres. It's just a matter of writing the sync algorithm now. My problem is detecting which side is the winning copy. When one side changes the data there shall be a different hash and the changed rows are detected. This part I understand. The problem is detecting which side is the latest change and which side should win. I might need to introduce a version column. If I had a last updated timestamp field I could use that. Or a version column but I am expressly trying to avoid introducing new columns to the schema. It means it's a lot harder. "owner": "chronological" "ownerlink": "/user/198/chronological" "permalink": "/project/52001/?l=en#t-146001" "created": "2022-08-16T13:32:29.650418" "happened": "2022-08-16T13:28:10" "models": - "id": "t-151001" "url": "https://github.com/samsquire/postgres-sync" "text": | I implemented column hashing "cost": -3.0 "currency": "HUR" "assets": | I implemented a hash that takes previous data and current column and the previous hash to produce a hash of the entire database. This allows us to synchronize with the minimum of data transmissions when I write the synchronizer part which shall rehash all its own data, then retrieve the hash of a binary search of the sorted data. "owner": "chronological" "ownerlink": "/user/198/chronological" "permalink": "/project/52001/?l=en#t-151001" "created": "2022-09-09T20:23:41.331046" "happened": "2022-09-09T20:21:55" "models": "item:result:add": "/admin/hlog/task/add/?parent=52001" "item:transfer:items": "item:transfer:add": "/admin/hlog/operation/add/?parent=52001" "item:comment:add": "/projects/addnote?parent=52001" "item:comment:add:csrf_token": "hh3DTbngWO8OEwyjz79B5Kr5ct5mEF6IEOFJWCgg2TxnPFK9nPKNc1PUsR49RHpf" "item:comment:form": |
  • Mark if the comment raises new questions.
  • Mark if the comment contributes potential solutions.
  • Mark if the comment contributes facts for reasoning.
  • Mark if the comment contributes tasks for doing.
  • Please, log in. # "comments": "base:js": "/static/js/base.c7357c06cc89.js"