For each data providing service (e.g., e-mail, twitter, weibo,...), implement a set of tables ("stream", "messages", "notifications"), and use cron.

Just like we have open source package like all-auth for social authentications, we could create an all-social-data for data retrieval from these services. Most of social networking services rely on 3 inventions: "stream", "messages", "notifications", etc. Implementing retrieval and storage of these things for each service could allow later to merge their data. Just add a new service, and you can integrate its data into your stream copies on hard drive.

This could be used as a core application, that would be useful for creating a universal client for all social networks, allowing the user to have one's all social data in an easily analyzable (e.g., PostgreSQL) database on one's desktop computer.

Anything like that already exists? Link it, if so. :)


(suppress notifications) (Optional) Please, log in.

[ruta], for writing the analysis scripts and keeping the analysis results on your own hard drives ;)

[Mindey], what purpose would you use it for? there are various tools for social media analytics/social listening that allow to see feeds, searches by keywords from multiple accounts in one place. is the most popular one.