Btukfyl

I've got an online application and I want to record key user events, such as when a user logged in, logged out, viewed certain record. For each of these interactions I can easily add a new row to a sql table with the user ID, event and timestamp, but won't I end up with an excessively large table pretty quickly?

EG - 100,000 users, 10 logs per session on average - that's a million rows already? Is this a problem? Am I worried about nothing?

I know that SQL databases can hold almost unlimited amounts of data, but the problem comes from when you then want to manipulate the data, and the time it takes. If I had a table of 400 million rows, and wanted to quickly extract those relevant to a specific user, I imagine I'd start seeing slow loadtimes.

Assuming you get 100.000 users per month, that's going to be what, a couple of megabytes of storage? My phone takes photos larger than that.

Just do the simplest thing that works at first. If it's inserting records in a database, do that.

Find out for yourself - whats data you are realy need?

• I think it some summary data

So, every day run some cron script witch will be summarize needful data about an user to an another table with more less data length.
After calculating an necessary data delete unuseful rows from an original table.

If I had a table of 400 million rows, and wanted to quickly extract those relevant to a specific user, I imagine I'd start seeing slow loadtimes.

Databases support indexes for exactly this type of operation. So, if you design your database properly, you will not see poor performance.

For really basic data stream collection, any SQL database will work, but you will eventually want to look at something more aligned to this type of data. For this type of time-series data, a streaming DB type might be more appropriate:

• RethinkDB


• Apache Kafka


• Amazon Kinesis