00:00:09 GMT it is possible to configure redis using env vars' 00:40:12 GMT Anyone running Redis on kubernetes in some sort of HA mode? I have only found this example: https://github.com/kubernetes/kubernetes/tree/master/examples/storage/redis 06:46:53 GMT felixjet: no, but using cmdline args 12:29:13 GMT Hello folks! First time here, so apologies if this turns out to be a noobish question. In our company, we've been using Redis as a dumb cache that can only be queried with a particular set of keys. So far so good, but I would love to do a bit more than that and make the querrying more intelligent. This is pretty much the data-set. Every day, a set of activities per user gets added to redis. Now, I would like to query all activities o 12:30:44 GMT Also, I would like to query atleast one more level, say, all the activities where the person did X. Now, I've considered using ZRange/ZAdd to play around with UTC timestamp as an index, but that takes care of one level. What about querrying another level in? What could be a good solution for this? 12:31:34 GMT Hello folks! First time here, so apologies if this turns out to be a noobish question. In our company, we've been using Redis as a dumb cache that can only be queried with a particular set of keys. So far so good, but I would love to do a bit more than that and make the querrying more intelligent. This is pretty much the data-set. Every day, a set of activities per user gets added to redis. Now, I would like to query all activities o 12:57:04 GMT H1ccup: your message got cut off, seems like the web client doesn't break it into multiple messages automatically 12:57:23 GMT you can't really search in redis 13:00:17 GMT @minus what about using a solution like this? https://gist.github.com/observerss/7129377 13:00:39 GMT minus: what about using a solution like this? https://gist.github.com/observerss/7129377 13:00:50 GMT got you the first time 13:01:11 GMT a typo in the headline doesn't make a good picture :D 13:01:49 GMT haha, I initially thought that it was a wrapper around Redis. And then, well. 13:03:12 GMT i'd probably go with a RDBMS instead of Redis if i did something like that 13:03:56 GMT I do have RDBMS (postgres) sitting underneath all this 13:04:17 GMT is it too slow to handle the full load? 13:04:23 GMT but the idea is to serve millions and millions of row faster, where people will mostly query between range 13:04:30 GMT and at most kind of activity the person is doing 13:09:35 GMT https://gist.github.com/observerss/7129377 ← the lua script in this should be using ZREVRANGEBYSCORE btw 13:11:08 GMT yes, I was talking more about the idea. I think I would go with my own implementation. But wanted to know how scalable such a solution, as this might end up creating too many keys, and too much overlap of data 13:12:24 GMT scalable? well, if you can separate it by user you can throw it at redis cluster and scale just fine 13:12:58 GMT hmm, makes sense. 13:13:28 GMT is there any need of writing advanced data-structures on top of redis? Say like what twitter did to make it work for our needs? 13:13:29 GMT my main concern would be how to get the data into redis reliably and consistently 13:14:10 GMT This redis cluster would be just used for reads. Everyday evening, or late night, new data will be sorted and dumped into it and made available again. 13:15:51 GMT reasonable. 13:16:38 GMT how about just adding a few postgres slaves instead? i can't imagine it being slow when you've got the index on the right columns 13:18:57 GMT well, if a table is say around 100 GB 13:19:22 GMT and you want to query a bunch of things a particular user has done from it, knowing that it doesn't change realtime, won't it be extremely slow? 13:21:29 GMT if you're querying by time and you have the time column indexed? blazing fast. filtering by time and specific tasks: just as fast (without some special index for it). faster in redis if you create the right index, but will need some extra memory 13:25:36 GMT surely, json data indexed by time on redis, and aggregated by time over redis would be much, much quicker than doing the same with postgres? 13:25:52 GMT just for the fact that redis is in-memory. 13:26:38 GMT indeed 13:28:52 GMT my only concern is this 13:29:56 GMT how denormalize can you possibly make a data-set. My dumb maths and some intuition suggests that only till secondary indexing - dealing with two indexes. Tomorrow if we go deeper, and want to query another subset of the data, data stored on Redis becomes useless 13:30:02 GMT or rather, it becomes too much to handle 13:37:47 GMT what do you mean by secondary indexing? 13:41:52 GMT I mean, querying a data using another index. So, say you can query a dataset using timestamp, or by a set of activities theuser does 13:47:40 GMT so you'd create indices for every thing you'd like to be able to query, containing ids of the data; for querying you fetch the ids from the indices, intersect them and fetch the respecitive stored data. doing that client-side will be rather slow because of network latency; doing it in lua has the potential of blocking redis if it's a lot of data. 13:52:30 GMT so how would you suggest to go about it? 13:55:40 GMT 1) write a baseline benchmark for postgres 2) get data into redis 3) get indices for data queried in 1) into redis (zset for time-based indices, sets for fixed-word ones) 4) write client-sided query solution for redis + benchmark 5) translate to lua and benchmark that as well 13:56:56 GMT benchmark for maybe 2-3 most common queries 14:00:51 GMT hmm 14:01:02 GMT why do the lua part? won't a python wrapper work good here? 14:02:01 GMT yeah, you can probably do it decently in python if you batch up the operations well 14:02:28 GMT but lua is probably a bunch faster 14:05:40 GMT Cool 14:05:43 GMT Thanks a lot sir. 14:08:51 GMT you're welcome, have fun building it. would be cool if you could show me some benchmark results when you get it done 14:09:20 GMT Most definitely.