03:33:58 GMT <mustmodify> Well, seems like there are two kinds of reads. The read-during-write scenario where I would like to be able to create / update / remove certain values by key... and the normal-read operation, during which I just want a whole ton of data exported quick.
03:34:21 GMT <mustmodify> Let's see. I could have an intermediate phase. Step 1, write hashes. Step 2, dump hashes to a list.
03:34:29 GMT <mustmodify> or I could have a list of keys.
03:34:34 GMT <mustmodify> and just dump that list.
03:34:53 GMT <mustmodify> I mean, dump the set of hashes with the keys whose names are in the list.
03:39:57 GMT <mustmodify> Well wait I could create a hash named bmi_data with keys like visit_1000 and visit_1012 etc and values that are json strings. Then use hvals to get the list of values.
06:44:05 GMT <*> antirez  wrote a design draft based on the "stream" data structure proposed here by forkfork
06:44:09 GMT <antirez> totally in love with the idea
09:39:36 GMT <manadart> Hi everyone. Since I missed it, can anyone give me the 411 on the proposed streams structure mooted yesterday?
13:29:23 GMT <mustmodify> antirez: ??
13:53:47 GMT <itamarhaber> mustmodify: looks like this one "12:49:33 GMT <forkfork> the module I'm planning on doing is to add a transaction log style data type - meaning that a very large number of subscribers can do something like pub sub without a lot of redis memory growth
13:53:47 GMT <itamarhaber> 12:52:03 GMT <forkfork> subscribers keeping their position in a message queue rather than having redis maintain where each consumer is up to and duplicating messages per subscriber" source: http://irclog.redis.io/redis-2016-05-20
13:57:10 GMT <mustmodify> nice. Something like Kafka ... linkedin's pubsub queue?
14:07:12 GMT <mustmodify> So this thing I'm doing... what kind of pattern is that? I mean, there's a worker... but the whole "spin up some threads to do work incrementally instead of doing the task in one go"... there's a name for that but it's not coming to me.
14:48:11 GMT <manadart> That does sound somewhat like Kafka - particularly consumers maintaining their own offset.
14:55:18 GMT <mustmodify> I haven't used it but I thought it sounded really awesome.
15:01:53 GMT <manadart> The whole stream idea can be quite powerful.
15:03:27 GMT <manadart> If you have a data structure bounded by a temporal window...
15:05:53 GMT <manadart> Exact example from a discussion at my job: emit x if a does not follow b within 2 minutes.
15:06:15 GMT <manadart> We use Streambase for this kind of thing, but it's pricey.
15:11:39 GMT <mustmodify> I see HDEL... is there a bulk delete?
15:11:45 GMT <mustmodify> I just want to get rid of the entire list.
15:12:11 GMT <mustmodify> Or should I `hdel hkeys hash_name`
15:12:47 GMT <manadart> DEL
15:13:25 GMT <mustmodify> Oh, I see.
15:13:28 GMT <mustmodify> duh.
15:13:29 GMT <mustmodify> thanks.
15:13:54 GMT <manadart> np
16:08:41 GMT <mustmodify> OK! I'm working on my first "big data" project so I have some questions that are probably ... sophomoric. I'm hoping this channel will not end up cursing my name. :) And by big data I mean... small-to-mid-size-data-but-big-enough.
16:09:22 GMT <mustmodify> There's this data schema called collection+JSON. It's super-extensible, useful, we're starting to use it in our APIs on this project.
16:09:47 GMT <mustmodify> it's kind of .... verbose. A tradeoff that comes with being extensible I guess.
16:10:27 GMT <mustmodify> I have a worker that generates data like this: https://gist.github.com/mustmodify/d22cbb30d4766b738439af6f8d36b85d
16:11:08 GMT <mustmodify> Sorry, I should be more clear. I have workers which generate one line of that json as a string and store it as a value in a hash.
16:11:21 GMT <mustmodify> and then I use hvals to get the complete list.
16:11:57 GMT <mustmodify> If I want to render this as collection+JSON, I could either write it as cJ initially in the worker, or transform it when I'm about to emit.
16:12:55 GMT <mustmodify> If I keep it like this I could more easily massage this data in other ways... like ... using it on the server side to find bad values. I could still do it the other way but cJ is, as I said more verbose and harder to parse.
16:13:17 GMT <mustmodify> Or I could just have the worker write data in the way each utility needs it written... or write other workers. So lots of options.
16:15:23 GMT <mustmodify> so the end question is ... should I couple the final representation with the worker?
16:18:02 GMT <mustmodify> To which my CS background says no. But in terms of optimization...
16:40:38 GMT <badboy_> 127.0.0.1:6379> bgsave
16:40:38 GMT <badboy_> (error) ERR
16:40:41 GMT <badboy_> that's a new one…
18:52:30 GMT <itamarhaber> mustmodify: If you want to bulk delete the fields in a Hash, just DEL the Hash's key
18:52:53 GMT <*> itamarhaber  should stop answering the history :)
18:53:59 GMT <mustmodify> yeah, someone mentioned that. I kind of thought of the del comnand as being for only values that you set with "set"
18:54:04 GMT <mustmodify> but obviously I was confused.