01:08:49 GMT <jamespedwards42_> Would it be possible to add the config epoch for each slot range to the CLUSTER SLOTS reply?
01:10:42 GMT <jamespedwards42_> I'm working on adding different partition handling strategies to my client and I figure with that info I could create a merged view using the latest epoch configs after querying multiple nodes.
01:18:40 GMT <jamespedwards42_> or is the config epoch global to each slot<=>node associations from the perspective of the node being queried? In which case I could pipeline a CLUSTER INFO call.
01:46:09 GMT <jamespedwards42_> I see that I can use CLUSTER NODES for this, but the documentation recommends to use CLUSTER SLOTS.
05:29:23 GMT <svm_invictvs> So, who uses Redisson?
05:29:35 GMT <svm_invictvs> Is there support for the begin/commit semantics in Redisson?
05:29:52 GMT <svm_invictvs> er, rather multi-exec
06:35:53 GMT <maodun> I'm trying to do some debugging via redis-cli. I'm debugging a lot of utf8 encoded text (chinese), but redis-cli is giving me back all the info in hex format. Is there a way I can ask it to show me the unicode characters instead? I tried setting the LC_ALL and LANG environment variables, but it didn't help
09:32:51 GMT <Tom___> Hello
09:34:39 GMT <Jognboo> Hello! Could anyone help me?
09:57:56 GMT <aphorise> I've found the cluster make in 3.0.7 rather unstable in high frequency writes - is it worth retrying on 3.2 branch? - just confussed what to build to get the latest version and what that is.
10:16:55 GMT <antirez> Storing sorted sets scores in binary inside RDB changed loading times from 3.9 to 3.1 in 1 million items test. Not huge but not a change to trow away for sure.
10:39:30 GMT <aphorise> antirez  RDB? - is that AWS RDB?
10:40:13 GMT <dvirsky__> antirez: that's what, 20%? that's awesome
10:55:22 GMT <markuman> aphorise: that has nothing to do with aws. http://redis.io/topics/persistence
10:56:52 GMT <aphorise> Thanks for the clarification.
10:58:23 GMT <antirez> dvirsky__: yep... I think the test I did actually showed less improvement than it could with real data.
10:59:47 GMT <dvirsky__> you mean with harder numbers to parse?
11:00:43 GMT <antirez> dvirsky__: yes, I tested with all scores "1.123" which is short
11:01:29 GMT <dvirsky__> yeah, then it will probably be even more significant
11:28:48 GMT <antirez> Added the feature to resolve the 9 chars name from the RDB signature ID in module-type loading ASAP... since to just say "can't load some module data" was really not enough.
11:28:49 GMT <antirez> 6209:M 01 Jun 13:27:46.008 # The RDB file contains module data I can't load: no matching module 'hellotype'
11:32:52 GMT <dvirsky__> antirez, will that cause a complete failure to load?
11:33:38 GMT <antirez> dvirsky__: yes
11:34:26 GMT <dvirsky__> so let's say I have data of a module and some other data and I want to split them to 2 instances, one without the module
11:34:40 GMT <dvirsky__> I'll have to do it manually
11:35:07 GMT <dvirsky__> perhaps it's worth a config option to ignore unknown types?
11:35:54 GMT <antirez> RDB is pretty much designed to be a "load or not" game, otherwise we need additional metadata to skip something that cannot be loaded
11:36:53 GMT <antirez> this also means that checking RDB for sanity when there is module data inside is hard
11:37:31 GMT <antirez> btw, the problem you mentioned is not module specific
11:37:43 GMT <antirez> like, I may want to put strings one side, lists in another instance
11:37:57 GMT <antirez> IMHO it's not Redis concern, but should be handled by, like AOF / RDB tools...
11:38:13 GMT <dvirsky__> well, we agreed we need rdb tool lib as part of redis
11:38:25 GMT <antirez> yep
11:38:31 GMT <antirez> but for how things are designed right now...
11:38:39 GMT <antirez> the RDB tools would require module-specific understanding
11:40:15 GMT <antirez> If we invest 4 bytes per each key saved, we can "jump" over.
11:40:46 GMT <antirez> well not 4 bytes, actually... Can be done in less using the normal RDB encoding
11:40:57 GMT <antirez> even just 1 byte if the serialization is short
11:41:10 GMT <antirez> perhaps it is a good idea to add this right now while we are in time.
11:41:39 GMT <*> antirez  returns to the term to make this change
11:41:42 GMT <dvirsky__> you mean the len of the data
11:41:45 GMT <dvirsky__> ?
11:41:46 GMT <antirez> yep
11:41:51 GMT <antirez> so that it is:
11:42:02 GMT <antirez> key | module ID | len | module data
11:42:07 GMT <antirez> currently "len" is missing
11:42:22 GMT <dvirsky__> so yes, by all means. make it a varint and go for it
11:42:42 GMT <antirez> all is varint inside RDB so the overhead we pay is proportional hopefully
11:42:55 GMT <dvirsky__> usually 1-2 bytes probably
11:43:04 GMT <dvirsky__> depends on the user of course but for most dbs
11:43:11 GMT <antirez> 1 byte up to 63 bytes and so forth
11:43:23 GMT <dvirsky__> 1 can encode up to 127
11:43:52 GMT <dvirsky__> (in the encoding I'm using in my engine at leat)
11:43:54 GMT <dvirsky__> (in the encoding I'm using in my engine at least)
11:43:56 GMT <antirez> the first two bits are used to understand if it's a 6 or 14 bit type
11:44:10 GMT <antirez> so that in 2 bytes you can store up to 2^14-1
11:44:23 GMT <dvirsky__> I'm using a different encoding
11:44:41 GMT <antirez> it depends on the target data, in Redis most lengths end to be > 63 < 2^14-1
11:44:44 GMT <antirez> so this was optimal
11:45:03 GMT <dvirsky__> the first bit signifies whether it's the last byte or not, the next 7 are the value
11:45:13 GMT <dvirsky__> you just iterate and stop when MSB is 0
11:45:19 GMT <dvirsky__> and shift
11:46:41 GMT <antirez> makes sense as well in certain situations, I've certain two initial bits combo where the next 6 have special meanings and stuff like that (to handle LZF compression and integer encoded strings)
11:47:36 GMT <dvirsky__> it's ideal if a lot of numbers are small, and it increases what's called "small". since I'm using delta encoding, most of the numbers I'm using are 1 byte
11:47:48 GMT <antirez> so a string which is a small number, is encoded in a single byte
11:48:09 GMT <dvirsky__> sure, that's a whole different story
11:48:19 GMT <dvirsky__> BTW google have an interesting algorithm called group varint
11:48:33 GMT <dvirsky__> where they encode 4 or 5 integers at once and use the first byte for meta data
11:48:37 GMT <dvirsky__> and it saves a lot of space
11:48:58 GMT <antirez> yep, assuming you have arrays of integers (not the RDB case)
11:49:02 GMT <dvirsky__> for the key header it's actually ideal
11:49:15 GMT <dvirsky__> but probably out of scope right now :)
11:50:48 GMT <antirez> Argh, to have the prefixed length complicates it all
11:51:01 GMT <dvirsky__> why?
11:51:18 GMT <antirez> currently the module method does not need to know the length beforehand, nor to buffer, it just writes whatever wants, like all the native types do with RDB
11:51:31 GMT <dvirsky__> oh right
11:51:36 GMT <antirez> to have the prefix I need to buffer, likely.
11:51:43 GMT <dvirsky__> you'll need to either buffer or seek back
11:51:45 GMT <dvirsky__> both suck
11:51:53 GMT <antirez> seek back is not possible because of the varint
11:52:06 GMT <dvirsky__> I did a trick once
11:52:07 GMT <antirez> so only buffering, apparently, is left
11:52:11 GMT <dvirsky__> that I read backwards :)
11:52:22 GMT <dvirsky__> you write forwards and read backwards
11:52:50 GMT <antirez> RDB is sequential... can't be inverted now ;-)
11:53:15 GMT <antirez> buffering for very large values is terrible
11:53:26 GMT <antirez> we can have a single huge data structure into a single key
11:53:37 GMT <antirez> to 2x memory while writing is totally bade
11:53:37 GMT <dvirsky__> no no don't buffer
11:53:39 GMT <dvirsky__> of course not
11:54:00 GMT <antirez> so what are the options? To have RDBs that can't be checked without module specific knowledge apparently
11:54:15 GMT <dvirsky__> or to lose the varint optimization
11:54:45 GMT <antirez> still needs to seek back, it is not possible since RIO targets are also sockets in disk-less replication
11:54:49 GMT <dvirsky__> and even then, the cost of seek will slow down rdb writes, especially on HDDs
11:54:51 GMT <antirez> we can't seek back as well
11:55:09 GMT <antirez> the only option I see is, like, to add two magic bytes at the end of every module-specific value
11:55:25 GMT <dvirsky__> or add the length just for module specific keys
11:55:26 GMT <antirez> so that RDB check tool can do a best-effort attempt
11:55:28 GMT <dvirsky__> which seems dirty
11:55:43 GMT <dvirsky__> and will require buffering
11:55:52 GMT <antirez> that was the initial project, only for module keys, but requires buffering
11:56:16 GMT <dvirsky__> so magic numbers at the end indeed
11:56:36 GMT <dvirsky__> combined with checking that right afterwards seems like a sane header
11:57:06 GMT <dvirsky__> but it means you'll need to escape the stream for those magic numbers
11:57:08 GMT <antirez> yep, it's very fragile since what follows every value is a single "type" byte that can be in the range 0-5 9-14
11:57:17 GMT <antirez> no... no escaping is possible
11:57:23 GMT <antirez> I mean, to do just a fragile best-effort thing
11:57:31 GMT <dvirsky__> so we're back to square one
11:57:34 GMT <antirez> so that RDB check tool says: Ok I'll try...
11:57:41 GMT <dvirsky__> having rdbs that might not load correctly is bleh
11:58:03 GMT <antirez> and will just try to seek "XY" (magic value) and check if next byte makes sense, otherwise seek next "XY" and so forth
11:58:08 GMT <antirez> kinda terrible
11:58:20 GMT <antirez> no you can always load correctly
11:58:22 GMT <antirez> in this model
11:58:24 GMT <dvirsky__> attack vector 101
11:58:26 GMT <antirez> the only problem you can have is to check
11:58:48 GMT <antirez> with the rdb-check-tool. May not be reliable to test for RDB correctness with the tool
11:58:55 GMT <antirez> the loading is deterministic since we just "skip" the two bytes
11:59:02 GMT <antirez> I mean: no module -> no loading anyway.
11:59:54 GMT <dvirsky__> it's either that or buffering or fragile shit
12:00:06 GMT <dvirsky__> so I'd go with robustness at the price of convenience
12:00:07 GMT <antirez> Yep, given that -> Let's do NOTHING AT ALL :-)
12:00:32 GMT <dvirsky__> at least we know why
12:00:53 GMT <antirez> yep... now actually we *can* do something... that is
12:01:01 GMT <antirez> now RDB check tool is a library inside Redis
12:01:07 GMT <antirez> so, if you load the module, it could work...
12:01:33 GMT <antirez> when it finds a module thing, it just calls the rdb_load function to skip the bytes
12:01:41 GMT <antirez> and frees the returned pointer with moduleType->free(ptr);
12:02:10 GMT <dvirsky__> that would work
12:02:27 GMT <antirez> yep, so getting rid of the prefixed length for now, we know there will be a way
12:13:06 GMT <pBBBBB> hi, is there a limit for pub sub? i keep getting disconnected when subscribing
12:56:03 GMT <makerbot_> has anyone seen an issue where the redis agent randomly crashes on a windows server?
14:00:58 GMT <dvirsky__> pBBBBB, this happens if you have a timeout defined on your pubsub connection
15:44:37 GMT <bobjames> hello, anyone using Pacemaker/Corosync with redis?
19:35:33 GMT <T-Sp00n> Hello
19:36:44 GMT <T-Sp00n> I was reding the documentation of Redis and noticed that both Cluster and Sentinel have failover, right? if yes, what is the difference? when is it better to use Cluster and when is it best to use Sentinel?
19:48:02 GMT <markuman> T-Sp00n: that are two different things
19:48:30 GMT <markuman> T-Sp00n: sentinal is handling just slaves and masters and monitoring
19:48:48 GMT <markuman> T-Sp00n: cluster is doring data sharding
19:50:50 GMT <T-Sp00n> In Redis cluster you have failover. Redis Sentinel also provides a failover mechanism by monitoring masters and slaves and initiating a failover.. My question what is the difference between the failover in each one and why do we have two?
19:53:35 GMT <T-Sp00n> @badboy_: I saw your slides on Redis Cluster where you mention failiure detection
19:54:47 GMT <badboy_> T-Sp00n: and?
19:55:07 GMT <badboy_> https://fnordig.de/2015/06/01/redis-sentinel-and-redis-cluster/
19:55:14 GMT <T-Sp00n> how is the failover different from Sentinel's
19:55:58 GMT <T-Sp00n> Oh let me check this first. Thanks
20:02:21 GMT <T-Sp00n> So I can use replication and Sentinel together right? if yes how do I configure replication?
20:03:03 GMT <T-Sp00n> just by adding this slaveof 192.168.1.1 6379 ?
20:03:29 GMT <T-Sp00n> well master IP and port
20:04:58 GMT <markuman> tias
20:05:31 GMT <T-Sp00n> ?
20:05:49 GMT <markuman> try it and see :)
20:06:33 GMT <T-Sp00n> I am just trying to understand the different ways that these can work together
20:08:00 GMT <T-Sp00n> can I have replication + sentinel? can I have also sharding? if yes what is best using Redis Cluster or Sentinel +replcation + sharding. what are the trade offs
20:09:18 GMT <markuman> uhh, I think you can have all you want. but it will turns into a configuration nightmare somehwere
20:09:57 GMT <markuman> but I've only played once with cluster. to maybe someone else can say more about it
20:10:32 GMT <T-Sp00n> @badboy_: what do you think?
20:11:03 GMT <T-Sp00n> markuman: Thank you :)
20:13:51 GMT <badboy_> T-Sp00n: did you read my post?
20:14:08 GMT <badboy_> you can't mix sentinel + cluster
20:14:27 GMT <T-Sp00n> yeah I got this
20:14:41 GMT <badboy_> standalone replication can be done with bare redis
20:14:55 GMT <T-Sp00n> and as in the sentinel section you mention masters and slaves then repliccation is standalone
20:14:56 GMT <badboy_> sentinel can monitor and failover
20:15:02 GMT <T-Sp00n> exactly
20:15:38 GMT <T-Sp00n> and sharding is not standalone right?
20:16:25 GMT <badboy_> right. Cluster is a sharding solution
20:16:52 GMT <badboy_> (you could of course build your own sharding and folks did so before Cluster ^^)
20:18:35 GMT <T-Sp00n> so main reason to use cluster is sharding. in case I need a dist sys solution (without sharding) it is better to use sentinel ?
20:18:57 GMT <badboy_> in what way distributed?
20:18:58 GMT <T-Sp00n> I am trying to figure out the trade offs
20:19:24 GMT <T-Sp00n> well scalable and fault tolerant
20:20:02 GMT <badboy_> depends on what you need
20:20:23 GMT <badboy_> dataser fits into a single machine? keep it simple, use replication and failover
20:21:00 GMT <badboy_> Cluster is a lot more compliated plus you need a smart client library
20:21:03 GMT <T-Sp00n> if not I need sharding and I use cluster
20:22:48 GMT <T-Sp00n> also looking at the slides, cluster and sentinel are like AP or CP?
20:23:49 GMT <badboy_> neither in the strict sense
20:24:28 GMT <T-Sp00n> what then?
20:25:54 GMT <badboy_> more in the direction of C but afaik it's not strictly consistent^^
20:26:57 GMT <T-Sp00n> because it is async replication ?
20:27:28 GMT <badboy_> yup
20:28:23 GMT <T-Sp00n> and it is not available because you write on masters only and read from masters/both ?
20:28:49 GMT <T-Sp00n> so more load on the master
21:07:50 GMT <wilornel__> Hey guys! I keep getting the "ERR max number of clients reached" error when I try to run a command inside `redis-cli`. I can get another server up and running without issues. Also, if I shut down that server which has a problem, and start a new one on the same port, I get the same issue again. Any ideas?