00:04:02 GMT <workxep> someone asked if they can use redis-cli to check which config is being used
00:04:12 GMT <workxep> soemone did not try `info` or read the docs..
00:04:26 GMT <workxep> I would answer said person directly but they also did not stay in the channel after asking said question :(
00:07:41 GMT <workxep> ugh this is weird
00:08:07 GMT <workxep> http://pastebin.com/TijaNkxr
00:08:17 GMT <workxep> redis-sentinel breaking on me suddenly
00:08:21 GMT <workxep> just moved a tier on prod to SSDs
00:08:24 GMT <workxep> initiated the failback
00:08:33 GMT <workxep> and now things suddenly go wrong
00:08:49 GMT <workxep> it's like doing convert-to-slave ad infinitum on the same instances
00:08:55 GMT <workxep> over and over
00:09:47 GMT <workxep> redis 3.0.7
00:09:51 GMT <workxep> (as well as sentinel)
00:10:03 GMT <workxep> all 3 of them are spewing the same thing
00:11:23 GMT <workxep> hi ctoa
00:15:53 GMT <workxep> http://stackoverflow.com/questions/37394453/redis-failover-cluster-convert-to-slave
00:15:56 GMT <workxep> looks like the same issue.
00:19:02 GMT <workxep> I see antirez commented and closed this several hours ago
00:19:07 GMT <workxep> https://github.com/antirez/redis/issues/3285
00:33:15 GMT <workxep> manually setting slave state on the shards we're bringing back up this time (we had rebuilt that host so both claimed to be masters)
00:47:23 GMT <workxep> hi antirez
00:48:07 GMT <workxep> I'm experiencing some strange issue with redis-sentinel after upgrading a redis tier (3.0.7) to ssds, which necessitated rebuilding hosts. ran into a bug you closed out earlier
00:48:14 GMT <workxep> https://github.com/antirez/redis/issues/3285
00:48:39 GMT <workxep> we brought a set of master shards down, added slaveof to their configs manually and verified they work as slaves
00:48:45 GMT <workxep> but still can't seem to failover
00:49:15 GMT <workxep> it gave +reboot,-sdown on all the slaves. though interestingly one of the sentinels has -failover-abort-slave-timeout
00:54:14 GMT <workxep> http://pastebin.com/eQRWdhU6
00:54:26 GMT <workxep> the other sentinels just have some +new-epoch, nothing else interesting... hrm
00:55:45 GMT <workxep> same thing doing it on a sentinel on another host
00:59:58 GMT <workxep> http://pastebin.com/6z23NycA
01:00:06 GMT <workxep> currently can't initiate a failover on any shard.
01:10:30 GMT <workxep> hm, we did rename the CONFIG command -- so if Sentinel uses CONFIG on the shards, that would explain it perhaps..
01:59:13 GMT <workxep> well looks like I f'd myself over by renaming CONFIG on prod. it seemed like a prudent suggestion
01:59:31 GMT <workxep> I didn't think about that Redis Sentinel would be using that command and that it doesn't support renaming commands
01:59:34 GMT <workxep> :|
02:19:31 GMT <workxep> antirez: so, yeah, I screwed us over by renaming CONFIG on Redis, not realizing that Sentinel doesn't support renamed commands.
02:19:38 GMT <workxep> Mea culpa.
02:20:20 GMT <workxep> do I win #redis dumbass of the day award?
03:15:26 GMT <xep> well all's well that ends well
10:53:43 GMT <antirez> For strange differences between OSX and Linux networking stack, on OSX there is still the bug of Redis starting if 127.0.0.1 or ::1 is already bound to another process.
13:17:29 GMT <antirez> Ready to release 3.2.1, already on Git. Checking a few things before finalizing at redis.io
15:17:02 GMT <badboy_> antirez: website not yet updated?
15:18:48 GMT <antirez> badboy_: update di theory, checking
15:18:59 GMT <antirez> oops
15:19:07 GMT <badboy_> :D not pushed?
15:20:17 GMT <antirez> There is a conflict in the remote git so it does not update, going to check locally what happened
15:21:17 GMT <antirez> fixed, thanks
16:13:48 GMT <xep> ooh 3.2.1
16:15:20 GMT <xep> now I'm tempted to upgrade to 3.2 branch with 3.2.1 being out
16:15:40 GMT <xep> the ability to control Lua script replication side effects sounds super nice
16:17:48 GMT <nhooey> How can I delete a key with a newline in it?
16:18:33 GMT <badboy_> nhooey: in redis-cli: DEL "foo\nbar"
16:18:44 GMT <badboy_> in language of your choice: just pass the string
16:18:54 GMT <nhooey> badboy_: It doesn't work.
16:19:19 GMT <badboy_> then the thing you are sending is wrong?
16:20:35 GMT <nhooey> Ah, double-quotes.
16:30:06 GMT <xep> heh
16:30:16 GMT <xep> I learned a hard, hard lesson yesterday
16:30:27 GMT <xep> someone had given me some great advice like rename "KEYS" on prod -- that is still great advice.
16:30:32 GMT <xep> but I renamed CONFIG as well
16:31:00 GMT <xep> not realizing that Sentinel uses that very same command and doesn't support renaming of commands
16:31:29 GMT <xep> foolishly didn't test it on dev/stage because that's something I just wanted on prod. :|
16:33:39 GMT <xep> but for all the time we spent fire fighting, I was looking at logs of various things... like latency spikes and accuracy stats for certain systems. amazingly for all that, the impact wasn't tragic heh
17:25:11 GMT <keldwud> hi, I was reading through the redis.conf comments and noticed a comment about Redis Cluster and needing to wait for more users to deploy it in production. I plan on deploying redis cluster in production here in the next month or so. Would it be useful or helpful to share my info and documentation with somebody and if so, whom should I contact?
17:55:41 GMT <Kevin__> Hello!
18:27:52 GMT <bjpenn> exit
18:33:41 GMT <workxep> sorry for the nitty PR, but I noticed this when someone in the chan was asking about this error and I discovered this function and the comment above it
18:33:50 GMT <workxep> https://github.com/antirez/redis/pull/3324
18:41:05 GMT <minus> it can be /usr/bin/redis-sentinel too, right? not really clear from the PR or the comment
18:41:53 GMT <workxep> minus: I currently have it running as redis-sentinel-tagged for a dumb, historic reason
18:42:08 GMT <workxep> and it works because the comparison is using strstr() which is a substring match
18:42:27 GMT <workxep> instead of strcmp() which is used just below for --sentinel
18:43:38 GMT <workxep> minus: path is not part of argv[0]
18:43:46 GMT <workxep> you can convince yourself with a test program as I have:
18:44:04 GMT <workxep> http://pastebin.com/T9Bu3mr2
18:44:20 GMT <workxep> I put hello in my $HOME/bin
18:44:33 GMT <workxep> (compiled that as 'hello', sorry, wasn't clear)
18:44:37 GMT <workxep> $ hello
18:44:39 GMT <workxep> hello
18:45:09 GMT <workxep> it would be kind of lame if the path was part of argv[0] :)
18:45:31 GMT <workxep> so, I'm not sure of why the historic use of strstr() other than allowing some flexibility in the binary name
18:45:50 GMT <workxep> but my PR is purely to correct the comment since it's a substring check, not an exact string check.
18:46:50 GMT <workxep> technically it does mean that a binary named 'this-is-not-redis-sentinel-actually' would be recognized as redis-sentinel. but anyone who names such a thing that way is just being silly imho
19:54:56 GMT <tryfan> hi all, I'm trying to do some testing with sentinel.  I kill a master and it's set into sdown state, but stays in sdown forever.  down-after-milliseconds = 5000 and failover-timeout = 60000
19:57:54 GMT <tryfan> I've got 4 sentinel nodes, and the only line in the sentinel logs after killing the master is the sdown state
21:07:02 GMT <workxep> argh another person with a problem not staying on IRC
21:07:11 GMT <workxep> someone tell tryfan (next time he's on) that 4 != quorum
21:07:20 GMT <workxep> if you don't have quorum with sentinels, you're in for a bad time.
21:07:36 GMT <workxep> you can have 3 or 5, not 4.
21:08:01 GMT <workxep> much like the US Supreme Court needs *9* justices, not 8 ;\