01:45:30 GMT <baiwfg2> hi. I'm new
08:07:15 GMT <mungustas> hi, I want to search for keys by pattern "*queue*"
08:07:22 GMT <mungustas> https://pastebin.com/raw/FL32TL9B
08:08:43 GMT <mungustas> how come KEYS *queue* and --scan --pattern "*queue*" gives different results ?!
08:42:52 GMT <mstaack> be careful using keys command in production
08:49:52 GMT <badboy_> mungustas: the dataset probably changed in between those two calls
09:00:51 GMT <mungustas> yeah I know
09:01:03 GMT <mungustas> I'm trying to avoid using KEYS
09:01:10 GMT <mungustas> by using --scan --pattern
09:01:34 GMT <mungustas> but was scratching my head why both commands give different results
09:01:58 GMT <mungustas> I am pretty sure dataset hasn't changed in between those calls
09:05:02 GMT <mungustas> https://pastebin.com/raw/sSbdPgJ1
09:05:14 GMT <mungustas> current results
09:09:17 GMT <mungustas> hm but maybe it does change
09:14:09 GMT <badboy_> ;)
09:14:18 GMT <badboy_> otherwise you would have found a serious bug and I doubt that
09:14:49 GMT <badboy_> > reports:webform_opens:processing:queue:2
09:15:06 GMT <badboy_> this looks like it is from some job queue, probably workers storing their progress in Redis
09:15:15 GMT <badboy_> would make sense that those keys are gone when the workers finished
09:20:21 GMT <ABK> we just updated redis to version 3.2.8 but can't get the sentinel client-reconfig-script to work anymore. Any ideas?
10:09:19 GMT <badboy_> ABK: there was a similar bug report
10:09:28 GMT <badboy_> ABK: what version did you have before?
10:11:39 GMT <badboy_> https://github.com/antirez/redis/issues/3903
11:28:40 GMT <mikecmpbll> is the redis sentinel suitable for client failover? if so, do people ever use redis sentinel and something like haproxy, if so why?
11:30:39 GMT <r23r> hi
11:36:14 GMT <Inge-> o_O
11:41:20 GMT <mstaack> hi
12:23:33 GMT <ABK> @badboy that's my colleague that wrote that questions.  We upgraded from 2.0.x
12:23:56 GMT <badboy_> ah :D
12:24:07 GMT <badboy_> let me check my sentinel setup
12:24:48 GMT <ABK> we can't find any docs other than the small inline ones in the configuration file.
12:25:30 GMT <badboy_> yeah
12:25:33 GMT <badboy_> well, it should work.
12:25:47 GMT <badboy_> give me a second, I'll try to reproduce it
12:31:40 GMT <badboy_> ABK: I can't reproduce it at all
12:32:15 GMT <badboy_> 1. Check configuration of all Sentinels
12:32:38 GMT <badboy_> 2. Check permissions of that script on all Sentinels (the script is on all Sentinel servers, right?)
12:32:51 GMT <badboy_> 3. Check that it has an appropiate shebang in the file
12:33:02 GMT <badboy_> 4. Check that the name nuvomaster is correct
14:07:31 GMT <ABK> @badboy 1) it's on every sentinel server and they are correct. 2 & 3) The script was getting executed before the update. 4) It's correct
14:08:16 GMT <badboy_> 2&3) still check it again ;)
14:08:22 GMT <ABK> @badboy Does your scripts get executed on failover?
14:08:39 GMT <badboy_> let me check the actual logs
14:08:41 GMT <ABK> @badboy We just did that now
14:08:42 GMT <ABK> :)
14:12:19 GMT <badboy_> funnily enough the docu is actual wrong it seems
14:12:27 GMT <badboy_> # <state> is currently always "failover"
14:12:46 GMT <badboy_> it's always "start"
14:14:35 GMT <ABK> @badboy We have a echo statement at the top of the script and it seams that they are never executed
14:14:51 GMT <badboy_> Notification Mi 29. Mär 14:31:19 CEST 2017
14:14:51 GMT <badboy_> +switch-master mymaster 127.0.0.1 6381 127.0.0.1 6380
14:15:08 GMT <badboy_> ABK: bash scripts?
14:15:32 GMT <byoungb> Does anyone know of a way too keep a slave that is just coming online from being used as a read slave from a redis sentinel client?
14:16:05 GMT <ABK> @badboy So your scripts are getting executed? Yes, it's bash scripts
14:16:26 GMT <badboy_> yes, they are executed
14:16:28 GMT <byoungb> When I add a redis slave node to the cluster I get a bunch of "BusyLoadingError: Redis is loading the dataset in memory" from clients trying to do reads on the not fully replicated slave
14:16:39 GMT <badboy_> ABK: again, do you have a shebang in those scripts?
14:16:56 GMT <badboy_> byoungb: that is expected
14:17:28 GMT <ABK> @badboy The perms used to be root:ubuntu, and that worked fine with 2.0.x; now we've set them to redis:redis, and they dont run. Yes, has a shebang, scripts havent been modified since the apt upgrade
14:17:33 GMT <byoungb> anyway to mitigate this?
14:18:07 GMT <byoungb> I will look at the python client and see if I can try another slave automatically
14:18:31 GMT <ABK> @badboy And just to clarify, we set the perms to redis:redis after the problem manifested
14:19:05 GMT <badboy_> byoungb: no. while the dataset is loaded, Redis will refuse to answer reads or writes
14:19:21 GMT <badboy_> ABK: no more ideas then.
14:19:30 GMT <badboy_> ABK: start tracing/debugging what's happening
14:20:03 GMT <ABK> ok thx
14:22:45 GMT <byoungb> are slaves that are loading the dataset marked as odown or sdown?
14:25:49 GMT <byoungb> @badboy I understand that... just doesn't seem to be very HA if adding a server to the sentinel cluster causes issues with some of my client connections
14:26:39 GMT <badboy_> byoungb: a slave loading a dataset will answer to PINGS with -LOADING and thus considered up
14:27:25 GMT <badboy_> answering to requests without a fully loaded dataset would mean to answer inconsistently, which is not much better than answering with an error
14:27:36 GMT <badboy_> (of course depending on your requirements)
14:27:53 GMT <badboy_> and your client can always decide to just ask the master in that case
14:37:07 GMT <byoungb> okay well looks like I get to maybe contribute to the python redis client
14:37:37 GMT <badboy_> :)
15:14:53 GMT <LSN> Anyone know what "-script-timeout" means in the redis-sentinel.log?
15:41:06 GMT <badboy_> LSN: raised if any scripts runs for more than 60 seconds
15:41:13 GMT <badboy_> (and the script is then killed)
15:57:47 GMT <LSN> Thanks badboy. Our scripts are placed on Amazon's NFS (EFS), its not fast, guess that could be a problem.
16:00:32 GMT <byoungb> badboy_: the LOADING error is not sent at connection time.... is there anyway that could be enabled? I just went through redis-py and how everything is handled... and there is no good way for me to fail over to another slave or to the master when the LOADING error is thrown at the command level and not at the connection level.
16:01:06 GMT <badboy_> no
16:03:10 GMT <byoungb> any ideas? because redis-py sets up a connection pool with all of the slaves in a round robin fasion and then fails to the master.... but this all happens when creating the connection pool
16:04:46 GMT <byoungb> so if a slave exists even if it is currently loading it is the only thing that reads will be sent to since it only adds the master if there are no slaves it could connect to
16:05:33 GMT <byoungb> I guess I could add a ping during the connection block to make sure it does not respond with LOADING
16:05:41 GMT <byoungb> does that sound acceptable?
16:10:38 GMT <badboy_> I don't know the internals
16:11:19 GMT <byoungb> yeah I guess I was just confirming that PINGing a slave right after connecting to it will return LOADING if it is loading in the dataset
16:11:51 GMT <badboy_> it should
16:12:07 GMT <byoungb> badboy_: cool thanks