00:43:22 GMT <xep> ahfeel: are you perchance using DNS?
00:43:45 GMT <xep> my rule on production: Avoid DNS. Use IPs.
00:44:34 GMT <xep> ahfeel: it's also best to have some kind of connection pooling to avoid having to keep opening new connections to Redis
00:45:17 GMT <xep> (and by IPs, I really mean VIPs)
01:27:54 GMT <Freman> we do something similar sort of
01:27:59 GMT <Freman> we use dns, srv records :D
01:28:16 GMT <Freman> the power of dynamically configuring everything, without the cost of doing lookups all the time
01:28:24 GMT <Freman> (even cached lookups cost)
01:29:08 GMT <Freman> ahfeel: we also have sometimes huge hangs, but we don't have them in the cli, just phpredis lib :(
01:59:04 GMT <xep> Freman: are you having PHP talk to Redis directly?
01:59:30 GMT <xep> we use twemproxy to add connection pooling
01:59:51 GMT <Freman> yeh, 6000 php-fpm processes talk to 8 redisd's
01:59:57 GMT <xep> there's certain TCP tunings that are good to have if you're going to do that
02:00:01 GMT <xep> do you have tw_recycle enabled ?
02:00:56 GMT <Freman> net.ipv4.tcp_tw_recycle = 1
02:01:07 GMT <Freman> we were looking at rmux
02:01:26 GMT <Freman> problem with either method is the way we've historically done caching (it's bad, trust me)
02:02:19 GMT <xep> hm haven't tried rmux
02:02:40 GMT <Freman> same basic thing
02:02:43 GMT <xep> we're using a fork of twemproxy with a patch to give it Redis Sentinel support
02:02:49 GMT <xep> yeah, was just looking at the GitHub page ;)
02:03:44 GMT <Freman> http://imgur.com/ajSiCLU the thing that concerns me is there doesn't appear to be enough connections to the redises... so either php-fpm/phpredis is doing it's own connection pooling, or we're borked somewhere
02:04:33 GMT <Freman> ftr, I don't like phpredis, but I can't convince them to move to something that actually provides error messages with context
02:23:56 GMT <xep> Freman: what version of PHPRedis ?
02:28:47 GMT <xep> and what version of PHP?
03:08:57 GMT <Freman> latest phpredis, and a few patchsets behind 5.6.something
03:32:25 GMT <Freman> "Error: Unhandled exception encountered: Redis server went away" php-redis's answer to everything
04:21:50 GMT <p1k> is it possible to connect with redis-cli via. url ? I don't see a flag for it
04:24:53 GMT <Freman> you can use redis-cli -h <host> -p <port>
04:25:29 GMT <xep> damn that was a crappy cable outage.
04:38:53 GMT <p1k> Freman: I'd like to use the url though e.g. redis://<password>@host:port
04:40:44 GMT <Freman> from the cli, I don't belive that's possible -h host -p port -a password
04:52:14 GMT <xep> p1k: submit a pull request? :)
04:56:09 GMT <p1k> xep: can
06:31:55 GMT <minus> not sure i'd accept such a PR
06:41:16 GMT <antirez_> Dvir: about the modules API you provided, it is similar to what the actual callbacks are in the code I already wrote:
06:41:33 GMT <antirez_>     void *(*rdbLoad)(rio *rdb);
06:41:33 GMT <antirez_>     int (*rdbSave)(rio *rdb, void *value);
06:41:33 GMT <antirez_>     int (*rewrite)(rio *aof, robj *key, void *value);
06:41:35 GMT <antirez_>     void (*free)(void *value);
06:42:20 GMT <antirez_> The serializer has to write N elements, and the deserializer gets the N elements.
06:43:35 GMT <antirez_> However I was thinking about putting the number of elements the deserializer gets directly into the arguments we pass to it to don't have a "get-len" style API, and force all the serialization/deserialization formats to have the concept of storing N elements. Of course N=1 is possible, there is just some overhead in the RDB (I think just 1 byte per element if I remember correctly).
06:43:57 GMT <antirez_> dvirsky__: I saw that you also thought at integer serialization/deserialization API
06:44:13 GMT <antirez_> dvirsky__: I was thinking about this as well however the RDB internal functions kinda are able to do this automatically
06:45:16 GMT <antirez_> dvirsky__: they can detect if a string looks like an integer end encode as needed, so it's not worth to show this to the user probably. However to have the API just to write "long long" values could be handy for the user for the (rare) data structures that are composed mostly of integers.
06:45:58 GMT <antirez_> dvirsky__: we can start like that and if there is friction serializing certain things we can improve the API a bit maybe.
06:46:00 GMT <dvirsky__> antirez: re passing the number, that's also in my proposal
06:46:08 GMT <antirez_> dvirsky__: yes, is a good idea probably
06:46:10 GMT <dvirsky__> re ints - it's a nice addition, not a must
06:46:31 GMT <dvirsky__> it can be added in librmutil or something
06:47:00 GMT <dvirsky__> for complex stuff people will probably want to use protobuf or thrift anyway
06:51:12 GMT <dvirsky__> antirez: are you going to expose rio directly or wrap it in some opaque like other parts of the API?
10:01:13 GMT <ahfeel> xep / Freman : Thank you very much for your inputs ! I will definitely check that. We're using DNS to connect to our redis, but the redis-cli local doesnt of course.
10:01:56 GMT <ahfeel> xep / Freman : I turned on pconnect instead of connect yesterday, I don't think that was the issue but I prefered to turn it off again :-/
10:11:20 GMT <antirez_> dvirsky__: rio will be exposed as an opaque type as usually, but people will never write/read low level to it, only pass it around to other modules functions that do the work
10:11:57 GMT <antirez_> So in Redis 3.2 we were not enjoying some memory savings because of a configure option missing in jemalloc. Fixed in unstable and will be part of 3.2.1 of course.
11:30:54 GMT <dvirsky__> antirez: cool re the memory savings. got an estimate on the amount?
11:34:37 GMT <oranagra> hi all, is this a unified channel for users and developers, or so we have a separate channel (like the redis-dev mailing list)
11:37:19 GMT <Inge-> this is more for users
11:37:38 GMT <Inge-> at least, most talk is about using Redis, not about developing it
11:37:46 GMT <Inge-> even though antirez is also here :)
11:38:21 GMT <dvirsky__> oranagra: unified channel for devs and users.
11:39:02 GMT <oranagra> antirez: i suggest creating a second channel so that we don't cause mix these topics.
11:39:47 GMT <dvirsky__> oranagra, suggested it too, salvatore wants to try to have the dev discussions here for now, and see if it works
11:41:04 GMT <dvirsky__> I'm cool with trying it
11:41:24 GMT <dvirsky__> we just need a bouncer server to read offline messages automatically without the bot
11:43:21 GMT <Inge-> not that much traffic in this channel really
11:44:16 GMT <dvirsky__> I haven't been here 4 years until now :)
12:23:26 GMT <yiftach> hi all!
12:23:34 GMT <dvirsky__> yiftach: hi
13:40:35 GMT <antirez_> Hey
13:40:49 GMT <antirez_> oranagra: I just merged your PR fixing BITFIELD for the integer encoded case
13:41:00 GMT <antirez_> oranagra: Refactored the code used 3 times to do a similar thing here: https://github.com/antirez/redis/commit/bee963c4459223d874e3294a0d8638a588d33c8e
13:41:23 GMT <antirez_> dvirsky__: please may you link me to your pending PR?
13:42:17 GMT <antirez_> oranagra: about the -dev channel, makes sense in the long run if here populates a bit. We'll be mostly the only ones to talk, but a few help requests from time to time.
13:42:52 GMT <antirez_> given the number of users is currently like Cathedral In The Desert ;-)
13:43:04 GMT <Inge-> :)
13:43:53 GMT <antirez_> dvirsky__: btw nothing stops me to add the bot the capability to say "!history" in priv message to retrieve the history in terms of it sending it to you
13:44:40 GMT <dvirsky__> slaveofn1: !search pull
13:44:49 GMT <dvirsky__> probably doing it wrong
13:44:51 GMT <antirez_> dvirsky__: haha that's a good one! :-)
13:45:03 GMT <antirez_> I'll write something during the WE
13:45:12 GMT <dvirsky__> https://github.com/antirez/redis/pull/3239
13:45:16 GMT <dvirsky__> antirez ^^
13:45:17 GMT <antirez_> dvirsky__: thx
13:52:46 GMT <dvirsky__> antirez_, is the PR comment  something I need to fix or just a comment on that user?
13:53:00 GMT <dvirsky__> because as far as I can see I comply with this style
13:55:48 GMT <dvirsky__> anyway, I have to go home. laterz
14:05:55 GMT <antirez_> dvirsky_: commented in the PR re style, it's fine AFAIK
14:06:19 GMT <antirez_> I use a mix of single lines if, if + statement in different lines without {}, and proper if + {} blocks.
14:06:33 GMT <antirez_> The rationale is that vertical space is important to optimize to read the code later.
14:30:50 GMT <antirez_> dvirsky_: PR merged + an additional commit to clarify the ownership thing.
14:53:36 GMT <oranagra> antirez, i see struct padding issue in dictEntry when arch is 32bit.. do we care?
15:50:05 GMT <antirez_> oranagra: yes... definitely, if it makes 64bit the same but 32bit better, why not?
15:50:42 GMT <antirez_> oranagra: some time ago I noticed but probably I thought since the size classes of jemalloc did not allowed certain values, it was the same, or something along those lines.
15:53:28 GMT <oranagra> it will be 32+64+32 and 32bit padding and will be taken from the 24byte bin, if we reorder it will be taken from the 16byte bin.
15:55:25 GMT <oranagra> shall i PR or are you changing yourself?
15:58:39 GMT <itamarhaber> Looks like all the cool kids are here
16:01:22 GMT <antirez_> oranagra: would appreciate PR in order to give proper credit
16:01:59 GMT <antirez_> itamarhaber: haha we are more of the "old coders" kind, cool kids was the next generation
16:02:19 GMT <antirez_> as you can see the topic is structures padding and not Javascript frameworks
16:02:49 GMT <oranagra> this is a one liner.. i don't care about the credit.. trying to avoid the work overhead of clicking so many mouse clicks in order to change one line.
16:02:59 GMT <antirez_> oranagra: ok changing it :-) Thanks
16:03:11 GMT <oranagra> i hate mice..
16:03:13 GMT <oranagra> and python
16:03:22 GMT <oranagra> but love pythons
16:03:34 GMT <antirez_> It's purely a matter of animals right? Not technologies
16:03:37 GMT <oranagra> and my keyboard (IBM Model-M)
16:03:42 GMT <yiftach> and I love slack
16:03:58 GMT <antirez_> oranagra: is the Model-M the one that clicks at each keystroke?
16:04:14 GMT <antirez_> yiftach: as in SlackWare?
16:04:20 GMT <antirez_> ;-)
16:04:41 GMT <yiftach> Ahaah
16:04:53 GMT <oranagra> yeah, the only true mechanical keyboard out there (i tried all).. they still make them: http://www.pckeyboard.com/
16:06:14 GMT <itamarhaber> oranagra: pity they don't make laptops with these
16:06:15 GMT <antirez_> I used to have one of those, rebranded HP or something like that, used it for so many years, then switched to more soft touching ones in order to make less noises back then when I had no proper office and worked in the kitchen
16:06:48 GMT <itamarhaber> Remember the old DOS utility that made soft keyboard click through the speaker?
16:08:52 GMT <antirez_> oranagra: I've the feeling that has it is it is already naturally ordered and takes 16 bytes
16:09:18 GMT <antirez_> itamarhaber: haha no, wow that is terrible
16:12:40 GMT <antirez_> oran: since now is 4+8+4 and padding in 32 bit system will be to be multiple of 4 bytes, there should not be padding currently, since AFAIK the structure will not make effort in order to pad the 8 bytes field in order to align it, since anyway, it will be accessed via 4 bytes operations.
16:12:53 GMT <antirez_> oranagra: I'm not 100% sure btw, we can check
16:14:31 GMT <yoav_s> oran on yoav's keyboard (before leaving home), the int64 must be memory aligned, so you'll get 4+4+8+4
16:15:04 GMT <antirez_> yoav_s: I've the feeling int64 is not memory aligned in 32 bit systems since the op is emulated with 32 bit math internally
16:15:10 GMT <antirez_> yoav_s: however I'm documenting myself
16:15:58 GMT <yoav_s> According to Oran that the compiler must align each type based on it's size according to some spec. I'm not sure which, worth googling for it..
16:16:14 GMT <antirez_> totally, going to google, thanks
16:16:37 GMT <antirez_> yoav_s: I saw your email about no reply, there is totally a part which is "unspecified" according to the implementation, thanks
16:16:45 GMT <antirez_> yoav_s: I plan to fix it tomorrow
16:17:11 GMT <antirez_> yoav_s: there are a few design questions to ask ourselves indeed, I'll try to find a solution that makes sense, propose it to you to get feedbacks, and see what happens. Makes sense?
16:19:37 GMT <antirez_> yoav_s: oranagra: https://gist.github.com/antirez/2b2c7d62606de5f2c3a35a5fde4811fb
16:20:46 GMT <yoav_s> Yes. From an internal discussion we had we're gearing towards the following approach: no support at all for "client reply X" inside multi-exec sequence. But we still don't have a good idea how to act when skip is specified before the multi -> should we skip the reply after exec or maybe something else?
16:21:30 GMT <antirez_> yoav_s: yep with another option being to ignore it under certain conditions
16:22:09 GMT <antirez_> re alignment, from my gist it looks like clang follows the behavior I expected. Not sure if the reason is the one I provided btw.
16:22:18 GMT <antirez_> Let's Google
16:25:52 GMT <yoav_s> Looks like oran was wrong, ran your code built with gcc, got the same result (0,4,12).
16:30:59 GMT <antirez_> yoav_s: oranagra: there are more hints here, it's a complex issue apparently: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=52023
16:31:19 GMT <antirez_> The very short version is that alignment requirements outside/inside structures are different
16:31:26 GMT <antirez_> (for the very same type)
17:20:45 GMT <itamarhaber> Seems like IRCing is harder nowadays... Admittedly, last time I was here was in the previous milennium
17:21:06 GMT <itamarhaber> join #SiTH
18:47:48 GMT <gorroth> hi
18:49:53 GMT <gorroth> i have a question re: redis in blackbox testing envs.  we use redis with one of our services, and we're trying to write some robot framework tests (python) that will hit one server that writes to redis, and then another service picks up that message and uses it.  the only problem is that once the other service pops the message, it's gone from redis.  in this special environment, is there a way to configure the redis server to hold onto messages for a while
18:50:00 GMT <gorroth> i know this is not a normal production use case
18:57:38 GMT <minus> not that i know. but if your tests need to intrude into redis that doesn't sound quite right
18:58:51 GMT <iamchrisf> Hey, is there a way to restore from an RDB file without taking the server offline?
19:00:22 GMT <minus> doesn't look like it. there wouldn't much of a point to it anyway
19:01:04 GMT <minus> if you want to restore it with minimal downtime i suggest you disable saving during runtime (config set save ""), put the file in place and then restart redis.
19:01:18 GMT <iamchrisf> It’s a 40GB data set...
19:01:22 GMT <minus> while it's loading it cannot be used, but that's the case anyway
19:01:24 GMT <iamchrisf> takes 5-6 mins to load into memory
19:01:29 GMT <iamchrisf> not an option
19:01:48 GMT <iamchrisf> I get a 40GB RDB nightly and need to hotswap in place .
19:02:14 GMT <gorroth> it's right.  we don't want to go into a month's long project just building a test case for this right now, working with the other team to get their service more testable
19:02:18 GMT <minus> loading in data inevitably blocks all requests
19:02:20 GMT <gorroth> it's nice in theory but not practical for the moment
19:02:34 GMT <iamchrisf> well it should only block the key that it’s loading at the time.
19:03:14 GMT <minus> gorroth: fair. maybe watching the operations that happen using monitor command could help?
19:03:54 GMT <minus> iamchrisf: loading a dump would remove everything that's in redis already anyway
19:04:52 GMT <minus> iamchrisf: you could, theoretically, launch a second instance with that data and copy stuff over "by hand" using a script
19:05:11 GMT <iamchrisf> minus: the second instance solution is what I’m looking at right now.
19:05:31 GMT <iamchrisf> But that means placing haproxy on all my APis that use redis and figuring out a way to verify data and flip the system live
19:05:54 GMT <iamchrisf> I was hoping 3.X had a better solution.  Using 2.8 right now.
19:08:15 GMT <minus> not sure how haproxy helps with restoring the data
19:10:06 GMT <iamchrisf> Switching active connections from one redis server to the other
19:14:34 GMT <minus> is it a static data set?
19:28:44 GMT <iamchrisf> it changes daily.
19:28:51 GMT <iamchrisf> so we dump it once a night and restore
19:29:55 GMT <minus> you have to do what you want to do now every day?
19:30:18 GMT <minus> why not write updates directly to that redis? well, i guess you have your reasons.
19:30:58 GMT <minus> using haproxy and swapping between 2 instances seems fine. you should probably be prepared for short outages when switching though
19:34:53 GMT <xep> iamchrisf: can you spin up up another instance, restore on there, and fail over to it?
19:35:02 GMT <xep> also could use twemproxy with Redis Sentinel support
19:35:17 GMT <xep> https://github.com/ifwe/twemproxy
19:39:06 GMT <minus> sentinel wouldn't really help here
19:42:32 GMT <iamchrisf> xep: so can I take a sentinel node offline restore to it then promote it as master?
19:42:58 GMT <iamchrisf> meanwhile twmproxy is handling connections
19:48:18 GMT <iamchrisf> Or do I even need twemproxy? Seems the scala client https://github.com/etaty/rediscala would automatically switch connections to a new master.
20:12:25 GMT <redis006> I am very curious to know the reason why Redis does not use CRDT for replication
20:13:46 GMT <redis006> I have read the: "Why merge operations are avoided" subsection in http://redis.io/topics/cluster-spec but I think using CRDT could be very beneficial
21:27:19 GMT <antirez_> iamchrisf: for your use case you should just use Linux firewalling IMHO to redirect the connections to the new instance (running like in a different port or IP) when you are ready loading
21:27:30 GMT <antirez_> iamchrisf: so you have two instances and you switch between them
21:27:48 GMT <antirez_> @iamchrisf if you want to avoid restarting the server, transfer the dump and use "DEBUG RELOAD"
21:27:57 GMT <antirez_> but is blocking
21:28:04 GMT <antirez_> so you need to switch between two instances anyway
21:31:16 GMT <oranagra> antirez: AFAIR DEBUG RELOAD also saves... i faced that same problem a few time, maybe we need to add DEBUG LOAD
21:33:09 GMT <oranagra> or maybe not as a debug command but a new global administrative command
21:34:35 GMT <antirez_> oranagra: OOPS you are right
21:35:15 GMT <oranagra> what do you think about that new command?
21:35:19 GMT <antirez_> DEBUG LOAD could be it. The reason this stuff is currently in "DEBUG" is that it's not always safe to reload under certain special conditions, perhaps. To guarantee something solid could be an issue.
21:35:37 GMT <antirez_> I mean, woudl be cool, there is to check if it is feasible to give this control
21:36:14 GMT <oranagra> so at least a debug command... 2 minutes to implement.
21:36:28 GMT <antirez_> Indeed, +1 per that, it's useful...
21:37:16 GMT <oranagra> do you have spare 2 minutes? or shall i put it in my todo list for PRs?
21:37:31 GMT <oranagra> actually if you have some spare minutes, use them to review my memory commands PR
21:37:53 GMT <itamarhaber> lol - subtle as ever oranagra
21:38:09 GMT <antirez_> oranagra: I've spare 2 min :-) But indeed better to review your PR. Now going to bed since need to getup at 7, but opening a tab with your PR right now so I'll process it tomorrow. Thanks!
21:38:45 GMT <oranagra> thanks. good night.
21:40:40 GMT <antirez_> Good night folks, see you tomorrow.
22:02:32 GMT <iamchrisf> oranagra: Thanks I’ll check out DEBUG RELOADand see how I fair.