00:13:49 GMT <varx> nvm, it's `CONFIG get maxclients`
07:27:18 GMT <dvirsky__> antirez, I've implemented numeric filters for the search modules, but they are very slow even though I'm using the low level API
07:27:45 GMT <dvirsky__> I'm thinking of implementing a specialized hash table over DMA (or later over data types) to speed it up
07:28:19 GMT <dvirsky__> one of the things I've noticed is that sdslen is taking up a lot of execution time, and I can see a couple of places to optimize it
07:29:43 GMT <antirez> dvirsky__: if you can *meter* the speedup and is > %5, or < 5% but the fix does not add complexity, well I can merge for sure.
07:30:00 GMT <antirez> dvirsky__: but I've the feeling is mostly cache misses time :-(
07:30:03 GMT <dvirsky__> it's probably like 10%
07:30:09 GMT <dvirsky__> I did 2 things:
07:30:39 GMT <dvirsky__> 1. the macros there use sizeof in runtime. this takes precious CPU cycles, so I "cached" the sizes of the header in static consts
07:31:00 GMT <antirez> sizeof() is resolved at compile time
07:31:20 GMT <dvirsky__> it did speed up things, so not sure why then
07:31:46 GMT <dvirsky__> another thing was to change the order of the switch cases
07:31:50 GMT <antirez> it can just happen that there are errors in the measurements or that the compiler generated slightly different code that in your CPU kinda happen to be faster for some strange reason
07:32:26 GMT <antirez> the order of the switches can make a difference, but risks to be workload-dependent. If we think that globally, we have more strings of a given range, that could be a valid approach.
07:33:17 GMT <dvirsky__> it helped more than the sizeof, but I think we can do something better by having an array of callbacks by header width, not sure what will be faster, but I wanted to explore that
07:36:21 GMT <antirez> dvirsky__: sometimes switch itself is compiled like a jump table IIRC
07:36:39 GMT <dvirsky__> I know that too, but even in O3 it made a difference
07:36:52 GMT <dvirsky__> maybe there's something more I'm missing. but on a very tight loop my module was spending like 20-30% of execution time in sdslen
07:37:09 GMT <dvirsky__> both when I'm doing a lot of dict lookups, and when I'm iterating a zset range
07:37:51 GMT <dvirsky__> I didn't benchmark the improvements in sdslen directly, just the improvements in my module's performance
07:37:53 GMT <antirez> Yes, but what happens usually is that you touch the string the first time with sdslen(), so you pay the cache miss time
07:38:11 GMT <antirez> so some times you see as accountable a function whose only fault is to access a piece of memory the first
07:38:28 GMT <dvirsky__> I was able to increase the RPS on a heavey numeric range from 210 to 240rps just by tweaking sdslen
07:39:17 GMT <dvirsky__> it's still too slow but it hints that there is something to be done there. I'll investigate some more
07:39:38 GMT <antirez> ok, thanks, it's a very often used function, worth a few testings.
07:40:33 GMT <dvirsky__> anyway, the whole numeric performance can take me a few days now, not sure if I should invest in it
07:45:20 GMT <tgy> antirez: any chance to have keyspace notifications send the value with the key instead of just sending the key?
07:45:47 GMT <tgy> if not, what should i use instead if i want to be notified of the new value when a key is modified or created?
07:45:53 GMT <dvirsky__> tgy: what if the value is a 100MB string or a 1M elements set?
07:45:56 GMT <antirez> tgy: no plans currently, why don't you PUBLISH it instead of relying on that?
07:46:19 GMT <antirez> in such an use case MULTI/MY_COMMAND/PUBLISH/EXEC is a good pattern
07:46:19 GMT <tgy> dvirsky__: i know, in my use case my keys are very tiny
07:46:37 GMT <dvirsky__> antirez: it's now possible to subscribe as a slave without sync, right? that would be ideal for that
07:46:44 GMT <tgy> antirez: right, sounds like what i should do :)
07:47:00 GMT <antirez> dvirsky__: no... experimental branch not merged
07:47:10 GMT <dvirsky__> should be in 4.0 too IMO :)
07:47:15 GMT <tgy> dvirsky__: can you explain a little bit what you just said? :)
07:47:17 GMT <antirez> I wrote it as a POC and never incorporated into the main code
07:47:23 GMT <tgy> i didn't get the 'sync' and 'slave' part
07:47:33 GMT <dvirsky__> basically your client can emulate a slave
07:47:41 GMT <antirez> https://github.com/antirez/redis/tree/nosync
07:47:46 GMT <dvirsky__> connect to a master and have all changes replicated to it
07:48:06 GMT <dvirsky__> the problem is that it will try to "sync", as in get all the current data first, when it connects
07:48:25 GMT <tgy> ah right, and again in my case that's not a problem
07:48:27 GMT <dvirsky__> someone implemented a feature where a slave can just say "start replicating now, I don't care about the past"
07:48:42 GMT <tgy> in my case i need the past
07:49:02 GMT <dvirsky__> oh... keyspace notifications won't help you with that either
07:49:10 GMT <antirez> yep the PUBLISH stuff is probably just much simpler and to the point in that case
07:49:24 GMT <dvirsky__> BTW this is an ideal use case for modules once we have proper hooks
07:49:39 GMT <tgy> i was planning on retrieving the current subset of the db i need first (with a big call) and then handle the updates
07:50:13 GMT <tgy> dvirsky__: ^ does that make sense?
07:50:46 GMT <dvirsky__> it's the only way, unless you implement your own mechanism as antirez suggested
07:51:34 GMT <tgy> thank you guys :)
10:14:24 GMT <dvirsky__> antirez: any chance to get something akin to ZCOUNT in the low level modules API?
10:15:20 GMT <dvirsky__> right now there is no efficient way to estimate the size of a range
10:24:49 GMT <antirez2> Sure it's planned
10:25:54 GMT <dvirsky__> cool I guess. I can use it to optimize numeric ranges and choose the strategy to use
10:46:54 GMT <rikemSen> Hi there! :)
10:48:17 GMT <rikemSen> I'd like to know, i'm currently working on a crawler and I'm wondering how could I design my redis informations. I have many url that had been crawled and will be again. So, for one URL I can have many timestamps
10:48:22 GMT <rikemSen> how to do that ?
10:49:18 GMT <rikemSen> Please :)
10:55:52 GMT <rikemSen> anyone ?
11:09:18 GMT <minus> i'm sure you can find a resouce to start out from on the internet
11:36:50 GMT <antirez2> I think you could find useful to read the pattern documented in RPOPLPUSH
11:43:47 GMT <minus> he's already gone
12:30:50 GMT <antirez2> People run away in no time after asking stuff
13:24:28 GMT <minus> irc is hard
13:52:04 GMT <veleno> hello. for research purposes, we’re gathering some measures of redis in cluster mode, in particular when resharding. we start with a cluster setup of 10 master nodes (and zero slaves), with 2688 keys. then we add 10 more masters (so, 20 nodes in total), and we start the resharding. This takes 14m30s. It seems quite slow. Then, we double again the size of the cluster (up to 40 masters), and the resharding now takes 58min. And once more, up to 80, it took
13:52:04 GMT <veleno> 4h19m to complete. This seems more than quadratic.. Are these performances to be expecte ?
13:59:52 GMT <veleno> nodes are connected via gigabit ethernet, keys are generally small and never bigger than 13Kb. in the 10-nodes scenarios, keys are fairly well balanced, at around 268 keys x node.
15:00:02 GMT <ws2k3> is there any way to speed up redis?
15:00:23 GMT <ws2k3> when doing 40 k ops/sec it already consumes 100 % cpu
15:10:48 GMT <colo-work> ws2k3, get faster CPUs
15:10:59 GMT <colo-work> use pipelining where possible
15:23:53 GMT <ws2k3> colo-work we use php-fpm so pipelining is not possible unfutunaly
15:25:04 GMT <ws2k3> colo-work and i run redis on a E5410 should be fast enough
15:26:13 GMT <ahfeel> ws2k3: why pipelining would not be possible ? If your redis server is on another machine, you have to try to group as much a possible your requests to minimize the roundtrips
15:26:56 GMT <ahfeel> I've been from 40ms spent in redis per http request (~40 redis calls) to 6-8ms by grouping keys in mget everywhere possible
17:13:33 GMT <goldstar1> do all sentinel port numbers have to be the same?
17:13:45 GMT <goldstar1> 26379
17:49:40 GMT <svm_invictvs> WIth Redis, how woudl I do a sum or an average over a bunch of keys?  Would that be best for a lua script?
18:15:13 GMT <alexi5> hello
18:35:22 GMT <itamarhaber> svm_invictvs: if you know the keys' names, a Lua script is a good choice indeed
18:35:36 GMT <svm_invictvs> It's load test data
18:36:10 GMT <svm_invictvs> Basically I have a key like load-test-1:<simulated-client-uuid>:<request-uuid>, so I could scan those keys right?
18:36:32 GMT <svm_invictvs> After my load test is done, just scan for the keys all prefixed with the load test I"m looking for and just dump that to a new hash, csv or whatever.
18:37:43 GMT <ws2k3> ahfeel how would i do that cause i use php-fpm
18:37:59 GMT <ws2k3> ahfeel so every php-fpm proces has to start a new connection to redis for the op
18:39:06 GMT <itamarhaber> You can do that, sure... but why? ;) What I mean is, what is the question that you're trying to answer and what are the requirements - scanning is always possible but takes time.
18:42:32 GMT <minus> ws2k3: if you have a redis connection pool as large as your php instance pool it should be just fine
18:45:00 GMT <ws2k3> minus hmm i think i dont understand
18:45:40 GMT <ws2k3> minus i have around 60 servers i plan to insert stuff into redis the problem im having now is that i can get only 40 k ops/sec on a single redis instandse and then my cpu is already full
18:46:03 GMT <minus> i see
18:46:07 GMT <ws2k3> minus and i just dont understand why redis cunsumes so much cpu
18:51:09 GMT <minus> ws2k3: what operations? did you compare with redis-benchmark?
18:51:54 GMT <ws2k3> minus i only use lpush and rpop nothing more. yes i runned a benchmark with redis-benchmarkt and that showed me 35/40 k ops/sec
18:52:33 GMT <minus> same high CPU usage?
18:53:03 GMT <ws2k3> minus the benchmarkt could not get any higher cause the cpu usage was 100 % thats why the benchmarkt could not get any higher
18:55:15 GMT <ws2k3> minus thats why i dont understand it. it should be able to perform alot better. i dont do expensive operation. and still it reaches 100 % cpu very early
18:55:54 GMT <minus> i can push about 100k ops/s at 90% CPU usage
18:56:00 GMT <minus> on my laptop
18:56:29 GMT <minus> where are you running it on? some VM?
18:56:36 GMT <ws2k3> minus. exacly. it seems im doing something wrong
18:56:50 GMT <ws2k3> minus yes im running it in a xen vm
18:56:58 GMT <minus> that sucks
18:57:25 GMT <minus> did you check the transparent huge pages settting already?
18:57:35 GMT <ws2k3> minus why? i run alot of stuff in xen vm's also heavy stuff and i never had any performence issues with it
18:57:44 GMT <ws2k3> minus yes i already did that setting
18:58:09 GMT <minus> can/did you run a benchmark outside of the VM for comparison?
18:58:41 GMT <ws2k3> minus onfurtunaly not i have this hardware only in xen but i can run a benchmarkt on another server
18:59:06 GMT <ws2k3> minus mu xen vm has 4 cores with 11 gb ram where i tested it
18:59:07 GMT <goldstar1> how do redis-sentinel instances acquire knowledge of other slaves ? each instance is aware of itself and the master only when it is initially configured and started
18:59:59 GMT <ws2k3> minus this is how i ran my benchmark ./redis-benchmark -h 1.1.1.1 -r 1000000 -n 2000000 -t lpush,lpop -q
19:00:09 GMT <minus> ws2k3: unfortunately i don't know anything about xen and i don't have a xen machine for testing either
19:00:30 GMT <minus> you ran it from a different host?
19:00:57 GMT <ws2k3> minus yes but when i run it localy i get the same issue
19:01:14 GMT <minus> mhm
19:01:33 GMT <minus> just ran the benchmark on my desktop and i get 260k/s :o
19:03:06 GMT <ws2k3> minus it gets stuck on 40 k
19:03:57 GMT <ws2k3> minus around 25 % us and 50 % sys load
19:17:58 GMT <truthadjustr> hi, why can't redis-sentinel NOT work with cluster-mode redis nodes?
19:18:33 GMT <truthadjustr> i thought, sentinel is intended to be combined with cluster. No?
19:18:49 GMT <ws2k3> truthadjustr redis cluster has sentinal kinda build in already so no
19:19:05 GMT <ws2k3> truthadjustr sentinal is only for non cluster redis
19:20:47 GMT <truthadjustr> ok got it. thanx
19:28:30 GMT <truthadjustr> with 3.2.0, is there no more need for twemproxy?
19:37:09 GMT <svm_invictvs> Is it bad to use KEYS in production to group data?
19:37:45 GMT <svm_invictvs> well, nevermind, that's kind of a dumb question
19:38:07 GMT <svm_invictvs> I just realized, why.  If know ahead of time a way to get all the keys then I don't need to scan for them.
19:57:29 GMT <truthadjustr> even if key is found in a clustered-node, why can't I read it if the cluster is down (some other node shuts down) ?
19:58:10 GMT <truthadjustr> seems to be valid to return the data in this live node that I am specifically doing a get
20:03:31 GMT <truthadjustr> ok.. got the correct config settings for this,
20:30:37 GMT <antirez3> There is a config option for that
20:30:45 GMT <antirez3> Ah indeed
20:37:36 GMT <dochttpd> Hi, I need to copy Redis from a remote linux host to my windows localhost. i got the server dump.rdb on my computer. how do i upload it to my redis local?
20:40:11 GMT <minus> put it in the right location (as indicated by your config) and start redis
20:41:09 GMT <minus> another method is to make your local redis a slave of the server until it's finished copying the data and then slaveof no one it
22:34:13 GMT <svm_invictvs> IN the lua scripts how would I throw an error?
22:34:23 GMT <svm_invictvs> Tell redis that the script encountered an error, (eg bad input parameters?)
22:34:27 GMT <svm_invictvs> Just print and return?
22:39:40 GMT <itamarhaber> svm_invictvs: you can use the helper error_reply like so `return redis.error_reply('bad input parameters')`
22:39:48 GMT <svm_invictvs> ah
22:39:50 GMT <svm_invictvs> Okay
22:40:04 GMT <svm_invictvs> Does it handle print?
22:40:09 GMT <itamarhaber> http://redis.io/commands/eval#helper-functions-to-return-redis-types
22:40:15 GMT <svm_invictvs> Like, let's say I want to print out the results of my script
22:40:39 GMT <itamarhaber> Print goes to stdout, that's where your Redis server prints its stuff
22:40:49 GMT <svm_invictvs> oh
22:40:55 GMT <svm_invictvs> That's not very helpful
22:41:02 GMT <svm_invictvs> I'm trying to get a CSV
22:41:19 GMT <svm_invictvs> I guss that also means some log at AWS now says, "Wrecked him?  Damn near killed him!"
22:41:27 GMT <svm_invictvs> (that asshole)
22:46:59 GMT <svm_invictvs> itamarhaber Thanks.
22:49:20 GMT <heyrhett> how many connections per second can a typical redis server handle?
22:49:56 GMT <heyrhett> I'm thinking of using it in a serverless architecture, so each lambda function creates a new connection
22:53:09 GMT <svm_invictvs> Why can't I name a script in lua?
22:53:16 GMT <svm_invictvs> SCRIPT LOAD myscript myscript.lua
22:56:08 GMT <minus> heyrhett: try. i guess the kernel will be the limiting factor
22:57:34 GMT <heyrhett> ok, so you think in the thousands?
22:59:13 GMT <minus> probably