Add redis support for distributed caching (#83)

* Add redis support for distributed caching

* Move towards the self-provided Redis connection pool and RESP protocol implementation.
Official redis client library won't work with yaegi.

* fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.

* fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.

* fixup! fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.

* fixup! fixup! fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.

* fixup! fixup! fixup! fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.

* ... and another all nighter.

* fixup! ... and another all nighter.

* fixup! fixup! ... and another all nighter.

* fixup! fixup! fixup! ... and another all nighter.

* Resolve issue #85 by adding ability to set custom claims in JWT tokens

* Remove redundant validation in auth middleware ( issue #89 )

* Add ability to set cookie prefix for session cookies ( #87 )

* fixup! Add ability to set cookie prefix for session cookies ( #87 )

* Add ability to set cookie max age - issue #91

* Potential fix for code scanning alert no. 10: Size computation for allocation may overflow

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fixup! Merge main into 0.8.0-redis: resolve conflicts

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
This commit is contained in:
2025-11-30 02:18:46 +00:00
committed by GitHub
co-authored by Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
parent 5fcbd54955
commit e64fc7f730
318 changed files with 100989 additions and 948 deletions
+6
View File
@@ -0,0 +1,6 @@
/integration/redis_src/
/integration/dump.rdb
*.swp
/integration/nodes.conf
.idea/
miniredis.iml
+328
View File
@@ -0,0 +1,328 @@
## Changelog
## v2.35.0
- add Lua redis.setresp({2,3})
- embed gopher-json package
- fix XAUTOCLAIM (thanks @kgunning)
- fix writeXpending (thanks @gnpaone)
- fix BLMOVE TTL special case
- constants for key types @alyssaruth
### v2.34.0
- fix ZINTERSTORE where target is one of the source sets
- added support for ZRank and ZRevRank with score (thanks Jeff Howell)
- fix MEMORY subcommand casing (thanks @joshaber)
- use streamCmp in Xtrim (thanks @daniel-cohere)
### v2.33.0
- minimum Go version is now 1.17
- fix integer overflow (thanks @wszaranski)
- test against the last BSD redis (7.2.4)
- ignore 'redis.set_repl()' call (thanks @TingluoHuang)
- various build fixes (thanks @wszaranski)
- add StartAddrTLS function (thanks @agriffaut)
- support for the NOMKSTREAM option for XADD (thanks @Jahaja)
- return empty array for SRANDMEMBER on nonexistent key (thanks @WKBae)
### v2.32.1
- support for SINTERCARD (thanks @s-barr-fetch)
- support for EXPIRETIME and PEXPIRETIME (thanks @wszaranski)
- fix GEO* units to be case insensitive
### v2.31.1
- support COUNT in SCAN and ZSCAN (thanks @BarakSilverfort)
- support for OBJECT IDLETIME (thanks @nerd2)
- support for HRANDFIELD (thanks @sejin-P)
### v2.31.0
- support for MEMORY USAGE (thanks @davidroman0O)
- test against Redis 7.2.0
- support for CLIENT SETNAME/GETNAME (thanks @mr-karan)
- fix very small numbers (thanks @zsh1995)
- use the same float-to-string logic real Redis uses
### v2.30.5
- support SMISMEMBER (thanks @sandyharvie)
### v2.30.4
- fix ZADD LT/LG (thanks @sejin-P)
- fix COPY (thanks @jerargus)
- quicker SPOP
### v2.30.3
- fix lua error_reply (thanks @pkierski)
- fix use of blocking functions in lua
- support for ZMSCORE (thanks @lsgndln)
- lua cache (thanks @tonyhb)
### v2.30.2
- support MINID in XADD (thanks @nathan-cormier)
- support BLMOVE (thanks @sevein)
- fix COMMAND (thanks @pje)
- fix 'XREAD ... $' on a non-existing stream
### v2.30.1
- support SET NX GET special case
### v2.30.0
- implement redis 7.0.x (from 6.X). Main changes:
- test against 7.0.7
- update error messages
- support nx|xx|gt|lt options in [P]EXPIRE[AT]
- update how deleted items are processed in pending queues in streams
### v2.23.1
- resolve $ to latest ID in XREAD (thanks @josh-hook)
- handle disconnect in blocking functions (thanks @jgirtakovskis)
- fix type conversion bug in redisToLua (thanks Sandy Harvie)
- BRPOP{LPUSH} timeout can be float since 6.0
### v2.23.0
- basic INFO support (thanks @kirill-a-belov)
- support COUNT in SSCAN (thanks @Abdi-dd)
- test and support Go 1.19
- support LPOS (thanks @ianstarz)
- support XPENDING, XGROUP {CREATECONSUMER,DESTROY,DELCONSUMER}, XINFO {CONSUMERS,GROUPS}, XCLAIM (thanks @sandyharvie)
### v2.22.0
- set miniredis.DumpMaxLineLen to get more Dump() info (thanks @afjoseph)
- fix invalid resposne of COMMAND (thanks @zsh1995)
- fix possibility to generate duplicate IDs in XADD (thanks @readams)
- adds support for XAUTOCLAIM min-idle parameter (thanks @readams)
### v2.21.0
- support for GETEX (thanks @dntj)
- support for GT and LT in ZADD (thanks @lsgndln)
- support for XAUTOCLAIM (thanks @randall-fulton)
### v2.20.0
- back to support Go >= 1.14 (thanks @ajatprabha and @marcind)
### v2.19.0
- support for TYPE in SCAN (thanks @0xDiddi)
- update BITPOS (thanks @dirkm)
- fix a lua redis.call() return value (thanks @mpetronic)
- update ZRANGE (thanks @valdemarpereira)
### v2.18.0
- support for ZUNION (thanks @propan)
- support for COPY (thanks @matiasinsaurralde and @rockitbaby)
- support for LMOVE (thanks @btwear)
### v2.17.0
- added miniredis.RunT(t)
### v2.16.1
- fix ZINTERSTORE with sets (thanks @lingjl2010 and @okhowang)
- fix exclusive ranges in XRANGE (thanks @joseotoro)
### v2.16.0
- simplify some code (thanks @zonque)
- support for EXAT/PXAT in SET
- support for XTRIM (thanks @joseotoro)
- support for ZRANDMEMBER
- support for redis.log() in lua (thanks @dirkm)
### v2.15.2
- Fix race condition in blocking code (thanks @zonque and @robx)
- XREAD accepts '$' as ID (thanks @bradengroom)
### v2.15.1
- EVAL should cache the script (thanks @guoshimin)
### v2.15.0
- target redis 6.2 and added new args to various commands
- support for all hyperlog commands (thanks @ilbaktin)
- support for GETDEL (thanks @wszaranski)
### v2.14.5
- added XPENDING
- support for BLOCK option in XREAD and XREADGROUP
### v2.14.4
- fix BITPOS error (thanks @xiaoyuzdy)
- small fixes for XREAD, XACK, and XDEL. Mostly error cases.
- fix empty EXEC return type (thanks @ashanbrown)
- fix XDEL (thanks @svakili and @yvesf)
- fix FLUSHALL for streams (thanks @svakili)
### v2.14.3
- fix problem where Lua code didn't set the selected DB
- update to redis 6.0.10 (thanks @lazappa)
### v2.14.2
- update LUA dependency
- deal with (p)unsubscribe when there are no channels
### v2.14.1
- mod tidy
### v2.14.0
- support for HELLO and the RESP3 protocol
- KEEPTTL in SET (thanks @johnpena)
### v2.13.3
- support Go 1.14 and 1.15
- update the `Check...()` methods
- support for XREAD (thanks @pieterlexis)
### v2.13.2
- Use SAN instead of CN in self signed cert for testing (thanks @johejo)
- Travis CI now tests against the most recent two versions of Go (thanks @johejo)
- changed unit and integration tests to compare raw payloads, not parsed payloads
- remove "redigo" dependency
### v2.13.1
- added HSTRLEN
- minimal support for ACL users in AUTH
### v2.13.0
- added RunTLS(...)
- added SetError(...)
### v2.12.0
- redis 6
- Lua json update (thanks @gsmith85)
- CLUSTER commands (thanks @kratisto)
- fix TOUCH
- fix a shutdown race condition
### v2.11.4
- ZUNIONSTORE now supports standard set types (thanks @wshirey)
### v2.11.3
- support for TOUCH (thanks @cleroux)
- support for cluster and stream commands (thanks @kak-tus)
### v2.11.2
- make sure Lua code is executed concurrently
- add command GEORADIUSBYMEMBER (thanks @kyeett)
### v2.11.1
- globals protection for Lua code (thanks @vk-outreach)
- HSET update (thanks @carlgreen)
- fix BLPOP block on shutdown (thanks @Asalle)
### v2.11.0
- added XRANGE/XREVRANGE, XADD, and XLEN (thanks @skateinmars)
- added GEODIST
- improved precision for geohashes, closer to what real redis does
- use 128bit floats internally for INCRBYFLOAT and related (thanks @timnd)
### v2.10.1
- added m.Server()
### v2.10.0
- added UNLINK
- fix DEL zero-argument case
- cleanup some direct access commands
- added GEOADD, GEOPOS, GEORADIUS, and GEORADIUS_RO
### v2.9.1
- fix issue with ZRANGEBYLEX
- fix issue with BRPOPLPUSH and direct access
### v2.9.0
- proper versioned import of github.com/gomodule/redigo (thanks @yfei1)
- fix messages generated by PSUBSCRIBE
- optional internal seed (thanks @zikaeroh)
### v2.8.0
Proper `v2` in go.mod.
### older
See https://github.com/alicebob/miniredis/releases for the full changelog
+21
View File
@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2014 Harmen
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+33
View File
@@ -0,0 +1,33 @@
.PHONY: test
test: ### Run unit tests
go test ./...
.PHONY: testrace
testrace: ### Run unit tests with race detector
go test -race ./...
.PHONY: int
int: ### Run integration tests (doesn't download redis server)
${MAKE} -C integration int
.PHONY: ci
ci: ### Run full tests suite (including download and compilation of proper redis server)
${MAKE} test
${MAKE} -C integration redis_src/redis-server int
${MAKE} testrace
.PHONY: clean
clean: ### Clean integration test files and remove compiled redis from integration/redis_src
${MAKE} -C integration clean
.PHONY: help
help:
ifeq ($(UNAME), Linux)
@grep -P '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | \
awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'
else
@# this is not tested, but prepared in advance for you, Mac drivers
@awk -F ':.*###' '$$0 ~ FS {printf "%15s%s\n", $$1 ":", $$2}' \
$(MAKEFILE_LIST) | grep -v '@awk' | sort
endif
+342
View File
@@ -0,0 +1,342 @@
# Miniredis
Pure Go Redis test server, used in Go unittests.
##
Sometimes you want to test code which uses Redis, without making it a full-blown
integration test.
Miniredis implements (parts of) the Redis server, to be used in unittests. It
enables a simple, cheap, in-memory, Redis replacement, with a real TCP interface. Think of it as the Redis version of `net/http/httptest`.
It saves you from using mock code, and since the redis server lives in the
test process you can query for values directly, without going through the server
stack.
There are no dependencies on external binaries, so you can easily integrate it in automated build processes.
Be sure to import v2:
```
import "github.com/alicebob/miniredis/v2"
```
## Commands
Implemented commands:
- Connection (complete)
- AUTH -- see RequireAuth()
- ECHO
- HELLO -- see RequireUserAuth()
- PING
- SELECT
- SWAPDB
- QUIT
- Key
- COPY
- DEL
- EXISTS
- EXPIRE
- EXPIREAT
- EXPIRETIME
- KEYS
- MOVE
- PERSIST
- PEXPIRE
- PEXPIREAT
- PEXPIRETIME
- PTTL
- RANDOMKEY -- see m.Seed(...)
- RENAME
- RENAMENX
- SCAN
- TOUCH
- TTL
- TYPE
- UNLINK
- Transactions (complete)
- DISCARD
- EXEC
- MULTI
- UNWATCH
- WATCH
- Server
- DBSIZE
- FLUSHALL
- FLUSHDB
- TIME -- returns time.Now() or value set by SetTime()
- COMMAND -- partly
- INFO -- partly, returns only "clients" section with one field "connected_clients"
- String keys (complete)
- APPEND
- BITCOUNT
- BITOP
- BITPOS
- DECR
- DECRBY
- GET
- GETBIT
- GETRANGE
- GETSET
- GETDEL
- GETEX
- INCR
- INCRBY
- INCRBYFLOAT
- MGET
- MSET
- MSETNX
- PSETEX
- SET
- SETBIT
- SETEX
- SETNX
- SETRANGE
- STRLEN
- Hash keys (complete)
- HDEL
- HEXISTS
- HGET
- HGETALL
- HINCRBY
- HINCRBYFLOAT
- HKEYS
- HLEN
- HMGET
- HMSET
- HRANDFIELD
- HSET
- HSETNX
- HSTRLEN
- HVALS
- HSCAN
- List keys (complete)
- BLPOP
- BRPOP
- BRPOPLPUSH
- LINDEX
- LINSERT
- LLEN
- LPOP
- LPUSH
- LPUSHX
- LRANGE
- LREM
- LSET
- LTRIM
- RPOP
- RPOPLPUSH
- RPUSH
- RPUSHX
- LMOVE
- BLMOVE
- Pub/Sub (complete)
- PSUBSCRIBE
- PUBLISH
- PUBSUB
- PUNSUBSCRIBE
- SUBSCRIBE
- UNSUBSCRIBE
- Set keys (complete)
- SADD
- SCARD
- SDIFF
- SDIFFSTORE
- SINTER
- SINTERSTORE
- SINTERCARD
- SISMEMBER
- SMEMBERS
- SMISMEMBER
- SMOVE
- SPOP -- see m.Seed(...)
- SRANDMEMBER -- see m.Seed(...)
- SREM
- SSCAN
- SUNION
- SUNIONSTORE
- Sorted Set keys (complete)
- ZADD
- ZCARD
- ZCOUNT
- ZINCRBY
- ZINTER
- ZINTERSTORE
- ZLEXCOUNT
- ZPOPMIN
- ZPOPMAX
- ZRANDMEMBER
- ZRANGE
- ZRANGEBYLEX
- ZRANGEBYSCORE
- ZRANK
- ZREM
- ZREMRANGEBYLEX
- ZREMRANGEBYRANK
- ZREMRANGEBYSCORE
- ZREVRANGE
- ZREVRANGEBYLEX
- ZREVRANGEBYSCORE
- ZREVRANK
- ZSCORE
- ZUNION
- ZUNIONSTORE
- ZSCAN
- Stream keys
- XACK
- XADD
- XAUTOCLAIM
- XCLAIM
- XDEL
- XGROUP CREATE
- XGROUP CREATECONSUMER
- XGROUP DESTROY
- XGROUP DELCONSUMER
- XINFO STREAM -- partly
- XINFO GROUPS
- XINFO CONSUMERS -- partly
- XLEN
- XRANGE
- XREAD
- XREADGROUP
- XREVRANGE
- XPENDING
- XTRIM
- Scripting
- EVAL
- EVALSHA
- SCRIPT LOAD
- SCRIPT EXISTS
- SCRIPT FLUSH
- GEO
- GEOADD
- GEODIST
- ~~GEOHASH~~
- GEOPOS
- GEORADIUS
- GEORADIUS_RO
- GEORADIUSBYMEMBER
- GEORADIUSBYMEMBER_RO
- Cluster
- CLUSTER SLOTS
- CLUSTER KEYSLOT
- CLUSTER NODES
- HyperLogLog (complete)
- PFADD
- PFCOUNT
- PFMERGE
## TTLs, key expiration, and time
Since miniredis is intended to be used in unittests TTLs don't decrease
automatically. You can use `TTL()` to get the TTL (as a time.Duration) of a
key. It will return 0 when no TTL is set.
`m.FastForward(d)` can be used to decrement all TTLs. All TTLs which become <=
0 will be removed.
EXPIREAT and PEXPIREAT values will be
converted to a duration. For that you can either set m.SetTime(t) to use that
time as the base for the (P)EXPIREAT conversion, or don't call SetTime(), in
which case time.Now() will be used.
SetTime() also sets the value returned by TIME, which defaults to time.Now().
It is not updated by FastForward, only by SetTime.
## Randomness and Seed()
Miniredis will use `math/rand`'s global RNG for randomness unless a seed is
provided by calling `m.Seed(...)`. If a seed is provided, then miniredis will
use its own RNG based on that seed.
Commands which use randomness are: RANDOMKEY, SPOP, and SRANDMEMBER.
## Example
``` Go
import (
...
"github.com/alicebob/miniredis/v2"
...
)
func TestSomething(t *testing.T) {
s := miniredis.RunT(t)
// Optionally set some keys your code expects:
s.Set("foo", "bar")
s.HSet("some", "other", "key")
// Run your code and see if it behaves.
// An example using the redigo library from "github.com/gomodule/redigo/redis":
c, err := redis.Dial("tcp", s.Addr())
_, err = c.Do("SET", "foo", "bar")
// Optionally check values in redis...
if got, err := s.Get("foo"); err != nil || got != "bar" {
t.Error("'foo' has the wrong value")
}
// ... or use a helper for that:
s.CheckGet(t, "foo", "bar")
// TTL and expiration:
s.Set("foo", "bar")
s.SetTTL("foo", 10*time.Second)
s.FastForward(11 * time.Second)
if s.Exists("foo") {
t.Fatal("'foo' should not have existed anymore")
}
}
```
## Not supported
Commands which will probably not be implemented:
- CLUSTER (all)
- ~~CLUSTER *~~
- ~~READONLY~~
- ~~READWRITE~~
- Key
- ~~DUMP~~
- ~~MIGRATE~~
- ~~OBJECT~~
- ~~RESTORE~~
- ~~WAIT~~
- Scripting
- ~~FCALL / FCALL_RO *~~
- ~~FUNCTION *~~
- ~~SCRIPT DEBUG~~
- ~~SCRIPT KILL~~
- Server
- ~~BGSAVE~~
- ~~BGWRITEAOF~~
- ~~CLIENT *~~
- ~~CONFIG *~~
- ~~DEBUG *~~
- ~~LASTSAVE~~
- ~~MONITOR~~
- ~~ROLE~~
- ~~SAVE~~
- ~~SHUTDOWN~~
- ~~SLAVEOF~~
- ~~SLOWLOG~~
- ~~SYNC~~
## &c.
Integration tests are run against Redis 7.2.4. The [./integration](./integration/) subdir
compares miniredis against a real redis instance.
The Redis 6 RESP3 protocol is supported. If there are problems, please open
an issue.
If you want to test Redis Sentinel have a look at [minisentinel](https://github.com/Bose/minisentinel).
A changelog is kept at [CHANGELOG.md](https://github.com/alicebob/miniredis/blob/master/CHANGELOG.md).
[![Go Reference](https://pkg.go.dev/badge/github.com/alicebob/miniredis/v2.svg)](https://pkg.go.dev/github.com/alicebob/miniredis/v2)
+63
View File
@@ -0,0 +1,63 @@
package miniredis
import (
"reflect"
"sort"
)
// T is implemented by Testing.T
type T interface {
Helper()
Errorf(string, ...interface{})
}
// CheckGet does not call Errorf() iff there is a string key with the
// expected value. Normal use case is `m.CheckGet(t, "username", "theking")`.
func (m *Miniredis) CheckGet(t T, key, expected string) {
t.Helper()
found, err := m.Get(key)
if err != nil {
t.Errorf("GET error, key %#v: %v", key, err)
return
}
if found != expected {
t.Errorf("GET error, key %#v: Expected %#v, got %#v", key, expected, found)
return
}
}
// CheckList does not call Errorf() iff there is a list key with the
// expected values.
// Normal use case is `m.CheckGet(t, "favorite_colors", "red", "green", "infrared")`.
func (m *Miniredis) CheckList(t T, key string, expected ...string) {
t.Helper()
found, err := m.List(key)
if err != nil {
t.Errorf("List error, key %#v: %v", key, err)
return
}
if !reflect.DeepEqual(expected, found) {
t.Errorf("List error, key %#v: Expected %#v, got %#v", key, expected, found)
return
}
}
// CheckSet does not call Errorf() iff there is a set key with the
// expected values.
// Normal use case is `m.CheckSet(t, "visited", "Rome", "Stockholm", "Dublin")`.
func (m *Miniredis) CheckSet(t T, key string, expected ...string) {
t.Helper()
found, err := m.Members(key)
if err != nil {
t.Errorf("Set error, key %#v: %v", key, err)
return
}
sort.Strings(expected)
if !reflect.DeepEqual(expected, found) {
t.Errorf("Set error, key %#v: Expected %#v, got %#v", key, expected, found)
return
}
}
+68
View File
@@ -0,0 +1,68 @@
package miniredis
import (
"fmt"
"strings"
"github.com/alicebob/miniredis/v2/server"
)
// commandsClient handles client operations.
func commandsClient(m *Miniredis) {
m.srv.Register("CLIENT", m.cmdClient)
}
// CLIENT
func (m *Miniredis) cmdClient(c *server.Peer, cmd string, args []string) {
if len(args) == 0 {
setDirty(c)
c.WriteError("ERR wrong number of arguments for 'client' command")
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
switch cmd := strings.ToUpper(args[0]); cmd {
case "SETNAME":
m.cmdClientSetName(c, args[1:])
case "GETNAME":
m.cmdClientGetName(c, args[1:])
default:
setDirty(c)
c.WriteError(fmt.Sprintf("ERR unknown subcommand '%s'. Try CLIENT HELP.", cmd))
}
})
}
// CLIENT SETNAME
func (m *Miniredis) cmdClientSetName(c *server.Peer, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError("ERR wrong number of arguments for 'client setname' command")
return
}
name := args[0]
if strings.ContainsAny(name, " \n") {
setDirty(c)
c.WriteError("ERR Client names cannot contain spaces, newlines or special characters.")
return
}
c.ClientName = name
c.WriteOK()
}
// CLIENT GETNAME
func (m *Miniredis) cmdClientGetName(c *server.Peer, args []string) {
if len(args) > 0 {
setDirty(c)
c.WriteError("ERR wrong number of arguments for 'client getname' command")
return
}
if c.ClientName == "" {
c.WriteNull()
} else {
c.WriteBulk(c.ClientName)
}
}
+67
View File
@@ -0,0 +1,67 @@
// Commands from https://redis.io/commands#cluster
package miniredis
import (
"fmt"
"strings"
"github.com/alicebob/miniredis/v2/server"
)
// commandsCluster handles some cluster operations.
func commandsCluster(m *Miniredis) {
m.srv.Register("CLUSTER", m.cmdCluster)
}
func (m *Miniredis) cmdCluster(c *server.Peer, cmd string, args []string) {
if !m.handleAuth(c) {
return
}
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
switch strings.ToUpper(args[0]) {
case "SLOTS":
m.cmdClusterSlots(c, cmd, args)
case "KEYSLOT":
m.cmdClusterKeySlot(c, cmd, args)
case "NODES":
m.cmdClusterNodes(c, cmd, args)
default:
setDirty(c)
c.WriteError(fmt.Sprintf("ERR 'CLUSTER %s' not supported", strings.Join(args, " ")))
return
}
}
// CLUSTER SLOTS
func (m *Miniredis) cmdClusterSlots(c *server.Peer, cmd string, args []string) {
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
c.WriteLen(1)
c.WriteLen(3)
c.WriteInt(0)
c.WriteInt(16383)
c.WriteLen(3)
c.WriteBulk(m.srv.Addr().IP.String())
c.WriteInt(m.srv.Addr().Port)
c.WriteBulk("09dbe9720cda62f7865eabc5fd8857c5d2678366")
})
}
// CLUSTER KEYSLOT
func (m *Miniredis) cmdClusterKeySlot(c *server.Peer, cmd string, args []string) {
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
c.WriteInt(163)
})
}
// CLUSTER NODES
func (m *Miniredis) cmdClusterNodes(c *server.Peer, cmd string, args []string) {
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
c.WriteBulk("e7d1eecce10fd6bb5eb35b9f99a514335d9ba9ca 127.0.0.1:7000@7000 myself,master - 0 0 1 connected 0-16383")
})
}
File diff suppressed because one or more lines are too long
+285
View File
@@ -0,0 +1,285 @@
// Commands from https://redis.io/commands#connection
package miniredis
import (
"fmt"
"strings"
"github.com/alicebob/miniredis/v2/server"
)
func commandsConnection(m *Miniredis) {
m.srv.Register("AUTH", m.cmdAuth)
m.srv.Register("ECHO", m.cmdEcho)
m.srv.Register("HELLO", m.cmdHello)
m.srv.Register("PING", m.cmdPing)
m.srv.Register("QUIT", m.cmdQuit)
m.srv.Register("SELECT", m.cmdSelect)
m.srv.Register("SWAPDB", m.cmdSwapdb)
}
// PING
func (m *Miniredis) cmdPing(c *server.Peer, cmd string, args []string) {
if !m.handleAuth(c) {
return
}
if len(args) > 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
payload := ""
if len(args) > 0 {
payload = args[0]
}
// PING is allowed in subscribed state
if sub := getCtx(c).subscriber; sub != nil {
c.Block(func(c *server.Writer) {
c.WriteLen(2)
c.WriteBulk("pong")
c.WriteBulk(payload)
})
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
if payload == "" {
c.WriteInline("PONG")
return
}
c.WriteBulk(payload)
})
}
// AUTH
func (m *Miniredis) cmdAuth(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if len(args) > 2 {
c.WriteError(msgSyntaxError)
return
}
if m.checkPubsub(c, cmd) {
return
}
ctx := getCtx(c)
if ctx.nested {
c.WriteError(msgNotFromScripts(ctx.nestedSHA))
return
}
var opts = struct {
username string
password string
}{
username: "default",
password: args[0],
}
if len(args) == 2 {
opts.username, opts.password = args[0], args[1]
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
if len(m.passwords) == 0 && opts.username == "default" {
c.WriteError("ERR AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?")
return
}
setPW, ok := m.passwords[opts.username]
if !ok {
c.WriteError("WRONGPASS invalid username-password pair")
return
}
if setPW != opts.password {
c.WriteError("WRONGPASS invalid username-password pair")
return
}
ctx.authenticated = true
c.WriteOK()
})
}
// HELLO
func (m *Miniredis) cmdHello(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
c.WriteError(errWrongNumber(cmd))
return
}
var opts struct {
version int
username string
password string
}
if ok := optIntErr(c, args[0], &opts.version, "ERR Protocol version is not an integer or out of range"); !ok {
return
}
args = args[1:]
switch opts.version {
case 2, 3:
default:
c.WriteError("NOPROTO unsupported protocol version")
return
}
var checkAuth bool
for len(args) > 0 {
switch strings.ToUpper(args[0]) {
case "AUTH":
if len(args) < 3 {
c.WriteError(fmt.Sprintf("ERR Syntax error in HELLO option '%s'", args[0]))
return
}
opts.username, opts.password, args = args[1], args[2], args[3:]
checkAuth = true
case "SETNAME":
if len(args) < 2 {
c.WriteError(fmt.Sprintf("ERR Syntax error in HELLO option '%s'", args[0]))
return
}
_, args = args[1], args[2:]
default:
c.WriteError(fmt.Sprintf("ERR Syntax error in HELLO option '%s'", args[0]))
return
}
}
if len(m.passwords) == 0 && opts.username == "default" {
// redis ignores legacy "AUTH" if it's not enabled.
checkAuth = false
}
if checkAuth {
setPW, ok := m.passwords[opts.username]
if !ok {
c.WriteError("WRONGPASS invalid username-password pair")
return
}
if setPW != opts.password {
c.WriteError("WRONGPASS invalid username-password pair")
return
}
getCtx(c).authenticated = true
}
c.Resp3 = opts.version == 3
c.WriteMapLen(7)
c.WriteBulk("server")
c.WriteBulk("miniredis")
c.WriteBulk("version")
c.WriteBulk("6.0.5")
c.WriteBulk("proto")
c.WriteInt(opts.version)
c.WriteBulk("id")
c.WriteInt(42)
c.WriteBulk("mode")
c.WriteBulk("standalone")
c.WriteBulk("role")
c.WriteBulk("master")
c.WriteBulk("modules")
c.WriteLen(0)
}
// ECHO
func (m *Miniredis) cmdEcho(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
msg := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
c.WriteBulk(msg)
})
}
// SELECT
func (m *Miniredis) cmdSelect(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.isValidCMD(c, cmd) {
return
}
var opts struct {
id int
}
if ok := optInt(c, args[0], &opts.id); !ok {
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
if opts.id < 0 {
c.WriteError(msgDBIndexOutOfRange)
setDirty(c)
return
}
ctx.selectedDB = opts.id
c.WriteOK()
})
}
// SWAPDB
func (m *Miniredis) cmdSwapdb(c *server.Peer, cmd string, args []string) {
if len(args) != 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
var opts struct {
id1 int
id2 int
}
if ok := optIntErr(c, args[0], &opts.id1, "ERR invalid first DB index"); !ok {
return
}
if ok := optIntErr(c, args[1], &opts.id2, "ERR invalid second DB index"); !ok {
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
if opts.id1 < 0 || opts.id2 < 0 {
c.WriteError(msgDBIndexOutOfRange)
setDirty(c)
return
}
m.swapDB(opts.id1, opts.id2)
c.WriteOK()
})
}
// QUIT
func (m *Miniredis) cmdQuit(c *server.Peer, cmd string, args []string) {
// QUIT isn't transactionfied and accepts any arguments.
c.WriteOK()
c.Close()
}
+813
View File
@@ -0,0 +1,813 @@
// Commands from https://redis.io/commands#generic
package miniredis
import (
"errors"
"fmt"
"sort"
"strconv"
"strings"
"time"
"github.com/alicebob/miniredis/v2/server"
)
const (
// expiretimeReplyNoExpiration is return value for EXPIRETIME and PEXPIRETIME if the key exists but has no associated expiration time
expiretimeReplyNoExpiration = -1
// expiretimeReplyMissingKey is return value for EXPIRETIME and PEXPIRETIME if the key does not exist
expiretimeReplyMissingKey = -2
)
func inSeconds(t time.Time) int {
return int(t.Unix())
}
func inMilliSeconds(t time.Time) int {
return int(t.UnixMilli())
}
// commandsGeneric handles EXPIRE, TTL, PERSIST, &c.
func commandsGeneric(m *Miniredis) {
m.srv.Register("COPY", m.cmdCopy)
m.srv.Register("DEL", m.cmdDel)
// DUMP
m.srv.Register("EXISTS", m.cmdExists)
m.srv.Register("EXPIRE", makeCmdExpire(m, false, time.Second))
m.srv.Register("EXPIREAT", makeCmdExpire(m, true, time.Second))
m.srv.Register("EXPIRETIME", m.makeCmdExpireTime(inSeconds))
m.srv.Register("PEXPIRETIME", m.makeCmdExpireTime(inMilliSeconds))
m.srv.Register("KEYS", m.cmdKeys)
// MIGRATE
m.srv.Register("MOVE", m.cmdMove)
// OBJECT
m.srv.Register("PERSIST", m.cmdPersist)
m.srv.Register("PEXPIRE", makeCmdExpire(m, false, time.Millisecond))
m.srv.Register("PEXPIREAT", makeCmdExpire(m, true, time.Millisecond))
m.srv.Register("PTTL", m.cmdPTTL)
m.srv.Register("RANDOMKEY", m.cmdRandomkey)
m.srv.Register("RENAME", m.cmdRename)
m.srv.Register("RENAMENX", m.cmdRenamenx)
// RESTORE
m.srv.Register("TOUCH", m.cmdTouch)
m.srv.Register("TTL", m.cmdTTL)
m.srv.Register("TYPE", m.cmdType)
m.srv.Register("SCAN", m.cmdScan)
// SORT
m.srv.Register("UNLINK", m.cmdDel)
}
type expireOpts struct {
key string
value int
nx bool
xx bool
gt bool
lt bool
}
func expireParse(cmd string, args []string) (*expireOpts, error) {
var opts expireOpts
opts.key = args[0]
if err := optIntSimple(args[1], &opts.value); err != nil {
return nil, err
}
args = args[2:]
for len(args) > 0 {
switch strings.ToLower(args[0]) {
case "nx":
opts.nx = true
case "xx":
opts.xx = true
case "gt":
opts.gt = true
case "lt":
opts.lt = true
default:
return nil, fmt.Errorf("ERR Unsupported option %s", args[0])
}
args = args[1:]
}
if opts.gt && opts.lt {
return nil, errors.New("ERR GT and LT options at the same time are not compatible")
}
if opts.nx && (opts.xx || opts.gt || opts.lt) {
return nil, errors.New("ERR NX and XX, GT or LT options at the same time are not compatible")
}
return &opts, nil
}
// generic expire command for EXPIRE, PEXPIRE, EXPIREAT, PEXPIREAT
// d is the time unit. If unix is set it'll be seen as a unixtimestamp and
// converted to a duration.
func makeCmdExpire(m *Miniredis, unix bool, d time.Duration) func(*server.Peer, string, []string) {
return func(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts, err := expireParse(cmd, args)
if err != nil {
setDirty(c)
c.WriteError(err.Error())
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
// Key must be present.
if _, ok := db.keys[opts.key]; !ok {
c.WriteInt(0)
return
}
oldTTL, ok := db.ttl[opts.key]
var newTTL time.Duration
if unix {
newTTL = m.at(opts.value, d)
} else {
newTTL = time.Duration(opts.value) * d
}
// > NX -- Set expiry only when the key has no expiry
if opts.nx && ok {
c.WriteInt(0)
return
}
// > XX -- Set expiry only when the key has an existing expiry
if opts.xx && !ok {
c.WriteInt(0)
return
}
// > GT -- Set expiry only when the new expiry is greater than current one
// (no exp == infinity)
if opts.gt && (!ok || newTTL <= oldTTL) {
c.WriteInt(0)
return
}
// > LT -- Set expiry only when the new expiry is less than current one
if opts.lt && ok && newTTL > oldTTL {
c.WriteInt(0)
return
}
db.ttl[opts.key] = newTTL
db.incr(opts.key)
db.checkTTL(opts.key)
c.WriteInt(1)
})
}
}
// makeCmdExpireTime creates server command function that returns the absolute Unix timestamp (since January 1, 1970)
// at which the given key will expire, in unit selected by time result strategy (e.g. seconds, milliseconds).
// For more information see redis documentation for [expiretime] and [pexpiretime].
//
// [expiretime]: https://redis.io/commands/expiretime/
// [pexpiretime]: https://redis.io/commands/pexpiretime/
func (m *Miniredis) makeCmdExpireTime(timeResultStrategy func(time.Time) int) server.Cmd {
return func(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if _, ok := db.keys[key]; !ok {
c.WriteInt(expiretimeReplyMissingKey)
return
}
ttl, ok := db.ttl[key]
if !ok {
c.WriteInt(expiretimeReplyNoExpiration)
return
}
c.WriteInt(timeResultStrategy(m.effectiveNow().Add(ttl)))
})
}
}
// TOUCH
func (m *Miniredis) cmdTouch(c *server.Peer, cmd string, args []string) {
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
if len(args) == 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
count := 0
for _, key := range args {
if db.exists(key) {
count++
}
}
c.WriteInt(count)
})
}
// TTL
func (m *Miniredis) cmdTTL(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if _, ok := db.keys[key]; !ok {
// No such key
c.WriteInt(-2)
return
}
v, ok := db.ttl[key]
if !ok {
// no expire value
c.WriteInt(-1)
return
}
c.WriteInt(int(v.Seconds()))
})
}
// PTTL
func (m *Miniredis) cmdPTTL(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if _, ok := db.keys[key]; !ok {
// no such key
c.WriteInt(-2)
return
}
v, ok := db.ttl[key]
if !ok {
// no expire value
c.WriteInt(-1)
return
}
c.WriteInt(int(v.Nanoseconds() / 1000000))
})
}
// PERSIST
func (m *Miniredis) cmdPersist(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if _, ok := db.keys[key]; !ok {
// no such key
c.WriteInt(0)
return
}
if _, ok := db.ttl[key]; !ok {
// no expire value
c.WriteInt(0)
return
}
delete(db.ttl, key)
db.incr(key)
c.WriteInt(1)
})
}
// DEL and UNLINK
func (m *Miniredis) cmdDel(c *server.Peer, cmd string, args []string) {
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
if len(args) == 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
count := 0
for _, key := range args {
if db.exists(key) {
count++
}
db.del(key, true) // delete expire
}
c.WriteInt(count)
})
}
// TYPE
func (m *Miniredis) cmdType(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError("usage error")
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
t, ok := db.keys[key]
if !ok {
c.WriteInline("none")
return
}
c.WriteInline(t)
})
}
// EXISTS
func (m *Miniredis) cmdExists(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
found := 0
for _, k := range args {
if db.exists(k) {
found++
}
}
c.WriteInt(found)
})
}
// MOVE
func (m *Miniredis) cmdMove(c *server.Peer, cmd string, args []string) {
if len(args) != 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
var opts struct {
key string
targetDB int
}
opts.key = args[0]
opts.targetDB, _ = strconv.Atoi(args[1])
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
if ctx.selectedDB == opts.targetDB {
c.WriteError("ERR source and destination objects are the same")
return
}
db := m.db(ctx.selectedDB)
targetDB := m.db(opts.targetDB)
if !db.move(opts.key, targetDB) {
c.WriteInt(0)
return
}
c.WriteInt(1)
})
}
// KEYS
func (m *Miniredis) cmdKeys(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
keys, _ := matchKeys(db.allKeys(), key)
c.WriteLen(len(keys))
for _, s := range keys {
c.WriteBulk(s)
}
})
}
// RANDOMKEY
func (m *Miniredis) cmdRandomkey(c *server.Peer, cmd string, args []string) {
if len(args) != 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if len(db.keys) == 0 {
c.WriteNull()
return
}
nr := m.randIntn(len(db.keys))
for k := range db.keys {
if nr == 0 {
c.WriteBulk(k)
return
}
nr--
}
})
}
// RENAME
func (m *Miniredis) cmdRename(c *server.Peer, cmd string, args []string) {
if len(args) != 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts := struct {
from string
to string
}{
from: args[0],
to: args[1],
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if !db.exists(opts.from) {
c.WriteError(msgKeyNotFound)
return
}
db.rename(opts.from, opts.to)
c.WriteOK()
})
}
// RENAMENX
func (m *Miniredis) cmdRenamenx(c *server.Peer, cmd string, args []string) {
if len(args) != 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts := struct {
from string
to string
}{
from: args[0],
to: args[1],
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if !db.exists(opts.from) {
c.WriteError(msgKeyNotFound)
return
}
if db.exists(opts.to) {
c.WriteInt(0)
return
}
db.rename(opts.from, opts.to)
c.WriteInt(1)
})
}
type scanOpts struct {
cursor int
count int
withMatch bool
match string
withType bool
_type string
}
func scanParse(cmd string, args []string) (*scanOpts, error) {
var opts scanOpts
if err := optIntSimple(args[0], &opts.cursor); err != nil {
return nil, errors.New(msgInvalidCursor)
}
args = args[1:]
// MATCH, COUNT and TYPE options
for len(args) > 0 {
if strings.ToLower(args[0]) == "count" {
if len(args) < 2 {
return nil, errors.New(msgSyntaxError)
}
count, err := strconv.Atoi(args[1])
if err != nil || count < 0 {
return nil, errors.New(msgInvalidInt)
}
if count == 0 {
return nil, errors.New(msgSyntaxError)
}
opts.count = count
args = args[2:]
continue
}
if strings.ToLower(args[0]) == "match" {
if len(args) < 2 {
return nil, errors.New(msgSyntaxError)
}
opts.withMatch = true
opts.match, args = args[1], args[2:]
continue
}
if strings.ToLower(args[0]) == "type" {
if len(args) < 2 {
return nil, errors.New(msgSyntaxError)
}
opts.withType = true
opts._type, args = strings.ToLower(args[1]), args[2:]
continue
}
return nil, errors.New(msgSyntaxError)
}
return &opts, nil
}
// SCAN
func (m *Miniredis) cmdScan(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts, err := scanParse(cmd, args)
if err != nil {
setDirty(c)
c.WriteError(err.Error())
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
// We return _all_ (matched) keys every time.
var keys []string
if opts.withType {
keys = make([]string, 0)
for k, t := range db.keys {
// type must be given exactly; no pattern matching is performed
if t == opts._type {
keys = append(keys, k)
}
}
} else {
keys = db.allKeys()
}
sort.Strings(keys) // To make things deterministic.
if opts.withMatch {
keys, _ = matchKeys(keys, opts.match)
}
low := opts.cursor
high := low + opts.count
// validate high is correct
if high > len(keys) || high == 0 {
high = len(keys)
}
if opts.cursor > high {
// invalid cursor
c.WriteLen(2)
c.WriteBulk("0") // no next cursor
c.WriteLen(0) // no elements
return
}
cursorValue := low + opts.count
if cursorValue >= len(keys) {
cursorValue = 0 // no next cursor
}
keys = keys[low:high]
c.WriteLen(2)
c.WriteBulk(fmt.Sprintf("%d", cursorValue))
c.WriteLen(len(keys))
for _, k := range keys {
c.WriteBulk(k)
}
})
}
type copyOpts struct {
from string
to string
destinationDB int
replace bool
}
func copyParse(cmd string, args []string) (*copyOpts, error) {
opts := copyOpts{
destinationDB: -1,
}
opts.from, opts.to, args = args[0], args[1], args[2:]
for len(args) > 0 {
switch strings.ToLower(args[0]) {
case "db":
if len(args) < 2 {
return nil, errors.New(msgSyntaxError)
}
if err := optIntSimple(args[1], &opts.destinationDB); err != nil {
return nil, err
}
if opts.destinationDB < 0 {
return nil, errors.New(msgDBIndexOutOfRange)
}
args = args[2:]
case "replace":
opts.replace = true
args = args[1:]
default:
return nil, errors.New(msgSyntaxError)
}
}
return &opts, nil
}
// COPY
func (m *Miniredis) cmdCopy(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts, err := copyParse(cmd, args)
if err != nil {
setDirty(c)
c.WriteError(err.Error())
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
fromDB, toDB := ctx.selectedDB, opts.destinationDB
if toDB == -1 {
toDB = fromDB
}
if fromDB == toDB && opts.from == opts.to {
c.WriteError("ERR source and destination objects are the same")
return
}
if !m.db(fromDB).exists(opts.from) {
c.WriteInt(0)
return
}
if !opts.replace {
if m.db(toDB).exists(opts.to) {
c.WriteInt(0)
return
}
}
m.copy(m.db(fromDB), opts.from, m.db(toDB), opts.to)
c.WriteInt(1)
})
}
+609
View File
@@ -0,0 +1,609 @@
// Commands from https://redis.io/commands#geo
package miniredis
import (
"fmt"
"sort"
"strconv"
"strings"
"github.com/alicebob/miniredis/v2/server"
)
// commandsGeo handles GEOADD, GEORADIUS etc.
func commandsGeo(m *Miniredis) {
m.srv.Register("GEOADD", m.cmdGeoadd)
m.srv.Register("GEODIST", m.cmdGeodist)
m.srv.Register("GEOPOS", m.cmdGeopos)
m.srv.Register("GEORADIUS", m.cmdGeoradius)
m.srv.Register("GEORADIUS_RO", m.cmdGeoradius)
m.srv.Register("GEORADIUSBYMEMBER", m.cmdGeoradiusbymember)
m.srv.Register("GEORADIUSBYMEMBER_RO", m.cmdGeoradiusbymember)
}
// GEOADD
func (m *Miniredis) cmdGeoadd(c *server.Peer, cmd string, args []string) {
if len(args) < 3 || len(args[1:])%3 != 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key, args := args[0], args[1:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if db.exists(key) && db.t(key) != keyTypeSortedSet {
c.WriteError(ErrWrongType.Error())
return
}
toSet := map[string]float64{}
for len(args) > 2 {
rawLong, rawLat, name := args[0], args[1], args[2]
args = args[3:]
longitude, err := strconv.ParseFloat(rawLong, 64)
if err != nil {
c.WriteError("ERR value is not a valid float")
return
}
latitude, err := strconv.ParseFloat(rawLat, 64)
if err != nil {
c.WriteError("ERR value is not a valid float")
return
}
if latitude < -85.05112878 ||
latitude > 85.05112878 ||
longitude < -180 ||
longitude > 180 {
c.WriteError(fmt.Sprintf("ERR invalid longitude,latitude pair %.6f,%.6f", longitude, latitude))
return
}
toSet[name] = float64(toGeohash(longitude, latitude))
}
set := 0
for name, score := range toSet {
if db.ssetAdd(key, score, name) {
set++
}
}
c.WriteInt(set)
})
}
// GEODIST
func (m *Miniredis) cmdGeodist(c *server.Peer, cmd string, args []string) {
if len(args) < 3 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key, from, to, args := args[0], args[1], args[2], args[3:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if !db.exists(key) {
c.WriteNull()
return
}
if db.t(key) != keyTypeSortedSet {
c.WriteError(ErrWrongType.Error())
return
}
unit := "m"
if len(args) > 0 {
unit, args = args[0], args[1:]
}
if len(args) > 0 {
c.WriteError(msgSyntaxError)
return
}
toMeter := parseUnit(unit)
if toMeter == 0 {
c.WriteError(msgUnsupportedUnit)
return
}
members := db.sortedsetKeys[key]
fromD, okFrom := members.get(from)
toD, okTo := members.get(to)
if !okFrom || !okTo {
c.WriteNull()
return
}
fromLo, fromLat := fromGeohash(uint64(fromD))
toLo, toLat := fromGeohash(uint64(toD))
dist := distance(fromLat, fromLo, toLat, toLo) / toMeter
c.WriteBulk(fmt.Sprintf("%.4f", dist))
})
}
// GEOPOS
func (m *Miniredis) cmdGeopos(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key, args := args[0], args[1:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if db.exists(key) && db.t(key) != keyTypeSortedSet {
c.WriteError(ErrWrongType.Error())
return
}
c.WriteLen(len(args))
for _, l := range args {
if !db.ssetExists(key, l) {
c.WriteLen(-1)
continue
}
score := db.ssetScore(key, l)
c.WriteLen(2)
long, lat := fromGeohash(uint64(score))
c.WriteBulk(fmt.Sprintf("%f", long))
c.WriteBulk(fmt.Sprintf("%f", lat))
}
})
}
type geoDistance struct {
Name string
Score float64
Distance float64
Longitude float64
Latitude float64
}
// GEORADIUS and GEORADIUS_RO
func (m *Miniredis) cmdGeoradius(c *server.Peer, cmd string, args []string) {
if len(args) < 5 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
longitude, err := strconv.ParseFloat(args[1], 64)
if err != nil {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
latitude, err := strconv.ParseFloat(args[2], 64)
if err != nil {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
radius, err := strconv.ParseFloat(args[3], 64)
if err != nil || radius < 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
toMeter := parseUnit(args[4])
if toMeter == 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
args = args[5:]
var opts struct {
withDist bool
withCoord bool
direction direction // unsorted
count int
withStore bool
storeKey string
withStoredist bool
storedistKey string
}
for len(args) > 0 {
arg := args[0]
args = args[1:]
switch strings.ToUpper(arg) {
case "WITHCOORD":
opts.withCoord = true
case "WITHDIST":
opts.withDist = true
case "ASC":
opts.direction = asc
case "DESC":
opts.direction = desc
case "COUNT":
if len(args) == 0 {
setDirty(c)
c.WriteError("ERR syntax error")
return
}
n, err := strconv.Atoi(args[0])
if err != nil {
setDirty(c)
c.WriteError(msgInvalidInt)
return
}
if n <= 0 {
setDirty(c)
c.WriteError("ERR COUNT must be > 0")
return
}
args = args[1:]
opts.count = n
case "STORE":
if len(args) == 0 {
setDirty(c)
c.WriteError("ERR syntax error")
return
}
opts.withStore = true
opts.storeKey = args[0]
args = args[1:]
case "STOREDIST":
if len(args) == 0 {
setDirty(c)
c.WriteError("ERR syntax error")
return
}
opts.withStoredist = true
opts.storedistKey = args[0]
args = args[1:]
default:
setDirty(c)
c.WriteError("ERR syntax error")
return
}
}
if strings.ToUpper(cmd) == "GEORADIUS_RO" && (opts.withStore || opts.withStoredist) {
setDirty(c)
c.WriteError("ERR syntax error")
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
if (opts.withStore || opts.withStoredist) && (opts.withDist || opts.withCoord) {
c.WriteError("ERR STORE option in GEORADIUS is not compatible with WITHDIST, WITHHASH and WITHCOORDS options")
return
}
db := m.db(ctx.selectedDB)
members := db.ssetElements(key)
matches := withinRadius(members, longitude, latitude, radius*toMeter)
// deal with ASC/DESC
if opts.direction != unsorted {
sort.Slice(matches, func(i, j int) bool {
if opts.direction == desc {
return matches[i].Distance > matches[j].Distance
}
return matches[i].Distance < matches[j].Distance
})
}
// deal with COUNT
if opts.count > 0 && len(matches) > opts.count {
matches = matches[:opts.count]
}
// deal with "STORE x"
if opts.withStore {
db.del(opts.storeKey, true)
for _, member := range matches {
db.ssetAdd(opts.storeKey, member.Score, member.Name)
}
c.WriteInt(len(matches))
return
}
// deal with "STOREDIST x"
if opts.withStoredist {
db.del(opts.storedistKey, true)
for _, member := range matches {
db.ssetAdd(opts.storedistKey, member.Distance/toMeter, member.Name)
}
c.WriteInt(len(matches))
return
}
c.WriteLen(len(matches))
for _, member := range matches {
if !opts.withDist && !opts.withCoord {
c.WriteBulk(member.Name)
continue
}
len := 1
if opts.withDist {
len++
}
if opts.withCoord {
len++
}
c.WriteLen(len)
c.WriteBulk(member.Name)
if opts.withDist {
c.WriteBulk(fmt.Sprintf("%.4f", member.Distance/toMeter))
}
if opts.withCoord {
c.WriteLen(2)
c.WriteBulk(fmt.Sprintf("%f", member.Longitude))
c.WriteBulk(fmt.Sprintf("%f", member.Latitude))
}
}
})
}
// GEORADIUSBYMEMBER and GEORADIUSBYMEMBER_RO
func (m *Miniredis) cmdGeoradiusbymember(c *server.Peer, cmd string, args []string) {
if len(args) < 4 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts := struct {
key string
member string
radius float64
toMeter float64
withDist bool
withCoord bool
direction direction // unsorted
count int
withStore bool
storeKey string
withStoredist bool
storedistKey string
}{
key: args[0],
member: args[1],
}
r, err := strconv.ParseFloat(args[2], 64)
if err != nil || r < 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
opts.radius = r
opts.toMeter = parseUnit(args[3])
if opts.toMeter == 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
args = args[4:]
for len(args) > 0 {
arg := args[0]
args = args[1:]
switch strings.ToUpper(arg) {
case "WITHCOORD":
opts.withCoord = true
case "WITHDIST":
opts.withDist = true
case "ASC":
opts.direction = asc
case "DESC":
opts.direction = desc
case "COUNT":
if len(args) == 0 {
setDirty(c)
c.WriteError("ERR syntax error")
return
}
n, err := strconv.Atoi(args[0])
if err != nil {
setDirty(c)
c.WriteError(msgInvalidInt)
return
}
if n <= 0 {
setDirty(c)
c.WriteError("ERR COUNT must be > 0")
return
}
args = args[1:]
opts.count = n
case "STORE":
if len(args) == 0 {
setDirty(c)
c.WriteError("ERR syntax error")
return
}
opts.withStore = true
opts.storeKey = args[0]
args = args[1:]
case "STOREDIST":
if len(args) == 0 {
setDirty(c)
c.WriteError("ERR syntax error")
return
}
opts.withStoredist = true
opts.storedistKey = args[0]
args = args[1:]
default:
setDirty(c)
c.WriteError("ERR syntax error")
return
}
}
if strings.ToUpper(cmd) == "GEORADIUSBYMEMBER_RO" && (opts.withStore || opts.withStoredist) {
setDirty(c)
c.WriteError("ERR syntax error")
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
if (opts.withStore || opts.withStoredist) && (opts.withDist || opts.withCoord) {
c.WriteError("ERR STORE option in GEORADIUS is not compatible with WITHDIST, WITHHASH and WITHCOORDS options")
return
}
db := m.db(ctx.selectedDB)
if !db.exists(opts.key) {
c.WriteNull()
return
}
if db.t(opts.key) != keyTypeSortedSet {
c.WriteError(ErrWrongType.Error())
return
}
// get position of member
if !db.ssetExists(opts.key, opts.member) {
c.WriteError("ERR could not decode requested zset member")
return
}
score := db.ssetScore(opts.key, opts.member)
longitude, latitude := fromGeohash(uint64(score))
members := db.ssetElements(opts.key)
matches := withinRadius(members, longitude, latitude, opts.radius*opts.toMeter)
// deal with ASC/DESC
if opts.direction != unsorted {
sort.Slice(matches, func(i, j int) bool {
if opts.direction == desc {
return matches[i].Distance > matches[j].Distance
}
return matches[i].Distance < matches[j].Distance
})
}
// deal with COUNT
if opts.count > 0 && len(matches) > opts.count {
matches = matches[:opts.count]
}
// deal with "STORE x"
if opts.withStore {
db.del(opts.storeKey, true)
for _, member := range matches {
db.ssetAdd(opts.storeKey, member.Score, member.Name)
}
c.WriteInt(len(matches))
return
}
// deal with "STOREDIST x"
if opts.withStoredist {
db.del(opts.storedistKey, true)
for _, member := range matches {
db.ssetAdd(opts.storedistKey, member.Distance/opts.toMeter, member.Name)
}
c.WriteInt(len(matches))
return
}
c.WriteLen(len(matches))
for _, member := range matches {
if !opts.withDist && !opts.withCoord {
c.WriteBulk(member.Name)
continue
}
len := 1
if opts.withDist {
len++
}
if opts.withCoord {
len++
}
c.WriteLen(len)
c.WriteBulk(member.Name)
if opts.withDist {
c.WriteBulk(fmt.Sprintf("%.4f", member.Distance/opts.toMeter))
}
if opts.withCoord {
c.WriteLen(2)
c.WriteBulk(fmt.Sprintf("%f", member.Longitude))
c.WriteBulk(fmt.Sprintf("%f", member.Latitude))
}
}
})
}
func withinRadius(members []ssElem, longitude, latitude, radius float64) []geoDistance {
matches := []geoDistance{}
for _, el := range members {
elLo, elLat := fromGeohash(uint64(el.score))
distanceInMeter := distance(latitude, longitude, elLat, elLo)
if distanceInMeter <= radius {
matches = append(matches, geoDistance{
Name: el.member,
Score: el.score,
Distance: distanceInMeter,
Longitude: elLo,
Latitude: elLat,
})
}
}
return matches
}
func parseUnit(u string) float64 {
switch strings.ToLower(u) {
case "m":
return 1
case "km":
return 1000
case "mi":
return 1609.34
case "ft":
return 0.3048
default:
return 0
}
}
+777
View File
@@ -0,0 +1,777 @@
// Commands from https://redis.io/commands#hash
package miniredis
import (
"math/big"
"strconv"
"strings"
"github.com/alicebob/miniredis/v2/server"
)
// commandsHash handles all hash value operations.
func commandsHash(m *Miniredis) {
m.srv.Register("HDEL", m.cmdHdel)
m.srv.Register("HEXISTS", m.cmdHexists)
m.srv.Register("HGET", m.cmdHget)
m.srv.Register("HGETALL", m.cmdHgetall)
m.srv.Register("HINCRBY", m.cmdHincrby)
m.srv.Register("HINCRBYFLOAT", m.cmdHincrbyfloat)
m.srv.Register("HKEYS", m.cmdHkeys)
m.srv.Register("HLEN", m.cmdHlen)
m.srv.Register("HMGET", m.cmdHmget)
m.srv.Register("HMSET", m.cmdHmset)
m.srv.Register("HSET", m.cmdHset)
m.srv.Register("HSETNX", m.cmdHsetnx)
m.srv.Register("HSTRLEN", m.cmdHstrlen)
m.srv.Register("HVALS", m.cmdHvals)
m.srv.Register("HSCAN", m.cmdHscan)
m.srv.Register("HRANDFIELD", m.cmdHrandfield)
}
// HSET
func (m *Miniredis) cmdHset(c *server.Peer, cmd string, args []string) {
if len(args) < 3 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key, pairs := args[0], args[1:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if len(pairs)%2 == 1 {
c.WriteError(errWrongNumber(cmd))
return
}
if t, ok := db.keys[key]; ok && t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
new := db.hashSet(key, pairs...)
c.WriteInt(new)
})
}
// HSETNX
func (m *Miniredis) cmdHsetnx(c *server.Peer, cmd string, args []string) {
if len(args) != 3 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts := struct {
key string
field string
value string
}{
key: args[0],
field: args[1],
value: args[2],
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if t, ok := db.keys[opts.key]; ok && t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
if _, ok := db.hashKeys[opts.key]; !ok {
db.hashKeys[opts.key] = map[string]string{}
db.keys[opts.key] = keyTypeHash
}
_, ok := db.hashKeys[opts.key][opts.field]
if ok {
c.WriteInt(0)
return
}
db.hashKeys[opts.key][opts.field] = opts.value
db.incr(opts.key)
c.WriteInt(1)
})
}
// HMSET
func (m *Miniredis) cmdHmset(c *server.Peer, cmd string, args []string) {
if len(args) < 3 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key, args := args[0], args[1:]
if len(args)%2 != 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if t, ok := db.keys[key]; ok && t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
for len(args) > 0 {
field, value := args[0], args[1]
args = args[2:]
db.hashSet(key, field, value)
}
c.WriteOK()
})
}
// HGET
func (m *Miniredis) cmdHget(c *server.Peer, cmd string, args []string) {
if len(args) != 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key, field := args[0], args[1]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
t, ok := db.keys[key]
if !ok {
c.WriteNull()
return
}
if t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
value, ok := db.hashKeys[key][field]
if !ok {
c.WriteNull()
return
}
c.WriteBulk(value)
})
}
// HDEL
func (m *Miniredis) cmdHdel(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts := struct {
key string
fields []string
}{
key: args[0],
fields: args[1:],
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
t, ok := db.keys[opts.key]
if !ok {
// No key is zero deleted
c.WriteInt(0)
return
}
if t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
deleted := 0
for _, f := range opts.fields {
_, ok := db.hashKeys[opts.key][f]
if !ok {
continue
}
delete(db.hashKeys[opts.key], f)
deleted++
}
c.WriteInt(deleted)
// Nothing left. Remove the whole key.
if len(db.hashKeys[opts.key]) == 0 {
db.del(opts.key, true)
}
})
}
// HEXISTS
func (m *Miniredis) cmdHexists(c *server.Peer, cmd string, args []string) {
if len(args) != 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts := struct {
key string
field string
}{
key: args[0],
field: args[1],
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
t, ok := db.keys[opts.key]
if !ok {
c.WriteInt(0)
return
}
if t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
if _, ok := db.hashKeys[opts.key][opts.field]; !ok {
c.WriteInt(0)
return
}
c.WriteInt(1)
})
}
// HGETALL
func (m *Miniredis) cmdHgetall(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
t, ok := db.keys[key]
if !ok {
c.WriteMapLen(0)
return
}
if t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
c.WriteMapLen(len(db.hashKeys[key]))
for _, k := range db.hashFields(key) {
c.WriteBulk(k)
c.WriteBulk(db.hashGet(key, k))
}
})
}
// HKEYS
func (m *Miniredis) cmdHkeys(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if !db.exists(key) {
c.WriteLen(0)
return
}
if db.t(key) != keyTypeHash {
c.WriteError(msgWrongType)
return
}
fields := db.hashFields(key)
c.WriteLen(len(fields))
for _, f := range fields {
c.WriteBulk(f)
}
})
}
// HSTRLEN
func (m *Miniredis) cmdHstrlen(c *server.Peer, cmd string, args []string) {
if len(args) != 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
hash, key := args[0], args[1]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
t, ok := db.keys[hash]
if !ok {
c.WriteInt(0)
return
}
if t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
keys := db.hashKeys[hash]
c.WriteInt(len(keys[key]))
})
}
// HVALS
func (m *Miniredis) cmdHvals(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
t, ok := db.keys[key]
if !ok {
c.WriteLen(0)
return
}
if t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
vals := db.hashValues(key)
c.WriteLen(len(vals))
for _, v := range vals {
c.WriteBulk(v)
}
})
}
// HLEN
func (m *Miniredis) cmdHlen(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
t, ok := db.keys[key]
if !ok {
c.WriteInt(0)
return
}
if t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
c.WriteInt(len(db.hashKeys[key]))
})
}
// HMGET
func (m *Miniredis) cmdHmget(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if t, ok := db.keys[key]; ok && t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
f, ok := db.hashKeys[key]
if !ok {
f = map[string]string{}
}
c.WriteLen(len(args) - 1)
for _, k := range args[1:] {
v, ok := f[k]
if !ok {
c.WriteNull()
continue
}
c.WriteBulk(v)
}
})
}
// HINCRBY
func (m *Miniredis) cmdHincrby(c *server.Peer, cmd string, args []string) {
if len(args) != 3 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts := struct {
key string
field string
delta int
}{
key: args[0],
field: args[1],
}
if ok := optInt(c, args[2], &opts.delta); !ok {
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if t, ok := db.keys[opts.key]; ok && t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
v, err := db.hashIncr(opts.key, opts.field, opts.delta)
if err != nil {
c.WriteError(err.Error())
return
}
c.WriteInt(v)
})
}
// HINCRBYFLOAT
func (m *Miniredis) cmdHincrbyfloat(c *server.Peer, cmd string, args []string) {
if len(args) != 3 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts := struct {
key string
field string
delta *big.Float
}{
key: args[0],
field: args[1],
}
delta, _, err := big.ParseFloat(args[2], 10, 128, 0)
if err != nil {
setDirty(c)
c.WriteError(msgInvalidFloat)
return
}
opts.delta = delta
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if t, ok := db.keys[opts.key]; ok && t != keyTypeHash {
c.WriteError(msgWrongType)
return
}
v, err := db.hashIncrfloat(opts.key, opts.field, opts.delta)
if err != nil {
c.WriteError(err.Error())
return
}
c.WriteBulk(formatBig(v))
})
}
// HSCAN
func (m *Miniredis) cmdHscan(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts := struct {
key string
cursor int
withMatch bool
match string
}{
key: args[0],
}
if ok := optIntErr(c, args[1], &opts.cursor, msgInvalidCursor); !ok {
return
}
args = args[2:]
// MATCH and COUNT options
for len(args) > 0 {
if strings.ToLower(args[0]) == "count" {
// we do nothing with count
if len(args) < 2 {
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
_, err := strconv.Atoi(args[1])
if err != nil {
setDirty(c)
c.WriteError(msgInvalidInt)
return
}
args = args[2:]
continue
}
if strings.ToLower(args[0]) == "match" {
if len(args) < 2 {
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
opts.withMatch = true
opts.match, args = args[1], args[2:]
continue
}
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
// return _all_ (matched) keys every time
if opts.cursor != 0 {
// Invalid cursor.
c.WriteLen(2)
c.WriteBulk("0") // no next cursor
c.WriteLen(0) // no elements
return
}
if db.exists(opts.key) && db.t(opts.key) != keyTypeHash {
c.WriteError(ErrWrongType.Error())
return
}
members := db.hashFields(opts.key)
if opts.withMatch {
members, _ = matchKeys(members, opts.match)
}
c.WriteLen(2)
c.WriteBulk("0") // no next cursor
// HSCAN gives key, values.
c.WriteLen(len(members) * 2)
for _, k := range members {
c.WriteBulk(k)
c.WriteBulk(db.hashGet(opts.key, k))
}
})
}
// HRANDFIELD
func (m *Miniredis) cmdHrandfield(c *server.Peer, cmd string, args []string) {
if len(args) > 3 || len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts := struct {
key string
count int
countSet bool
withValues bool
}{
key: args[0],
}
if len(args) > 1 {
if ok := optIntErr(c, args[1], &opts.count, msgInvalidInt); !ok {
return
}
opts.countSet = true
}
if len(args) == 3 {
if strings.ToLower(args[2]) == "withvalues" {
opts.withValues = true
} else {
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
}
withTx(m, c, func(peer *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
members := db.hashFields(opts.key)
m.shuffle(members)
if !opts.countSet {
// > When called with just the key argument, return a random field from the
// hash value stored at key.
if len(members) == 0 {
peer.WriteNull()
return
}
peer.WriteBulk(members[0])
return
}
if len(members) > abs(opts.count) {
members = members[:abs(opts.count)]
}
switch {
case opts.count >= 0:
// if count is positive there can't be duplicates, and the length is restricted
case opts.count < 0:
// if count is negative there can be duplicates, but length will match
if len(members) > 0 {
for len(members) < -opts.count {
members = append(members, members[m.randIntn(len(members))])
}
}
}
if opts.withValues {
peer.WriteMapLen(len(members))
for _, m := range members {
peer.WriteBulk(m)
peer.WriteBulk(db.hashGet(opts.key, m))
}
return
}
peer.WriteLen(len(members))
for _, m := range members {
peer.WriteBulk(m)
}
})
}
func abs(n int) int {
if n < 0 {
return -n
}
return n
}
+95
View File
@@ -0,0 +1,95 @@
package miniredis
import "github.com/alicebob/miniredis/v2/server"
// commandsHll handles all hll related operations.
func commandsHll(m *Miniredis) {
m.srv.Register("PFADD", m.cmdPfadd)
m.srv.Register("PFCOUNT", m.cmdPfcount)
m.srv.Register("PFMERGE", m.cmdPfmerge)
}
// PFADD
func (m *Miniredis) cmdPfadd(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key, items := args[0], args[1:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if db.exists(key) && db.t(key) != keyTypeHll {
c.WriteError(ErrNotValidHllValue.Error())
return
}
altered := db.hllAdd(key, items...)
c.WriteInt(altered)
})
}
// PFCOUNT
func (m *Miniredis) cmdPfcount(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
keys := args
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
count, err := db.hllCount(keys)
if err != nil {
c.WriteError(err.Error())
return
}
c.WriteInt(count)
})
}
// PFMERGE
func (m *Miniredis) cmdPfmerge(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
keys := args
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if err := db.hllMerge(keys); err != nil {
c.WriteError(err.Error())
return
}
c.WriteOK()
})
}
+40
View File
@@ -0,0 +1,40 @@
package miniredis
import (
"fmt"
"github.com/alicebob/miniredis/v2/server"
)
// Command 'INFO' from https://redis.io/commands/info/
func (m *Miniredis) cmdInfo(c *server.Peer, cmd string, args []string) {
if !m.isValidCMD(c, cmd) {
return
}
if len(args) > 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
const (
clientsSectionName = "clients"
clientsSectionContent = "# Clients\nconnected_clients:%d\r\n"
)
var result string
for _, key := range args {
if key != clientsSectionName {
setDirty(c)
c.WriteError(fmt.Sprintf("section (%s) is not supported", key))
return
}
}
result = fmt.Sprintf(clientsSectionContent, m.Server().ClientsLen())
c.WriteBulk(result)
})
}
File diff suppressed because it is too large Load Diff
+58
View File
@@ -0,0 +1,58 @@
package miniredis
import (
"fmt"
"strings"
"github.com/alicebob/miniredis/v2/server"
)
// commandsObject handles all object operations.
func commandsObject(m *Miniredis) {
m.srv.Register("OBJECT", m.cmdObject)
}
// OBJECT
func (m *Miniredis) cmdObject(c *server.Peer, cmd string, args []string) {
if len(args) == 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
switch sub := strings.ToLower(args[0]); sub {
case "idletime":
m.cmdObjectIdletime(c, args[1:])
default:
setDirty(c)
c.WriteError(fmt.Sprintf(msgFObjectUsage, sub))
}
}
// OBJECT IDLETIME
func (m *Miniredis) cmdObjectIdletime(c *server.Peer, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber("object|idletime"))
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
t, ok := db.lru[key]
if !ok {
c.WriteNull()
return
}
c.WriteInt(int(db.master.effectiveNow().Sub(t).Seconds()))
})
}
+262
View File
@@ -0,0 +1,262 @@
// Commands from https://redis.io/commands#pubsub
package miniredis
import (
"fmt"
"strings"
"github.com/alicebob/miniredis/v2/server"
)
// commandsPubsub handles all PUB/SUB operations.
func commandsPubsub(m *Miniredis) {
m.srv.Register("SUBSCRIBE", m.cmdSubscribe)
m.srv.Register("UNSUBSCRIBE", m.cmdUnsubscribe)
m.srv.Register("PSUBSCRIBE", m.cmdPsubscribe)
m.srv.Register("PUNSUBSCRIBE", m.cmdPunsubscribe)
m.srv.Register("PUBLISH", m.cmdPublish)
m.srv.Register("PUBSUB", m.cmdPubSub)
}
// SUBSCRIBE
func (m *Miniredis) cmdSubscribe(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
ctx := getCtx(c)
if ctx.nested {
c.WriteError(msgNotFromScripts(ctx.nestedSHA))
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
sub := m.subscribedState(c)
for _, channel := range args {
n := sub.Subscribe(channel)
c.Block(func(w *server.Writer) {
w.WritePushLen(3)
w.WriteBulk("subscribe")
w.WriteBulk(channel)
w.WriteInt(n)
})
}
})
}
// UNSUBSCRIBE
func (m *Miniredis) cmdUnsubscribe(c *server.Peer, cmd string, args []string) {
if !m.handleAuth(c) {
return
}
ctx := getCtx(c)
if ctx.nested {
c.WriteError(msgNotFromScripts(ctx.nestedSHA))
return
}
channels := args
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
sub := m.subscribedState(c)
if len(channels) == 0 {
channels = sub.Channels()
}
// there is no de-duplication
for _, channel := range channels {
n := sub.Unsubscribe(channel)
c.Block(func(w *server.Writer) {
w.WritePushLen(3)
w.WriteBulk("unsubscribe")
w.WriteBulk(channel)
w.WriteInt(n)
})
}
if len(channels) == 0 {
// special case: there is always a reply
c.Block(func(w *server.Writer) {
w.WritePushLen(3)
w.WriteBulk("unsubscribe")
w.WriteNull()
w.WriteInt(0)
})
}
if sub.Count() == 0 {
endSubscriber(m, c)
}
})
}
// PSUBSCRIBE
func (m *Miniredis) cmdPsubscribe(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
ctx := getCtx(c)
if ctx.nested {
c.WriteError(msgNotFromScripts(ctx.nestedSHA))
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
sub := m.subscribedState(c)
for _, pat := range args {
n := sub.Psubscribe(pat)
c.Block(func(w *server.Writer) {
w.WritePushLen(3)
w.WriteBulk("psubscribe")
w.WriteBulk(pat)
w.WriteInt(n)
})
}
})
}
// PUNSUBSCRIBE
func (m *Miniredis) cmdPunsubscribe(c *server.Peer, cmd string, args []string) {
if !m.handleAuth(c) {
return
}
ctx := getCtx(c)
if ctx.nested {
c.WriteError(msgNotFromScripts(ctx.nestedSHA))
return
}
patterns := args
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
sub := m.subscribedState(c)
if len(patterns) == 0 {
patterns = sub.Patterns()
}
// there is no de-duplication
for _, pat := range patterns {
n := sub.Punsubscribe(pat)
c.Block(func(w *server.Writer) {
w.WritePushLen(3)
w.WriteBulk("punsubscribe")
w.WriteBulk(pat)
w.WriteInt(n)
})
}
if len(patterns) == 0 {
// special case: there is always a reply
c.Block(func(w *server.Writer) {
w.WritePushLen(3)
w.WriteBulk("punsubscribe")
w.WriteNull()
w.WriteInt(0)
})
}
if sub.Count() == 0 {
endSubscriber(m, c)
}
})
}
// PUBLISH
func (m *Miniredis) cmdPublish(c *server.Peer, cmd string, args []string) {
if len(args) != 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
channel, mesg := args[0], args[1]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
c.WriteInt(m.publish(channel, mesg))
})
}
// PUBSUB
func (m *Miniredis) cmdPubSub(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if m.checkPubsub(c, cmd) {
return
}
subcommand := strings.ToUpper(args[0])
subargs := args[1:]
var argsOk bool
switch subcommand {
case "CHANNELS":
argsOk = len(subargs) < 2
case "NUMSUB":
argsOk = true
case "NUMPAT":
argsOk = len(subargs) == 0
default:
setDirty(c)
c.WriteError(fmt.Sprintf(msgFPubsubUsageSimple, subcommand))
return
}
if !argsOk {
setDirty(c)
c.WriteError(fmt.Sprintf(msgFPubsubUsage, subcommand))
return
}
if !m.handleAuth(c) {
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
switch subcommand {
case "CHANNELS":
pat := ""
if len(subargs) == 1 {
pat = subargs[0]
}
allsubs := m.allSubscribers()
channels := activeChannels(allsubs, pat)
c.WriteLen(len(channels))
for _, channel := range channels {
c.WriteBulk(channel)
}
case "NUMSUB":
subs := m.allSubscribers()
c.WriteLen(len(subargs) * 2)
for _, channel := range subargs {
c.WriteBulk(channel)
c.WriteInt(countSubs(subs, channel))
}
case "NUMPAT":
c.WriteInt(countPsubs(m.allSubscribers()))
}
})
}
+343
View File
@@ -0,0 +1,343 @@
package miniredis
import (
"crypto/sha1"
"encoding/hex"
"fmt"
"io"
"strconv"
"strings"
"sync"
lua "github.com/yuin/gopher-lua"
"github.com/yuin/gopher-lua/parse"
luajson "github.com/alicebob/miniredis/v2/gopher-json"
"github.com/alicebob/miniredis/v2/server"
)
func commandsScripting(m *Miniredis) {
m.srv.Register("EVAL", m.cmdEval)
m.srv.Register("EVALSHA", m.cmdEvalsha)
m.srv.Register("SCRIPT", m.cmdScript)
}
var (
parsedScripts = sync.Map{}
)
// Execute lua. Needs to run m.Lock()ed, from within withTx().
// Returns true if the lua was OK (and hence should be cached).
func (m *Miniredis) runLuaScript(c *server.Peer, sha, script string, args []string) bool {
l := lua.NewState(lua.Options{SkipOpenLibs: true})
defer l.Close()
// Taken from the go-lua manual
for _, pair := range []struct {
n string
f lua.LGFunction
}{
{lua.LoadLibName, lua.OpenPackage},
{lua.BaseLibName, lua.OpenBase},
{lua.CoroutineLibName, lua.OpenCoroutine},
{lua.TabLibName, lua.OpenTable},
{lua.StringLibName, lua.OpenString},
{lua.MathLibName, lua.OpenMath},
{lua.DebugLibName, lua.OpenDebug},
} {
if err := l.CallByParam(lua.P{
Fn: l.NewFunction(pair.f),
NRet: 0,
Protect: true,
}, lua.LString(pair.n)); err != nil {
panic(err)
}
}
luajson.Preload(l)
requireGlobal(l, "cjson", "json")
// set global variable KEYS
keysTable := l.NewTable()
keysS, args := args[0], args[1:]
keysLen, err := strconv.Atoi(keysS)
if err != nil {
c.WriteError(msgInvalidInt)
return false
}
if keysLen < 0 {
c.WriteError(msgNegativeKeysNumber)
return false
}
if keysLen > len(args) {
c.WriteError(msgInvalidKeysNumber)
return false
}
keys, args := args[:keysLen], args[keysLen:]
for i, k := range keys {
l.RawSet(keysTable, lua.LNumber(i+1), lua.LString(k))
}
l.SetGlobal("KEYS", keysTable)
argvTable := l.NewTable()
for i, a := range args {
l.RawSet(argvTable, lua.LNumber(i+1), lua.LString(a))
}
l.SetGlobal("ARGV", argvTable)
redisFuncs, redisConstants := mkLua(m.srv, c, sha)
// Register command handlers
l.Push(l.NewFunction(func(l *lua.LState) int {
mod := l.RegisterModule("redis", redisFuncs).(*lua.LTable)
for k, v := range redisConstants {
mod.RawSetString(k, v)
}
l.Push(mod)
return 1
}))
_ = doScript(l, protectGlobals)
l.Push(lua.LString("redis"))
l.Call(1, 0)
// lua can call redis.setresp(...), but it's tmp state.
oldresp := c.Resp3
if err := doScript(l, script); err != nil {
c.WriteError(err.Error())
return false
}
luaToRedis(l, c, l.Get(1))
c.Resp3 = oldresp
c.SwitchResp3 = nil
return true
}
// doScript pre-compiles the given script into a Lua prototype,
// then executes the pre-compiled function against the given lua state.
//
// This is thread-safe.
func doScript(l *lua.LState, script string) error {
proto, err := compile(script)
if err != nil {
return fmt.Errorf(errLuaParseError(err))
}
lfunc := l.NewFunctionFromProto(proto)
l.Push(lfunc)
if err := l.PCall(0, lua.MultRet, nil); err != nil {
// ensure we wrap with the correct format.
return fmt.Errorf(errLuaParseError(err))
}
return nil
}
func compile(script string) (*lua.FunctionProto, error) {
if val, ok := parsedScripts.Load(script); ok {
return val.(*lua.FunctionProto), nil
}
chunk, err := parse.Parse(strings.NewReader(script), "<string>")
if err != nil {
return nil, err
}
proto, err := lua.Compile(chunk, "")
if err != nil {
return nil, err
}
parsedScripts.Store(script, proto)
return proto, nil
}
func (m *Miniredis) cmdEval(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
ctx := getCtx(c)
if ctx.nested {
c.WriteError(msgNotFromScripts(ctx.nestedSHA))
return
}
script, args := args[0], args[1:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
sha := sha1Hex(script)
ok := m.runLuaScript(c, sha, script, args)
if ok {
m.scripts[sha] = script
}
})
}
func (m *Miniredis) cmdEvalsha(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
ctx := getCtx(c)
if ctx.nested {
c.WriteError(msgNotFromScripts(ctx.nestedSHA))
return
}
sha, args := args[0], args[1:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
script, ok := m.scripts[sha]
if !ok {
c.WriteError(msgNoScriptFound)
return
}
m.runLuaScript(c, sha, script, args)
})
}
func (m *Miniredis) cmdScript(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
ctx := getCtx(c)
if ctx.nested {
c.WriteError(msgNotFromScripts(ctx.nestedSHA))
return
}
var opts struct {
subcmd string
script string
}
opts.subcmd, args = args[0], args[1:]
switch strings.ToLower(opts.subcmd) {
case "load":
if len(args) != 1 {
setDirty(c)
c.WriteError(fmt.Sprintf(msgFScriptUsage, "LOAD"))
return
}
opts.script = args[0]
case "exists":
if len(args) == 0 {
setDirty(c)
c.WriteError(errWrongNumber("script|exists"))
return
}
case "flush":
if len(args) == 1 {
switch strings.ToUpper(args[0]) {
case "SYNC", "ASYNC":
args = args[1:]
default:
}
}
if len(args) != 0 {
setDirty(c)
c.WriteError(msgScriptFlush)
return
}
default:
setDirty(c)
c.WriteError(fmt.Sprintf(msgFScriptUsageSimple, strings.ToUpper(opts.subcmd)))
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
switch strings.ToLower(opts.subcmd) {
case "load":
if _, err := parse.Parse(strings.NewReader(opts.script), "user_script"); err != nil {
c.WriteError(errLuaParseError(err))
return
}
sha := sha1Hex(opts.script)
m.scripts[sha] = opts.script
c.WriteBulk(sha)
case "exists":
c.WriteLen(len(args))
for _, arg := range args {
if _, ok := m.scripts[arg]; ok {
c.WriteInt(1)
} else {
c.WriteInt(0)
}
}
case "flush":
m.scripts = map[string]string{}
c.WriteOK()
}
})
}
func sha1Hex(s string) string {
h := sha1.New()
io.WriteString(h, s)
return hex.EncodeToString(h.Sum(nil))
}
// requireGlobal imports module modName into the global namespace with the
// identifier id. panics if an error results from the function execution
func requireGlobal(l *lua.LState, id, modName string) {
if err := l.CallByParam(lua.P{
Fn: l.GetGlobal("require"),
NRet: 1,
Protect: true,
}, lua.LString(modName)); err != nil {
panic(err)
}
mod := l.Get(-1)
l.Pop(1)
l.SetGlobal(id, mod)
}
// the following script protects globals
// it is based on: http://metalua.luaforge.net/src/lib/strict.lua.html
var protectGlobals = `
local dbg=debug
local mt = {}
setmetatable(_G, mt)
mt.__newindex = function (t, n, v)
if dbg.getinfo(2) then
local w = dbg.getinfo(2, "S").what
if w ~= "C" then
error("Script attempted to create global variable '"..tostring(n).."'", 2)
end
end
rawset(t, n, v)
end
mt.__index = function (t, n)
if dbg.getinfo(2) and dbg.getinfo(2, "S").what ~= "C" then
error("Script attempted to access nonexistent global variable '"..tostring(n).."'", 2)
end
return rawget(t, n)
end
debug = nil
`
+177
View File
@@ -0,0 +1,177 @@
// Commands from https://redis.io/commands#server
package miniredis
import (
"fmt"
"strconv"
"strings"
"github.com/alicebob/miniredis/v2/server"
"github.com/alicebob/miniredis/v2/size"
)
func commandsServer(m *Miniredis) {
m.srv.Register("COMMAND", m.cmdCommand)
m.srv.Register("DBSIZE", m.cmdDbsize)
m.srv.Register("FLUSHALL", m.cmdFlushall)
m.srv.Register("FLUSHDB", m.cmdFlushdb)
m.srv.Register("INFO", m.cmdInfo)
m.srv.Register("TIME", m.cmdTime)
m.srv.Register("MEMORY", m.cmdMemory)
}
// MEMORY
func (m *Miniredis) cmdMemory(c *server.Peer, cmd string, args []string) {
if len(args) == 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
cmd, args := strings.ToLower(args[0]), args[1:]
switch cmd {
case "usage":
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber("memory|usage"))
return
}
if len(args) > 1 {
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
var (
value interface{}
ok bool
)
switch db.keys[args[0]] {
case keyTypeString:
value, ok = db.stringKeys[args[0]]
case keyTypeSet:
value, ok = db.setKeys[args[0]]
case keyTypeHash:
value, ok = db.hashKeys[args[0]]
case keyTypeList:
value, ok = db.listKeys[args[0]]
case keyTypeHll:
value, ok = db.hllKeys[args[0]]
case keyTypeSortedSet:
value, ok = db.sortedsetKeys[args[0]]
case keyTypeStream:
value, ok = db.streamKeys[args[0]]
}
if !ok {
c.WriteNull()
return
}
c.WriteInt(size.Of(value))
default:
c.WriteError(fmt.Sprintf(msgMemorySubcommand, strings.ToUpper(cmd)))
}
})
}
// DBSIZE
func (m *Miniredis) cmdDbsize(c *server.Peer, cmd string, args []string) {
if len(args) > 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
c.WriteInt(len(db.keys))
})
}
// FLUSHALL
func (m *Miniredis) cmdFlushall(c *server.Peer, cmd string, args []string) {
if len(args) > 0 && strings.ToLower(args[0]) == "async" {
args = args[1:]
}
if len(args) > 0 {
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
m.flushAll()
c.WriteOK()
})
}
// FLUSHDB
func (m *Miniredis) cmdFlushdb(c *server.Peer, cmd string, args []string) {
if len(args) > 0 && strings.ToLower(args[0]) == "async" {
args = args[1:]
}
if len(args) > 0 {
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
m.db(ctx.selectedDB).flush()
c.WriteOK()
})
}
// TIME
func (m *Miniredis) cmdTime(c *server.Peer, cmd string, args []string) {
if len(args) > 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
now := m.effectiveNow()
nanos := now.UnixNano()
seconds := nanos / 1_000_000_000
microseconds := (nanos / 1_000) % 1_000_000
c.WriteLen(2)
c.WriteBulk(strconv.FormatInt(seconds, 10))
c.WriteBulk(strconv.FormatInt(microseconds, 10))
})
}
+836
View File
@@ -0,0 +1,836 @@
// Commands from https://redis.io/commands#set
package miniredis
import (
"fmt"
"strconv"
"strings"
"github.com/alicebob/miniredis/v2/server"
)
// commandsSet handles all set value operations.
func commandsSet(m *Miniredis) {
m.srv.Register("SADD", m.cmdSadd)
m.srv.Register("SCARD", m.cmdScard)
m.srv.Register("SDIFF", m.cmdSdiff)
m.srv.Register("SDIFFSTORE", m.cmdSdiffstore)
m.srv.Register("SINTERCARD", m.cmdSintercard)
m.srv.Register("SINTER", m.cmdSinter)
m.srv.Register("SINTERSTORE", m.cmdSinterstore)
m.srv.Register("SISMEMBER", m.cmdSismember)
m.srv.Register("SMEMBERS", m.cmdSmembers)
m.srv.Register("SMISMEMBER", m.cmdSmismember)
m.srv.Register("SMOVE", m.cmdSmove)
m.srv.Register("SPOP", m.cmdSpop)
m.srv.Register("SRANDMEMBER", m.cmdSrandmember)
m.srv.Register("SREM", m.cmdSrem)
m.srv.Register("SUNION", m.cmdSunion)
m.srv.Register("SUNIONSTORE", m.cmdSunionstore)
m.srv.Register("SSCAN", m.cmdSscan)
}
// SADD
func (m *Miniredis) cmdSadd(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key, elems := args[0], args[1:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if db.exists(key) && db.t(key) != keyTypeSet {
c.WriteError(ErrWrongType.Error())
return
}
added := db.setAdd(key, elems...)
c.WriteInt(added)
})
}
// SCARD
func (m *Miniredis) cmdScard(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if !db.exists(key) {
c.WriteInt(0)
return
}
if db.t(key) != "set" {
c.WriteError(ErrWrongType.Error())
return
}
members := db.setMembers(key)
c.WriteInt(len(members))
})
}
// SDIFF
func (m *Miniredis) cmdSdiff(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
keys := args
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
set, err := db.setDiff(keys)
if err != nil {
c.WriteError(err.Error())
return
}
c.WriteSetLen(len(set))
for k := range set {
c.WriteBulk(k)
}
})
}
// SDIFFSTORE
func (m *Miniredis) cmdSdiffstore(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
dest, keys := args[0], args[1:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
set, err := db.setDiff(keys)
if err != nil {
c.WriteError(err.Error())
return
}
db.del(dest, true)
db.setSet(dest, set)
c.WriteInt(len(set))
})
}
// SINTER
func (m *Miniredis) cmdSinter(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
keys := args
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
set, err := db.setInter(keys)
if err != nil {
c.WriteError(err.Error())
return
}
c.WriteLen(len(set))
for k := range set {
c.WriteBulk(k)
}
})
}
// SINTERSTORE
func (m *Miniredis) cmdSinterstore(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
dest, keys := args[0], args[1:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
set, err := db.setInter(keys)
if err != nil {
c.WriteError(err.Error())
return
}
db.del(dest, true)
db.setSet(dest, set)
c.WriteInt(len(set))
})
}
// SINTERCARD
func (m *Miniredis) cmdSintercard(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts := struct {
keys []string
limit int
}{}
numKeys, err := strconv.Atoi(args[0])
if err != nil {
setDirty(c)
c.WriteError("ERR numkeys should be greater than 0")
return
}
if numKeys < 1 {
setDirty(c)
c.WriteError("ERR numkeys should be greater than 0")
return
}
args = args[1:]
if len(args) < numKeys {
setDirty(c)
c.WriteError("ERR Number of keys can't be greater than number of args")
return
}
opts.keys = args[:numKeys]
args = args[numKeys:]
if len(args) == 2 && strings.ToLower(args[0]) == "limit" {
l, err := strconv.Atoi(args[1])
if err != nil {
setDirty(c)
c.WriteError(msgInvalidInt)
return
}
if l < 0 {
setDirty(c)
c.WriteError(msgLimitIsNegative)
return
}
opts.limit = l
} else if len(args) > 0 {
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
count, err := db.setIntercard(opts.keys, opts.limit)
if err != nil {
c.WriteError(err.Error())
return
}
c.WriteInt(count)
})
}
// SISMEMBER
func (m *Miniredis) cmdSismember(c *server.Peer, cmd string, args []string) {
if len(args) != 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key, value := args[0], args[1]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if !db.exists(key) {
c.WriteInt(0)
return
}
if db.t(key) != "set" {
c.WriteError(ErrWrongType.Error())
return
}
if db.setIsMember(key, value) {
c.WriteInt(1)
return
}
c.WriteInt(0)
})
}
// SMEMBERS
func (m *Miniredis) cmdSmembers(c *server.Peer, cmd string, args []string) {
if len(args) != 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if !db.exists(key) {
c.WriteSetLen(0)
return
}
if db.t(key) != "set" {
c.WriteError(ErrWrongType.Error())
return
}
members := db.setMembers(key)
c.WriteSetLen(len(members))
for _, elem := range members {
c.WriteBulk(elem)
}
})
}
// SMISMEMBER
func (m *Miniredis) cmdSmismember(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key, values := args[0], args[1:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if !db.exists(key) {
c.WriteLen(len(values))
for range values {
c.WriteInt(0)
}
return
}
if db.t(key) != "set" {
c.WriteError(ErrWrongType.Error())
return
}
c.WriteLen(len(values))
for _, value := range values {
if db.setIsMember(key, value) {
c.WriteInt(1)
} else {
c.WriteInt(0)
}
}
return
})
}
// SMOVE
func (m *Miniredis) cmdSmove(c *server.Peer, cmd string, args []string) {
if len(args) != 3 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
src, dst, member := args[0], args[1], args[2]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if !db.exists(src) {
c.WriteInt(0)
return
}
if db.t(src) != "set" {
c.WriteError(ErrWrongType.Error())
return
}
if db.exists(dst) && db.t(dst) != "set" {
c.WriteError(ErrWrongType.Error())
return
}
if !db.setIsMember(src, member) {
c.WriteInt(0)
return
}
db.setRem(src, member)
db.setAdd(dst, member)
c.WriteInt(1)
})
}
// SPOP
func (m *Miniredis) cmdSpop(c *server.Peer, cmd string, args []string) {
if len(args) == 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
opts := struct {
key string
withCount bool
count int
}{
count: 1,
}
opts.key, args = args[0], args[1:]
if len(args) > 0 {
v, err := strconv.Atoi(args[0])
if err != nil {
setDirty(c)
c.WriteError(msgInvalidInt)
return
}
if v < 0 {
setDirty(c)
c.WriteError(msgOutOfRange)
return
}
opts.count = v
opts.withCount = true
args = args[1:]
}
if len(args) > 0 {
setDirty(c)
c.WriteError(msgInvalidInt)
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if !db.exists(opts.key) {
if !opts.withCount {
c.WriteNull()
return
}
c.WriteLen(0)
return
}
if db.t(opts.key) != "set" {
c.WriteError(ErrWrongType.Error())
return
}
var deleted []string
members := db.setMembers(opts.key)
for i := 0; i < opts.count; i++ {
if len(members) == 0 {
break
}
i := m.randIntn(len(members))
member := members[i]
members = delElem(members, i)
db.setRem(opts.key, member)
deleted = append(deleted, member)
}
// without `count` return a single value
if !opts.withCount {
if len(deleted) == 0 {
c.WriteNull()
return
}
c.WriteBulk(deleted[0])
return
}
// with `count` return a list
c.WriteLen(len(deleted))
for _, v := range deleted {
c.WriteBulk(v)
}
})
}
// SRANDMEMBER
func (m *Miniredis) cmdSrandmember(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if len(args) > 2 {
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key := args[0]
count := 0
withCount := false
if len(args) == 2 {
var err error
count, err = strconv.Atoi(args[1])
if err != nil {
setDirty(c)
c.WriteError(msgInvalidInt)
return
}
withCount = true
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if !db.exists(key) {
if withCount {
c.WriteLen(0)
return
}
c.WriteNull()
return
}
if db.t(key) != "set" {
c.WriteError(ErrWrongType.Error())
return
}
members := db.setMembers(key)
if count < 0 {
// Non-unique elements is allowed with negative count.
c.WriteLen(-count)
for count != 0 {
member := members[m.randIntn(len(members))]
c.WriteBulk(member)
count++
}
return
}
// Must be unique elements.
m.shuffle(members)
if count > len(members) {
count = len(members)
}
if !withCount {
c.WriteBulk(members[0])
return
}
c.WriteLen(count)
for i := range make([]struct{}, count) {
c.WriteBulk(members[i])
}
})
}
// SREM
func (m *Miniredis) cmdSrem(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
key, fields := args[0], args[1:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
if !db.exists(key) {
c.WriteInt(0)
return
}
if db.t(key) != "set" {
c.WriteError(ErrWrongType.Error())
return
}
c.WriteInt(db.setRem(key, fields...))
})
}
// SUNION
func (m *Miniredis) cmdSunion(c *server.Peer, cmd string, args []string) {
if len(args) < 1 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
keys := args
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
set, err := db.setUnion(keys)
if err != nil {
c.WriteError(err.Error())
return
}
c.WriteLen(len(set))
for k := range set {
c.WriteBulk(k)
}
})
}
// SUNIONSTORE
func (m *Miniredis) cmdSunionstore(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
dest, keys := args[0], args[1:]
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
set, err := db.setUnion(keys)
if err != nil {
c.WriteError(err.Error())
return
}
db.del(dest, true)
db.setSet(dest, set)
c.WriteInt(len(set))
})
}
// SSCAN
func (m *Miniredis) cmdSscan(c *server.Peer, cmd string, args []string) {
if len(args) < 2 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
var opts struct {
key string
value int
cursor int
count int
withMatch bool
match string
}
opts.key = args[0]
if ok := optIntErr(c, args[1], &opts.cursor, msgInvalidCursor); !ok {
return
}
args = args[2:]
// MATCH and COUNT options
for len(args) > 0 {
if strings.ToLower(args[0]) == "count" {
if len(args) < 2 {
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
count, err := strconv.Atoi(args[1])
if err != nil || count < 0 {
setDirty(c)
c.WriteError(msgInvalidInt)
return
}
if count == 0 {
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
opts.count = count
args = args[2:]
continue
}
if strings.ToLower(args[0]) == "match" {
if len(args) < 2 {
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
opts.withMatch = true
opts.match = args[1]
args = args[2:]
continue
}
setDirty(c)
c.WriteError(msgSyntaxError)
return
}
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
db := m.db(ctx.selectedDB)
// return _all_ (matched) keys every time
if db.exists(opts.key) && db.t(opts.key) != "set" {
c.WriteError(ErrWrongType.Error())
return
}
members := db.setMembers(opts.key)
if opts.withMatch {
members, _ = matchKeys(members, opts.match)
}
low := opts.cursor
high := low + opts.count
// validate high is correct
if high > len(members) || high == 0 {
high = len(members)
}
if opts.cursor > high {
// invalid cursor
c.WriteLen(2)
c.WriteBulk("0") // no next cursor
c.WriteLen(0) // no elements
return
}
cursorValue := low + opts.count
if cursorValue > len(members) {
cursorValue = 0 // no next cursor
}
members = members[low:high]
c.WriteLen(2)
c.WriteBulk(fmt.Sprintf("%d", cursorValue))
c.WriteLen(len(members))
for _, k := range members {
c.WriteBulk(k)
}
})
}
func delElem(ls []string, i int) []string {
// this swap+truncate is faster but changes behaviour:
// ls[i] = ls[len(ls)-1]
// ls = ls[:len(ls)-1]
// so we do the dumb thing:
ls = append(ls[:i], ls[i+1:]...)
return ls
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+179
View File
@@ -0,0 +1,179 @@
// Commands from https://redis.io/commands#transactions
package miniredis
import (
"github.com/alicebob/miniredis/v2/server"
)
// commandsTransaction handles MULTI &c.
func commandsTransaction(m *Miniredis) {
m.srv.Register("DISCARD", m.cmdDiscard)
m.srv.Register("EXEC", m.cmdExec)
m.srv.Register("MULTI", m.cmdMulti)
m.srv.Register("UNWATCH", m.cmdUnwatch)
m.srv.Register("WATCH", m.cmdWatch)
}
// MULTI
func (m *Miniredis) cmdMulti(c *server.Peer, cmd string, args []string) {
if len(args) != 0 {
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
ctx := getCtx(c)
if ctx.nested {
c.WriteError(msgNotFromScripts(ctx.nestedSHA))
return
}
if inTx(ctx) {
c.WriteError("ERR MULTI calls can not be nested")
return
}
startTx(ctx)
c.WriteOK()
}
// EXEC
func (m *Miniredis) cmdExec(c *server.Peer, cmd string, args []string) {
if len(args) != 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
ctx := getCtx(c)
if ctx.nested {
c.WriteError(msgNotFromScripts(ctx.nestedSHA))
return
}
if !inTx(ctx) {
c.WriteError("ERR EXEC without MULTI")
return
}
if ctx.dirtyTransaction {
c.WriteError("EXECABORT Transaction discarded because of previous errors.")
// a failed EXEC finishes the tx
stopTx(ctx)
return
}
m.Lock()
defer m.Unlock()
// Check WATCHed keys.
for t, version := range ctx.watch {
if m.db(t.db).keyVersion[t.key] > version {
// Abort! Abort!
stopTx(ctx)
c.WriteLen(-1)
return
}
}
c.WriteLen(len(ctx.transaction))
for _, cb := range ctx.transaction {
cb(c, ctx)
}
// wake up anyone who waits on anything.
m.signal.Broadcast()
stopTx(ctx)
}
// DISCARD
func (m *Miniredis) cmdDiscard(c *server.Peer, cmd string, args []string) {
if len(args) != 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
ctx := getCtx(c)
if !inTx(ctx) {
c.WriteError("ERR DISCARD without MULTI")
return
}
stopTx(ctx)
c.WriteOK()
}
// WATCH
func (m *Miniredis) cmdWatch(c *server.Peer, cmd string, args []string) {
if len(args) == 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
ctx := getCtx(c)
if ctx.nested {
c.WriteError(msgNotFromScripts(ctx.nestedSHA))
return
}
if inTx(ctx) {
c.WriteError("ERR WATCH in MULTI")
return
}
m.Lock()
defer m.Unlock()
db := m.db(ctx.selectedDB)
for _, key := range args {
watch(db, ctx, key)
}
c.WriteOK()
}
// UNWATCH
func (m *Miniredis) cmdUnwatch(c *server.Peer, cmd string, args []string) {
if len(args) != 0 {
setDirty(c)
c.WriteError(errWrongNumber(cmd))
return
}
if !m.handleAuth(c) {
return
}
if m.checkPubsub(c, cmd) {
return
}
// Doesn't matter if UNWATCH is in a TX or not. Looks like a Redis bug to me.
unwatch(getCtx(c))
withTx(m, c, func(c *server.Peer, ctx *connCtx) {
// Do nothing if it's called in a transaction.
c.WriteOK()
})
}
+790
View File
@@ -0,0 +1,790 @@
package miniredis
import (
"errors"
"fmt"
"math"
"math/big"
"sort"
"strconv"
"time"
)
var (
errInvalidEntryID = errors.New("stream ID is invalid")
)
// exists also updates the lru
func (db *RedisDB) exists(k string) bool {
_, ok := db.keys[k]
if ok {
db.lru[k] = db.master.effectiveNow()
}
return ok
}
// t gives the type of a key, or ""
func (db *RedisDB) t(k string) string {
return db.keys[k]
}
// incr increases the version and the lru timestamp
func (db *RedisDB) incr(k string) {
db.lru[k] = db.master.effectiveNow()
db.keyVersion[k]++
}
// allKeys returns all keys. Sorted.
func (db *RedisDB) allKeys() []string {
res := make([]string, 0, len(db.keys))
for k := range db.keys {
res = append(res, k)
}
sort.Strings(res) // To make things deterministic.
return res
}
// flush removes all keys and values.
func (db *RedisDB) flush() {
db.keys = map[string]string{}
db.lru = map[string]time.Time{}
db.stringKeys = map[string]string{}
db.hashKeys = map[string]hashKey{}
db.listKeys = map[string]listKey{}
db.setKeys = map[string]setKey{}
db.hllKeys = map[string]*hll{}
db.sortedsetKeys = map[string]sortedSet{}
db.ttl = map[string]time.Duration{}
db.streamKeys = map[string]*streamKey{}
}
// move something to another db. Will return ok. Or not.
func (db *RedisDB) move(key string, to *RedisDB) bool {
if _, ok := to.keys[key]; ok {
return false
}
t, ok := db.keys[key]
if !ok {
return false
}
to.keys[key] = db.keys[key]
switch t {
case keyTypeString:
to.stringKeys[key] = db.stringKeys[key]
case keyTypeHash:
to.hashKeys[key] = db.hashKeys[key]
case keyTypeList:
to.listKeys[key] = db.listKeys[key]
case keyTypeSet:
to.setKeys[key] = db.setKeys[key]
case keyTypeSortedSet:
to.sortedsetKeys[key] = db.sortedsetKeys[key]
case keyTypeStream:
to.streamKeys[key] = db.streamKeys[key]
case keyTypeHll:
to.hllKeys[key] = db.hllKeys[key]
default:
panic("unhandled key type")
}
if v, ok := db.ttl[key]; ok {
to.ttl[key] = v
}
to.incr(key)
db.del(key, true)
return true
}
func (db *RedisDB) rename(from, to string) {
db.del(to, true)
switch db.t(from) {
case keyTypeString:
db.stringKeys[to] = db.stringKeys[from]
case keyTypeHash:
db.hashKeys[to] = db.hashKeys[from]
case keyTypeList:
db.listKeys[to] = db.listKeys[from]
case keyTypeSet:
db.setKeys[to] = db.setKeys[from]
case keyTypeSortedSet:
db.sortedsetKeys[to] = db.sortedsetKeys[from]
case keyTypeStream:
db.streamKeys[to] = db.streamKeys[from]
case keyTypeHll:
db.hllKeys[to] = db.hllKeys[from]
default:
panic("missing case")
}
db.keys[to] = db.keys[from]
if v, ok := db.ttl[from]; ok {
db.ttl[to] = v
}
db.incr(to)
db.del(from, true)
}
func (db *RedisDB) del(k string, delTTL bool) {
if !db.exists(k) {
return
}
t := db.t(k)
delete(db.keys, k)
delete(db.lru, k)
db.keyVersion[k]++
if delTTL {
delete(db.ttl, k)
}
switch t {
case keyTypeString:
delete(db.stringKeys, k)
case keyTypeHash:
delete(db.hashKeys, k)
case keyTypeList:
delete(db.listKeys, k)
case keyTypeSet:
delete(db.setKeys, k)
case keyTypeSortedSet:
delete(db.sortedsetKeys, k)
case keyTypeStream:
delete(db.streamKeys, k)
case keyTypeHll:
delete(db.hllKeys, k)
default:
panic("Unknown key type: " + t)
}
}
// stringGet returns the string key or "" on error/nonexists.
func (db *RedisDB) stringGet(k string) string {
if t, ok := db.keys[k]; !ok || t != keyTypeString {
return ""
}
return db.stringKeys[k]
}
// stringSet force set()s a key. Does not touch expire.
func (db *RedisDB) stringSet(k, v string) {
db.del(k, false)
db.keys[k] = keyTypeString
db.stringKeys[k] = v
db.incr(k)
}
// change int key value
func (db *RedisDB) stringIncr(k string, delta int) (int, error) {
v := 0
if sv, ok := db.stringKeys[k]; ok {
var err error
v, err = strconv.Atoi(sv)
if err != nil {
return 0, ErrIntValueError
}
}
if delta > 0 {
if math.MaxInt-delta < v {
return 0, ErrIntValueOverflowError
}
} else {
if math.MinInt-delta > v {
return 0, ErrIntValueOverflowError
}
}
v += delta
db.stringSet(k, strconv.Itoa(v))
return v, nil
}
// change float key value
func (db *RedisDB) stringIncrfloat(k string, delta *big.Float) (*big.Float, error) {
v := big.NewFloat(0.0)
v.SetPrec(128)
if sv, ok := db.stringKeys[k]; ok {
var err error
v, _, err = big.ParseFloat(sv, 10, 128, 0)
if err != nil {
return nil, ErrFloatValueError
}
}
v.Add(v, delta)
db.stringSet(k, formatBig(v))
return v, nil
}
// listLpush is 'left push', aka unshift. Returns the new length.
func (db *RedisDB) listLpush(k, v string) int {
l, ok := db.listKeys[k]
if !ok {
db.keys[k] = keyTypeList
}
l = append([]string{v}, l...)
db.listKeys[k] = l
db.incr(k)
return len(l)
}
// 'left pop', aka shift.
func (db *RedisDB) listLpop(k string) string {
l := db.listKeys[k]
el := l[0]
l = l[1:]
if len(l) == 0 {
db.del(k, true)
} else {
db.listKeys[k] = l
}
db.incr(k)
return el
}
func (db *RedisDB) listPush(k string, v ...string) int {
l, ok := db.listKeys[k]
if !ok {
db.keys[k] = keyTypeList
}
l = append(l, v...)
db.listKeys[k] = l
db.incr(k)
return len(l)
}
func (db *RedisDB) listPop(k string) string {
l := db.listKeys[k]
el := l[len(l)-1]
l = l[:len(l)-1]
if len(l) == 0 {
db.del(k, true)
} else {
db.listKeys[k] = l
db.incr(k)
}
return el
}
// setset replaces a whole set.
func (db *RedisDB) setSet(k string, set setKey) {
db.keys[k] = keyTypeSet
db.setKeys[k] = set
db.incr(k)
}
// setadd adds members to a set. Returns nr of new keys.
func (db *RedisDB) setAdd(k string, elems ...string) int {
s, ok := db.setKeys[k]
if !ok {
s = setKey{}
db.keys[k] = keyTypeSet
}
added := 0
for _, e := range elems {
if _, ok := s[e]; !ok {
added++
}
s[e] = struct{}{}
}
db.setKeys[k] = s
db.incr(k)
return added
}
// setrem removes members from a set. Returns nr of deleted keys.
func (db *RedisDB) setRem(k string, fields ...string) int {
s, ok := db.setKeys[k]
if !ok {
return 0
}
removed := 0
for _, f := range fields {
if _, ok := s[f]; ok {
removed++
delete(s, f)
}
}
if len(s) == 0 {
db.del(k, true)
} else {
db.setKeys[k] = s
}
db.incr(k)
return removed
}
// All members of a set.
func (db *RedisDB) setMembers(k string) []string {
set := db.setKeys[k]
members := make([]string, 0, len(set))
for k := range set {
members = append(members, k)
}
sort.Strings(members)
return members
}
// Is a SET value present?
func (db *RedisDB) setIsMember(k, v string) bool {
set, ok := db.setKeys[k]
if !ok {
return false
}
_, ok = set[v]
return ok
}
// hashFields returns all (sorted) keys ('fields') for a hash key.
func (db *RedisDB) hashFields(k string) []string {
v := db.hashKeys[k]
var r []string
for k := range v {
r = append(r, k)
}
sort.Strings(r)
return r
}
// hashValues returns all (sorted) values a hash key.
func (db *RedisDB) hashValues(k string) []string {
h := db.hashKeys[k]
var r []string
for _, v := range h {
r = append(r, v)
}
sort.Strings(r)
return r
}
// hashGet a value
func (db *RedisDB) hashGet(key, field string) string {
return db.hashKeys[key][field]
}
// hashSet returns the number of new keys
func (db *RedisDB) hashSet(k string, fv ...string) int {
if t, ok := db.keys[k]; ok && t != keyTypeHash {
db.del(k, true)
}
db.keys[k] = keyTypeHash
if _, ok := db.hashKeys[k]; !ok {
db.hashKeys[k] = map[string]string{}
}
new := 0
for idx := 0; idx < len(fv)-1; idx = idx + 2 {
f, v := fv[idx], fv[idx+1]
_, ok := db.hashKeys[k][f]
db.hashKeys[k][f] = v
db.incr(k)
if !ok {
new++
}
}
return new
}
// hashIncr changes int key value
func (db *RedisDB) hashIncr(key, field string, delta int) (int, error) {
v := 0
if h, ok := db.hashKeys[key]; ok {
if f, ok := h[field]; ok {
var err error
v, err = strconv.Atoi(f)
if err != nil {
return 0, ErrIntValueError
}
}
}
v += delta
db.hashSet(key, field, strconv.Itoa(v))
return v, nil
}
// hashIncrfloat changes float key value
func (db *RedisDB) hashIncrfloat(key, field string, delta *big.Float) (*big.Float, error) {
v := big.NewFloat(0.0)
v.SetPrec(128)
if h, ok := db.hashKeys[key]; ok {
if f, ok := h[field]; ok {
var err error
v, _, err = big.ParseFloat(f, 10, 128, 0)
if err != nil {
return nil, ErrFloatValueError
}
}
}
v.Add(v, delta)
db.hashSet(key, field, formatBig(v))
return v, nil
}
// sortedSet set returns a sortedSet as map
func (db *RedisDB) sortedSet(key string) map[string]float64 {
ss := db.sortedsetKeys[key]
return map[string]float64(ss)
}
// ssetSet sets a complete sorted set.
func (db *RedisDB) ssetSet(key string, sset sortedSet) {
db.keys[key] = keyTypeSortedSet
db.incr(key)
db.sortedsetKeys[key] = sset
}
// ssetAdd adds member to a sorted set. Returns whether this was a new member.
func (db *RedisDB) ssetAdd(key string, score float64, member string) bool {
ss, ok := db.sortedsetKeys[key]
if !ok {
ss = newSortedSet()
db.keys[key] = keyTypeSortedSet
}
_, ok = ss[member]
ss[member] = score
db.sortedsetKeys[key] = ss
db.incr(key)
return !ok
}
// All members from a sorted set, ordered by score.
func (db *RedisDB) ssetMembers(key string) []string {
ss, ok := db.sortedsetKeys[key]
if !ok {
return nil
}
elems := ss.byScore(asc)
members := make([]string, 0, len(elems))
for _, e := range elems {
members = append(members, e.member)
}
return members
}
// All members+scores from a sorted set, ordered by score.
func (db *RedisDB) ssetElements(key string) ssElems {
ss, ok := db.sortedsetKeys[key]
if !ok {
return nil
}
return ss.byScore(asc)
}
func (db *RedisDB) ssetRandomMember(key string) string {
elems := db.ssetElements(key)
if len(elems) == 0 {
return ""
}
return elems[db.master.randIntn(len(elems))].member
}
// ssetCard is the sorted set cardinality.
func (db *RedisDB) ssetCard(key string) int {
ss := db.sortedsetKeys[key]
return ss.card()
}
// ssetRank is the sorted set rank.
func (db *RedisDB) ssetRank(key, member string, d direction) (int, bool) {
ss := db.sortedsetKeys[key]
return ss.rankByScore(member, d)
}
// ssetScore is sorted set score.
func (db *RedisDB) ssetScore(key, member string) float64 {
ss := db.sortedsetKeys[key]
return ss[member]
}
// ssetMScore returns multiple scores of a list of members in a sorted set.
func (db *RedisDB) ssetMScore(key string, members []string) []float64 {
scores := make([]float64, 0, len(members))
ss := db.sortedsetKeys[key]
for _, member := range members {
scores = append(scores, ss[member])
}
return scores
}
// ssetRem is sorted set key delete.
func (db *RedisDB) ssetRem(key, member string) bool {
ss := db.sortedsetKeys[key]
_, ok := ss[member]
delete(ss, member)
if len(ss) == 0 {
// Delete key on removal of last member
db.del(key, true)
}
return ok
}
// ssetExists tells if a member exists in a sorted set.
func (db *RedisDB) ssetExists(key, member string) bool {
ss := db.sortedsetKeys[key]
_, ok := ss[member]
return ok
}
// ssetIncrby changes float sorted set score.
func (db *RedisDB) ssetIncrby(k, m string, delta float64) float64 {
ss, ok := db.sortedsetKeys[k]
if !ok {
ss = newSortedSet()
db.keys[k] = keyTypeSortedSet
db.sortedsetKeys[k] = ss
}
v, _ := ss.get(m)
v += delta
ss.set(v, m)
db.incr(k)
return v
}
// setDiff implements the logic behind SDIFF*
func (db *RedisDB) setDiff(keys []string) (setKey, error) {
key := keys[0]
keys = keys[1:]
if db.exists(key) && db.t(key) != keyTypeSet {
return nil, ErrWrongType
}
s := setKey{}
for k := range db.setKeys[key] {
s[k] = struct{}{}
}
for _, sk := range keys {
if !db.exists(sk) {
continue
}
if db.t(sk) != keyTypeSet {
return nil, ErrWrongType
}
for e := range db.setKeys[sk] {
delete(s, e)
}
}
return s, nil
}
// setInter implements the logic behind SINTER*
// len keys needs to be > 0
func (db *RedisDB) setInter(keys []string) (setKey, error) {
// all keys must either not exist, or be of type "set".
for _, key := range keys {
if db.exists(key) && db.t(key) != keyTypeSet {
return nil, ErrWrongType
}
}
key := keys[0]
keys = keys[1:]
if !db.exists(key) {
return nil, nil
}
if db.t(key) != keyTypeSet {
return nil, ErrWrongType
}
s := setKey{}
for k := range db.setKeys[key] {
s[k] = struct{}{}
}
for _, sk := range keys {
if !db.exists(sk) {
return setKey{}, nil
}
if db.t(sk) != keyTypeSet {
return nil, ErrWrongType
}
other := db.setKeys[sk]
for e := range s {
if _, ok := other[e]; ok {
continue
}
delete(s, e)
}
}
return s, nil
}
// setIntercard implements the logic behind SINTER*
// len keys needs to be > 0
func (db *RedisDB) setIntercard(keys []string, limit int) (int, error) {
// all keys must either not exist, or be of type "set".
allExist := true
for _, key := range keys {
exists := db.exists(key)
allExist = allExist && exists
if exists && db.t(key) != "set" {
return 0, ErrWrongType
}
}
if !allExist {
return 0, nil
}
smallestKey := keys[0]
smallestIdx := 0
for i, key := range keys {
if len(db.setKeys[key]) < len(db.setKeys[smallestKey]) {
smallestKey = key
smallestIdx = i
}
}
keys[smallestIdx] = keys[len(keys)-1]
keys = keys[:len(keys)-1]
count := 0
for item := range db.setKeys[smallestKey] {
inIntersection := true
for _, key := range keys {
if _, ok := db.setKeys[key][item]; !ok {
inIntersection = false
break
}
}
if inIntersection {
count++
if count == limit {
break
}
}
}
return count, nil
}
// setUnion implements the logic behind SUNION*
func (db *RedisDB) setUnion(keys []string) (setKey, error) {
key := keys[0]
keys = keys[1:]
if db.exists(key) && db.t(key) != "set" {
return nil, ErrWrongType
}
s := setKey{}
for k := range db.setKeys[key] {
s[k] = struct{}{}
}
for _, sk := range keys {
if !db.exists(sk) {
continue
}
if db.t(sk) != "set" {
return nil, ErrWrongType
}
for e := range db.setKeys[sk] {
s[e] = struct{}{}
}
}
return s, nil
}
func (db *RedisDB) newStream(key string) (*streamKey, error) {
if s, err := db.stream(key); err != nil {
return nil, err
} else if s != nil {
return nil, fmt.Errorf("ErrAlreadyExists")
}
db.keys[key] = keyTypeStream
s := newStreamKey()
db.streamKeys[key] = s
db.incr(key)
return s, nil
}
// return existing stream, or nil.
func (db *RedisDB) stream(key string) (*streamKey, error) {
if db.exists(key) && db.t(key) != keyTypeStream {
return nil, ErrWrongType
}
return db.streamKeys[key], nil
}
// return existing stream group, or nil.
func (db *RedisDB) streamGroup(key, group string) (*streamGroup, error) {
s, err := db.stream(key)
if err != nil || s == nil {
return nil, err
}
return s.groups[group], nil
}
// fastForward proceeds the current timestamp with duration, works as a time machine
func (db *RedisDB) fastForward(duration time.Duration) {
for _, key := range db.allKeys() {
if value, ok := db.ttl[key]; ok {
db.ttl[key] = value - duration
db.checkTTL(key)
}
}
}
func (db *RedisDB) checkTTL(key string) {
if v, ok := db.ttl[key]; ok && v <= 0 {
db.del(key, true)
}
}
// hllAdd adds members to a hll. Returns 1 if at least 1 if internal HyperLogLog was altered, otherwise 0
func (db *RedisDB) hllAdd(k string, elems ...string) int {
s, ok := db.hllKeys[k]
if !ok {
s = newHll()
db.keys[k] = keyTypeHll
}
hllAltered := 0
for _, e := range elems {
if s.Add([]byte(e)) {
hllAltered = 1
}
}
db.hllKeys[k] = s
db.incr(k)
return hllAltered
}
// hllCount estimates the amount of members added to hll by hllAdd. If called with several arguments, hllCount returns a sum of estimations
func (db *RedisDB) hllCount(keys []string) (int, error) {
countOverall := 0
for _, key := range keys {
if db.exists(key) && db.t(key) != keyTypeHll {
return 0, ErrNotValidHllValue
}
if !db.exists(key) {
continue
}
countOverall += db.hllKeys[key].Count()
}
return countOverall, nil
}
// hllMerge merges all the hlls provided as keys to the first key. Creates a new hll in the first key if it contains nothing
func (db *RedisDB) hllMerge(keys []string) error {
for _, key := range keys {
if db.exists(key) && db.t(key) != keyTypeHll {
return ErrNotValidHllValue
}
}
destKey := keys[0]
restKeys := keys[1:]
var destHll *hll
if db.exists(destKey) {
destHll = db.hllKeys[destKey]
} else {
destHll = newHll()
}
for _, key := range restKeys {
if !db.exists(key) {
continue
}
destHll.Merge(db.hllKeys[key])
}
db.hllKeys[destKey] = destHll
db.keys[destKey] = keyTypeHll
db.incr(destKey)
return nil
}
+824
View File
@@ -0,0 +1,824 @@
package miniredis
// Commands to modify and query our databases directly.
import (
"errors"
"math/big"
"time"
)
var (
// ErrKeyNotFound is returned when a key doesn't exist.
ErrKeyNotFound = errors.New(msgKeyNotFound)
// ErrWrongType when a key is not the right type.
ErrWrongType = errors.New(msgWrongType)
// ErrNotValidHllValue when a key is not a valid HyperLogLog string value.
ErrNotValidHllValue = errors.New(msgNotValidHllValue)
// ErrIntValueError can returned by INCRBY
ErrIntValueError = errors.New(msgInvalidInt)
// ErrIntValueOverflowError can be returned by INCR, DECR, INCRBY, DECRBY
ErrIntValueOverflowError = errors.New(msgIntOverflow)
// ErrFloatValueError can returned by INCRBYFLOAT
ErrFloatValueError = errors.New(msgInvalidFloat)
)
// Select sets the DB id for all direct commands.
func (m *Miniredis) Select(i int) {
m.Lock()
defer m.Unlock()
m.selectedDB = i
}
// Keys returns all keys from the selected database, sorted.
func (m *Miniredis) Keys() []string {
return m.DB(m.selectedDB).Keys()
}
// Keys returns all keys, sorted.
func (db *RedisDB) Keys() []string {
db.master.Lock()
defer db.master.Unlock()
return db.allKeys()
}
// FlushAll removes all keys from all databases.
func (m *Miniredis) FlushAll() {
m.Lock()
defer m.Unlock()
defer m.signal.Broadcast()
m.flushAll()
}
func (m *Miniredis) flushAll() {
for _, db := range m.dbs {
db.flush()
}
}
// FlushDB removes all keys from the selected database.
func (m *Miniredis) FlushDB() {
m.DB(m.selectedDB).FlushDB()
}
// FlushDB removes all keys.
func (db *RedisDB) FlushDB() {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
db.flush()
}
// Get returns string keys added with SET.
func (m *Miniredis) Get(k string) (string, error) {
return m.DB(m.selectedDB).Get(k)
}
// Get returns a string key.
func (db *RedisDB) Get(k string) (string, error) {
db.master.Lock()
defer db.master.Unlock()
if !db.exists(k) {
return "", ErrKeyNotFound
}
if db.t(k) != keyTypeString {
return "", ErrWrongType
}
return db.stringGet(k), nil
}
// Set sets a string key. Removes expire.
func (m *Miniredis) Set(k, v string) error {
return m.DB(m.selectedDB).Set(k, v)
}
// Set sets a string key. Removes expire.
// Unlike redis the key can't be an existing non-string key.
func (db *RedisDB) Set(k, v string) error {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
if db.exists(k) && db.t(k) != keyTypeString {
return ErrWrongType
}
db.del(k, true) // Remove expire
db.stringSet(k, v)
return nil
}
// Incr changes a int string value by delta.
func (m *Miniredis) Incr(k string, delta int) (int, error) {
return m.DB(m.selectedDB).Incr(k, delta)
}
// Incr changes a int string value by delta.
func (db *RedisDB) Incr(k string, delta int) (int, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
if db.exists(k) && db.t(k) != keyTypeString {
return 0, ErrWrongType
}
return db.stringIncr(k, delta)
}
// IncrByFloat increments the float value of a key by the given delta.
// is an alias for Miniredis.Incrfloat
func (m *Miniredis) IncrByFloat(k string, delta float64) (float64, error) {
return m.Incrfloat(k, delta)
}
// Incrfloat changes a float string value by delta.
func (m *Miniredis) Incrfloat(k string, delta float64) (float64, error) {
return m.DB(m.selectedDB).Incrfloat(k, delta)
}
// Incrfloat changes a float string value by delta.
func (db *RedisDB) Incrfloat(k string, delta float64) (float64, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
if db.exists(k) && db.t(k) != keyTypeString {
return 0, ErrWrongType
}
v, err := db.stringIncrfloat(k, big.NewFloat(delta))
if err != nil {
return 0, err
}
vf, _ := v.Float64()
return vf, nil
}
// List returns the list k, or an error if it's not there or something else.
// This is the same as the Redis command `LRANGE 0 -1`, but you can do your own
// range-ing.
func (m *Miniredis) List(k string) ([]string, error) {
return m.DB(m.selectedDB).List(k)
}
// List returns the list k, or an error if it's not there or something else.
// This is the same as the Redis command `LRANGE 0 -1`, but you can do your own
// range-ing.
func (db *RedisDB) List(k string) ([]string, error) {
db.master.Lock()
defer db.master.Unlock()
if !db.exists(k) {
return nil, ErrKeyNotFound
}
if db.t(k) != keyTypeList {
return nil, ErrWrongType
}
return db.listKeys[k], nil
}
// Lpush prepends one value to a list. Returns the new length.
func (m *Miniredis) Lpush(k, v string) (int, error) {
return m.DB(m.selectedDB).Lpush(k, v)
}
// Lpush prepends one value to a list. Returns the new length.
func (db *RedisDB) Lpush(k, v string) (int, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
if db.exists(k) && db.t(k) != keyTypeList {
return 0, ErrWrongType
}
return db.listLpush(k, v), nil
}
// Lpop removes and returns the last element in a list.
func (m *Miniredis) Lpop(k string) (string, error) {
return m.DB(m.selectedDB).Lpop(k)
}
// Lpop removes and returns the last element in a list.
func (db *RedisDB) Lpop(k string) (string, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
if !db.exists(k) {
return "", ErrKeyNotFound
}
if db.t(k) != keyTypeList {
return "", ErrWrongType
}
return db.listLpop(k), nil
}
// RPush appends one or multiple values to a list. Returns the new length.
// An alias for Push
func (m *Miniredis) RPush(k string, v ...string) (int, error) {
return m.Push(k, v...)
}
// Push add element at the end. Returns the new length.
func (m *Miniredis) Push(k string, v ...string) (int, error) {
return m.DB(m.selectedDB).Push(k, v...)
}
// Push add element at the end. Is called RPUSH in redis. Returns the new length.
func (db *RedisDB) Push(k string, v ...string) (int, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
if db.exists(k) && db.t(k) != keyTypeList {
return 0, ErrWrongType
}
return db.listPush(k, v...), nil
}
// RPop is an alias for Pop
func (m *Miniredis) RPop(k string) (string, error) {
return m.Pop(k)
}
// Pop removes and returns the last element. Is called RPOP in Redis.
func (m *Miniredis) Pop(k string) (string, error) {
return m.DB(m.selectedDB).Pop(k)
}
// Pop removes and returns the last element. Is called RPOP in Redis.
func (db *RedisDB) Pop(k string) (string, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
if !db.exists(k) {
return "", ErrKeyNotFound
}
if db.t(k) != keyTypeList {
return "", ErrWrongType
}
return db.listPop(k), nil
}
// SAdd adds keys to a set. Returns the number of new keys.
// Alias for SetAdd
func (m *Miniredis) SAdd(k string, elems ...string) (int, error) {
return m.SetAdd(k, elems...)
}
// SetAdd adds keys to a set. Returns the number of new keys.
func (m *Miniredis) SetAdd(k string, elems ...string) (int, error) {
return m.DB(m.selectedDB).SetAdd(k, elems...)
}
// SetAdd adds keys to a set. Returns the number of new keys.
func (db *RedisDB) SetAdd(k string, elems ...string) (int, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
if db.exists(k) && db.t(k) != keyTypeSet {
return 0, ErrWrongType
}
return db.setAdd(k, elems...), nil
}
// SMembers returns all keys in a set, sorted.
// Alias for Members.
func (m *Miniredis) SMembers(k string) ([]string, error) {
return m.Members(k)
}
// Members returns all keys in a set, sorted.
func (m *Miniredis) Members(k string) ([]string, error) {
return m.DB(m.selectedDB).Members(k)
}
// Members gives all set keys. Sorted.
func (db *RedisDB) Members(k string) ([]string, error) {
db.master.Lock()
defer db.master.Unlock()
if !db.exists(k) {
return nil, ErrKeyNotFound
}
if db.t(k) != keyTypeSet {
return nil, ErrWrongType
}
return db.setMembers(k), nil
}
// SIsMember tells if value is in the set.
// Alias for IsMember
func (m *Miniredis) SIsMember(k, v string) (bool, error) {
return m.IsMember(k, v)
}
// IsMember tells if value is in the set.
func (m *Miniredis) IsMember(k, v string) (bool, error) {
return m.DB(m.selectedDB).IsMember(k, v)
}
// IsMember tells if value is in the set.
func (db *RedisDB) IsMember(k, v string) (bool, error) {
db.master.Lock()
defer db.master.Unlock()
if !db.exists(k) {
return false, ErrKeyNotFound
}
if db.t(k) != keyTypeSet {
return false, ErrWrongType
}
return db.setIsMember(k, v), nil
}
// HKeys returns all (sorted) keys ('fields') for a hash key.
func (m *Miniredis) HKeys(k string) ([]string, error) {
return m.DB(m.selectedDB).HKeys(k)
}
// HKeys returns all (sorted) keys ('fields') for a hash key.
func (db *RedisDB) HKeys(key string) ([]string, error) {
db.master.Lock()
defer db.master.Unlock()
if !db.exists(key) {
return nil, ErrKeyNotFound
}
if db.t(key) != keyTypeHash {
return nil, ErrWrongType
}
return db.hashFields(key), nil
}
// Del deletes a key and any expiration value. Returns whether there was a key.
func (m *Miniredis) Del(k string) bool {
return m.DB(m.selectedDB).Del(k)
}
// Del deletes a key and any expiration value. Returns whether there was a key.
func (db *RedisDB) Del(k string) bool {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
if !db.exists(k) {
return false
}
db.del(k, true)
return true
}
// Unlink deletes a key and any expiration value. Returns where there was a key.
// It's exactly the same as Del() and is not async. It is here for the consistency.
func (m *Miniredis) Unlink(k string) bool {
return m.Del(k)
}
// Unlink deletes a key and any expiration value. Returns where there was a key.
// It's exactly the same as Del() and is not async. It is here for the consistency.
func (db *RedisDB) Unlink(k string) bool {
return db.Del(k)
}
// TTL is the left over time to live. As set via EXPIRE, PEXPIRE, EXPIREAT,
// PEXPIREAT.
// Note: this direct function returns 0 if there is no TTL set, unlike redis,
// which returns -1.
func (m *Miniredis) TTL(k string) time.Duration {
return m.DB(m.selectedDB).TTL(k)
}
// TTL is the left over time to live. As set via EXPIRE, PEXPIRE, EXPIREAT,
// PEXPIREAT.
// 0 if not set.
func (db *RedisDB) TTL(k string) time.Duration {
db.master.Lock()
defer db.master.Unlock()
return db.ttl[k]
}
// SetTTL sets the TTL of a key.
func (m *Miniredis) SetTTL(k string, ttl time.Duration) {
m.DB(m.selectedDB).SetTTL(k, ttl)
}
// SetTTL sets the time to live of a key.
func (db *RedisDB) SetTTL(k string, ttl time.Duration) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
db.ttl[k] = ttl
db.incr(k)
}
// Type gives the type of a key, or ""
func (m *Miniredis) Type(k string) string {
return m.DB(m.selectedDB).Type(k)
}
// Type gives the type of a key, or ""
func (db *RedisDB) Type(k string) string {
db.master.Lock()
defer db.master.Unlock()
return db.t(k)
}
// Exists tells whether a key exists.
func (m *Miniredis) Exists(k string) bool {
return m.DB(m.selectedDB).Exists(k)
}
// Exists tells whether a key exists.
func (db *RedisDB) Exists(k string) bool {
db.master.Lock()
defer db.master.Unlock()
return db.exists(k)
}
// HGet returns hash keys added with HSET.
// This will return an empty string if the key is not set. Redis would return
// a nil.
// Returns empty string when the key is of a different type.
func (m *Miniredis) HGet(k, f string) string {
return m.DB(m.selectedDB).HGet(k, f)
}
// HGet returns hash keys added with HSET.
// Returns empty string when the key is of a different type.
func (db *RedisDB) HGet(k, f string) string {
db.master.Lock()
defer db.master.Unlock()
h, ok := db.hashKeys[k]
if !ok {
return ""
}
return h[f]
}
// HSet sets hash keys.
// If there is another key by the same name it will be gone.
func (m *Miniredis) HSet(k string, fv ...string) {
m.DB(m.selectedDB).HSet(k, fv...)
}
// HSet sets hash keys.
// If there is another key by the same name it will be gone.
func (db *RedisDB) HSet(k string, fv ...string) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
db.hashSet(k, fv...)
}
// HDel deletes a hash key.
func (m *Miniredis) HDel(k, f string) {
m.DB(m.selectedDB).HDel(k, f)
}
// HDel deletes a hash key.
func (db *RedisDB) HDel(k, f string) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
db.hdel(k, f)
}
func (db *RedisDB) hdel(k, f string) {
if _, ok := db.hashKeys[k]; !ok {
return
}
delete(db.hashKeys[k], f)
db.incr(k)
}
// HIncrBy increases the integer value of a hash field by delta (int).
func (m *Miniredis) HIncrBy(k, f string, delta int) (int, error) {
return m.HIncr(k, f, delta)
}
// HIncr increases a key/field by delta (int).
func (m *Miniredis) HIncr(k, f string, delta int) (int, error) {
return m.DB(m.selectedDB).HIncr(k, f, delta)
}
// HIncr increases a key/field by delta (int).
func (db *RedisDB) HIncr(k, f string, delta int) (int, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
return db.hashIncr(k, f, delta)
}
// HIncrByFloat increases a key/field by delta (float).
func (m *Miniredis) HIncrByFloat(k, f string, delta float64) (float64, error) {
return m.HIncrfloat(k, f, delta)
}
// HIncrfloat increases a key/field by delta (float).
func (m *Miniredis) HIncrfloat(k, f string, delta float64) (float64, error) {
return m.DB(m.selectedDB).HIncrfloat(k, f, delta)
}
// HIncrfloat increases a key/field by delta (float).
func (db *RedisDB) HIncrfloat(k, f string, delta float64) (float64, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
v, err := db.hashIncrfloat(k, f, big.NewFloat(delta))
if err != nil {
return 0, err
}
vf, _ := v.Float64()
return vf, nil
}
// SRem removes fields from a set. Returns number of deleted fields.
func (m *Miniredis) SRem(k string, fields ...string) (int, error) {
return m.DB(m.selectedDB).SRem(k, fields...)
}
// SRem removes fields from a set. Returns number of deleted fields.
func (db *RedisDB) SRem(k string, fields ...string) (int, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
if !db.exists(k) {
return 0, ErrKeyNotFound
}
if db.t(k) != keyTypeSet {
return 0, ErrWrongType
}
return db.setRem(k, fields...), nil
}
// ZAdd adds a score,member to a sorted set.
func (m *Miniredis) ZAdd(k string, score float64, member string) (bool, error) {
return m.DB(m.selectedDB).ZAdd(k, score, member)
}
// ZAdd adds a score,member to a sorted set.
func (db *RedisDB) ZAdd(k string, score float64, member string) (bool, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
if db.exists(k) && db.t(k) != keyTypeSortedSet {
return false, ErrWrongType
}
return db.ssetAdd(k, score, member), nil
}
// ZMembers returns all members of a sorted set by score
func (m *Miniredis) ZMembers(k string) ([]string, error) {
return m.DB(m.selectedDB).ZMembers(k)
}
// ZMembers returns all members of a sorted set by score
func (db *RedisDB) ZMembers(k string) ([]string, error) {
db.master.Lock()
defer db.master.Unlock()
if !db.exists(k) {
return nil, ErrKeyNotFound
}
if db.t(k) != keyTypeSortedSet {
return nil, ErrWrongType
}
return db.ssetMembers(k), nil
}
// SortedSet returns a raw string->float64 map.
func (m *Miniredis) SortedSet(k string) (map[string]float64, error) {
return m.DB(m.selectedDB).SortedSet(k)
}
// SortedSet returns a raw string->float64 map.
func (db *RedisDB) SortedSet(k string) (map[string]float64, error) {
db.master.Lock()
defer db.master.Unlock()
if !db.exists(k) {
return nil, ErrKeyNotFound
}
if db.t(k) != keyTypeSortedSet {
return nil, ErrWrongType
}
return db.sortedSet(k), nil
}
// ZRem deletes a member. Returns whether the was a key.
func (m *Miniredis) ZRem(k, member string) (bool, error) {
return m.DB(m.selectedDB).ZRem(k, member)
}
// ZRem deletes a member. Returns whether the was a key.
func (db *RedisDB) ZRem(k, member string) (bool, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
if !db.exists(k) {
return false, ErrKeyNotFound
}
if db.t(k) != keyTypeSortedSet {
return false, ErrWrongType
}
return db.ssetRem(k, member), nil
}
// ZScore gives the score of a sorted set member.
func (m *Miniredis) ZScore(k, member string) (float64, error) {
return m.DB(m.selectedDB).ZScore(k, member)
}
// ZScore gives the score of a sorted set member.
func (db *RedisDB) ZScore(k, member string) (float64, error) {
db.master.Lock()
defer db.master.Unlock()
if !db.exists(k) {
return 0, ErrKeyNotFound
}
if db.t(k) != keyTypeSortedSet {
return 0, ErrWrongType
}
return db.ssetScore(k, member), nil
}
// ZScore gives scores of a list of members in a sorted set.
func (m *Miniredis) ZMScore(k string, members ...string) ([]float64, error) {
return m.DB(m.selectedDB).ZMScore(k, members)
}
func (db *RedisDB) ZMScore(k string, members []string) ([]float64, error) {
db.master.Lock()
defer db.master.Unlock()
if !db.exists(k) {
return nil, ErrKeyNotFound
}
if db.t(k) != keyTypeSortedSet {
return nil, ErrWrongType
}
return db.ssetMScore(k, members), nil
}
// XAdd adds an entry to a stream. `id` can be left empty or be '*'.
// If a value is given normal XADD rules apply. Values should be an even
// length.
func (m *Miniredis) XAdd(k string, id string, values []string) (string, error) {
return m.DB(m.selectedDB).XAdd(k, id, values)
}
// XAdd adds an entry to a stream. `id` can be left empty or be '*'.
// If a value is given normal XADD rules apply. Values should be an even
// length.
func (db *RedisDB) XAdd(k string, id string, values []string) (string, error) {
db.master.Lock()
defer db.master.Unlock()
defer db.master.signal.Broadcast()
s, err := db.stream(k)
if err != nil {
return "", err
}
if s == nil {
s, _ = db.newStream(k)
}
return s.add(id, values, db.master.effectiveNow())
}
// Stream returns a slice of stream entries. Oldest first.
func (m *Miniredis) Stream(k string) ([]StreamEntry, error) {
return m.DB(m.selectedDB).Stream(k)
}
// Stream returns a slice of stream entries. Oldest first.
func (db *RedisDB) Stream(key string) ([]StreamEntry, error) {
db.master.Lock()
defer db.master.Unlock()
s, err := db.stream(key)
if err != nil {
return nil, err
}
if s == nil {
return nil, nil
}
return s.entries, nil
}
// Publish a message to subscribers. Returns the number of receivers.
func (m *Miniredis) Publish(channel, message string) int {
m.Lock()
defer m.Unlock()
return m.publish(channel, message)
}
// PubSubChannels is "PUBSUB CHANNELS <pattern>". An empty pattern is fine
// (meaning all channels).
// Returned channels will be ordered alphabetically.
func (m *Miniredis) PubSubChannels(pattern string) []string {
m.Lock()
defer m.Unlock()
return activeChannels(m.allSubscribers(), pattern)
}
// PubSubNumSub is "PUBSUB NUMSUB [channels]". It returns all channels with their
// subscriber count.
func (m *Miniredis) PubSubNumSub(channels ...string) map[string]int {
m.Lock()
defer m.Unlock()
subs := m.allSubscribers()
res := map[string]int{}
for _, channel := range channels {
res[channel] = countSubs(subs, channel)
}
return res
}
// PubSubNumPat is "PUBSUB NUMPAT"
func (m *Miniredis) PubSubNumPat() int {
m.Lock()
defer m.Unlock()
return countPsubs(m.allSubscribers())
}
// PfAdd adds keys to a hll. Returns the flag which equals to 1 if the inner hll value has been changed.
func (m *Miniredis) PfAdd(k string, elems ...string) (int, error) {
return m.DB(m.selectedDB).HllAdd(k, elems...)
}
// HllAdd adds keys to a hll. Returns the flag which equals to true if the inner hll value has been changed.
func (db *RedisDB) HllAdd(k string, elems ...string) (int, error) {
db.master.Lock()
defer db.master.Unlock()
if db.exists(k) && db.t(k) != keyTypeHll {
return 0, ErrWrongType
}
return db.hllAdd(k, elems...), nil
}
// PfCount returns an estimation of the amount of elements previously added to a hll.
func (m *Miniredis) PfCount(keys ...string) (int, error) {
return m.DB(m.selectedDB).HllCount(keys...)
}
// HllCount returns an estimation of the amount of elements previously added to a hll.
func (db *RedisDB) HllCount(keys ...string) (int, error) {
db.master.Lock()
defer db.master.Unlock()
return db.hllCount(keys)
}
// PfMerge merges all the input hlls into a hll under destKey key.
func (m *Miniredis) PfMerge(destKey string, sourceKeys ...string) error {
return m.DB(m.selectedDB).HllMerge(destKey, sourceKeys...)
}
// HllMerge merges all the input hlls into a hll under destKey key.
func (db *RedisDB) HllMerge(destKey string, sourceKeys ...string) error {
db.master.Lock()
defer db.master.Unlock()
return db.hllMerge(append([]string{destKey}, sourceKeys...))
}
// Copy a value.
// Needs the IDs of both the source and dest DBs (which can differ).
// Returns ErrKeyNotFound if src does not exist.
// Overwrites dest if it already exists (unlike the redis command, which needs a flag to allow that).
func (m *Miniredis) Copy(srcDB int, src string, destDB int, dest string) error {
return m.copy(m.DB(srcDB), src, m.DB(destDB), dest)
}
+26
View File
@@ -0,0 +1,26 @@
This code is derived from the C code in redis-7.2.0/deps/fpconv/*, which has
this license:
Boost Software License - Version 1.0 - August 17th, 2003
Permission is hereby granted, free of charge, to any person or organization
obtaining a copy of the software and accompanying documentation covered by
this license (the "Software") to use, reproduce, display, distribute,
execute, and transmit the Software, and to prepare derivative works of the
Software, and to permit third-parties to whom the Software is furnished to
do so, all subject to the following:
The copyright notices in the Software and this entire statement, including
the above license grant, this restriction and the following disclaimer,
must be included in all copies of the Software, in whole or in part, and
all derivative works of the Software, unless such copies or derivative
works are solely in the form of machine-executable object code generated by
a source language processor.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
+6
View File
@@ -0,0 +1,6 @@
.PHONY: test fuzz
test:
go test
fuzz:
go test -fuzz=Fuzz
+3
View File
@@ -0,0 +1,3 @@
This is a translation of the actual C code in Redis (7.2) which does the float
-> string conversion.
Strconv does a close enough job, but we can use the exact same logic, so why not.
+286
View File
@@ -0,0 +1,286 @@
package fpconv
import (
"math"
)
var (
fracmask uint64 = 0x000FFFFFFFFFFFFF
expmask uint64 = 0x7FF0000000000000
hiddenbit uint64 = 0x0010000000000000
signmask uint64 = 0x8000000000000000
expbias int64 = 1023 + 52
zeros = []rune("0000000000000000000000")
tens = []uint64{
10000000000000000000,
1000000000000000000,
100000000000000000,
10000000000000000,
1000000000000000,
100000000000000,
10000000000000,
1000000000000,
100000000000,
10000000000,
1000000000,
100000000,
10000000,
1000000,
100000,
10000,
1000,
100,
10,
1}
)
func absv(n int) int {
if n < 0 {
return -n
}
return n
}
func minv(a, b int) int {
if a < b {
return a
}
return b
}
func Dtoa(d float64) string {
var (
dest [25]rune // Note C has 24, which is broken
digits [18]rune
str_len int = 0
neg = false
)
if get_dbits(d)&signmask != 0 {
dest[0] = '-'
str_len++
neg = true
}
if spec := filter_special(d, dest[str_len:]); spec != 0 {
return string(dest[:str_len+spec])
}
var (
k int = 0
ndigits int = grisu2(d, &digits, &k)
)
str_len += emit_digits(&digits, ndigits, dest[str_len:], k, neg)
return string(dest[:str_len])
}
func filter_special(fp float64, dest []rune) int {
if fp == 0.0 {
dest[0] = '0'
return 1
}
if math.IsNaN(fp) {
dest[0] = 'n'
dest[1] = 'a'
dest[2] = 'n'
return 3
}
if math.IsInf(fp, 0) {
dest[0] = 'i'
dest[1] = 'n'
dest[2] = 'f'
return 3
}
return 0
}
func grisu2(d float64, digits *[18]rune, K *int) int {
w := build_fp(d)
lower, upper := get_normalized_boundaries(w)
w = normalize(w)
var k int64
cp := find_cachedpow10(upper.exp, &k)
w = multiply(w, cp)
upper = multiply(upper, cp)
lower = multiply(lower, cp)
lower.frac++
upper.frac--
*K = int(-k)
return generate_digits(w, upper, lower, digits[:], K)
}
func emit_digits(digits *[18]rune, ndigits int, dest []rune, K int, neg bool) int {
exp := int(absv(K + ndigits - 1))
/* write plain integer */
if K >= 0 && (exp < (ndigits + 7)) {
copy(dest, digits[:ndigits])
copy(dest[ndigits:], zeros[:K])
return ndigits + K
}
/* write decimal w/o scientific notation */
if K < 0 && (K > -7 || exp < 4) {
offset := int(ndigits - absv(K))
/* fp < 1.0 -> write leading zero */
if offset <= 0 {
offset = -offset
dest[0] = '0'
dest[1] = '.'
copy(dest[2:], zeros[:offset])
copy(dest[offset+2:], digits[:ndigits])
return ndigits + 2 + offset
/* fp > 1.0 */
} else {
copy(dest, digits[:offset])
dest[offset] = '.'
copy(dest[offset+1:], digits[offset:offset+ndigits-offset])
return ndigits + 1
}
}
/* write decimal w/ scientific notation */
l := 18 // was: 18-neg
if neg {
l--
}
ndigits = minv(ndigits, l)
var idx int = 0
dest[idx] = digits[0]
idx++
if ndigits > 1 {
dest[idx] = '.'
idx++
copy(dest[idx:], digits[+1:ndigits-1+1])
idx += ndigits - 1
}
dest[idx] = 'e'
idx++
sign := '+'
if K+ndigits-1 < 0 {
sign = '-'
}
dest[idx] = sign
idx++
var cent rune = 0
if exp > 99 {
cent = rune(exp / 100)
dest[idx] = cent + '0'
idx++
exp -= int(cent) * 100
}
if exp > 9 {
dec := rune(exp / 10)
dest[idx] = dec + '0'
idx++
exp -= int(dec) * 10
} else if cent != 0 {
dest[idx] = '0'
idx++
}
dest[idx] = rune(exp%10) + '0'
idx++
return idx
}
func generate_digits(fp, upper, lower Fp, digits []rune, K *int) int {
var (
wfrac = uint64(upper.frac - fp.frac)
delta = uint64(upper.frac - lower.frac)
)
one := Fp{
frac: 1 << -upper.exp,
exp: upper.exp,
}
part1 := uint64(upper.frac >> -one.exp)
part2 := uint64(upper.frac & (one.frac - 1))
var (
idx = 0
kappa = 10
index = 10
)
/* 1000000000 */
for ; kappa > 0; index++ {
div := tens[index]
digit := part1 / div
if digit != 0 || idx != 0 {
digits[idx] = rune(digit) + '0'
idx++
}
part1 -= digit * div
kappa--
tmp := (part1 << -one.exp) + part2
if tmp <= delta {
*K += kappa
round_digit(digits, idx, delta, tmp, div<<-one.exp, wfrac)
return idx
}
}
/* 10 */
index = 18
for {
var unit uint64 = tens[index]
part2 *= 10
delta *= 10
kappa--
digit := part2 >> -one.exp
if digit != 0 || idx != 0 {
digits[idx] = rune(digit) + '0'
idx++
}
part2 &= uint64(one.frac) - 1
if part2 < delta {
*K += kappa
round_digit(digits, idx, delta, part2, uint64(one.frac), wfrac*unit)
return idx
}
index--
}
}
func round_digit(digits []rune,
ndigits int,
delta uint64,
rem uint64,
kappa uint64,
frac uint64) {
for rem < frac && delta-rem >= kappa &&
(rem+kappa < frac || frac-rem > rem+kappa-frac) {
digits[ndigits-1]--
rem += kappa
}
}
+96
View File
@@ -0,0 +1,96 @@
package fpconv
import (
"math"
)
type (
Fp struct {
frac uint64
exp int64
}
)
func build_fp(d float64) Fp {
bits := get_dbits(d)
fp := Fp{
frac: bits & fracmask,
exp: int64((bits & expmask) >> 52),
}
if fp.exp != 0 {
fp.frac += hiddenbit
fp.exp -= expbias
} else {
fp.exp = -expbias + 1
}
return fp
}
func normalize(fp Fp) Fp {
for (fp.frac & hiddenbit) == 0 {
fp.frac <<= 1
fp.exp--
}
var shift int64 = 64 - 52 - 1
fp.frac <<= shift
fp.exp -= shift
return fp
}
func multiply(a, b Fp) Fp {
lomask := uint64(0x00000000FFFFFFFF)
var (
ah_bl = uint64((a.frac >> 32) * (b.frac & lomask))
al_bh = uint64((a.frac & lomask) * (b.frac >> 32))
al_bl = uint64((a.frac & lomask) * (b.frac & lomask))
ah_bh = uint64((a.frac >> 32) * (b.frac >> 32))
)
tmp := uint64((ah_bl & lomask) + (al_bh & lomask) + (al_bl >> 32))
/* round up */
tmp += uint64(1) << 31
return Fp{
ah_bh + (ah_bl >> 32) + (al_bh >> 32) + (tmp >> 32),
a.exp + b.exp + 64,
}
}
func get_dbits(d float64) uint64 {
return math.Float64bits(d)
}
func get_normalized_boundaries(fp Fp) (Fp, Fp) {
upper := Fp{
frac: (fp.frac << 1) + 1,
exp: fp.exp - 1,
}
for (upper.frac & (hiddenbit << 1)) == 0 {
upper.frac <<= 1
upper.exp--
}
var u_shift int64 = 64 - 52 - 2
upper.frac <<= u_shift
upper.exp = upper.exp - u_shift
l_shift := int64(1)
if fp.frac == hiddenbit {
l_shift = 2
}
lower := Fp{
frac: (fp.frac << l_shift) - 1,
exp: fp.exp - l_shift,
}
lower.frac <<= lower.exp - upper.exp
lower.exp = upper.exp
return lower, upper
}
+82
View File
@@ -0,0 +1,82 @@
package fpconv
var (
npowers int64 = 87
steppowers int64 = 8
firstpower int64 = -348 /* 10 ^ -348 */
expmax = -32
expmin = -60
powers_ten = []Fp{
{18054884314459144840, -1220}, {13451937075301367670, -1193},
{10022474136428063862, -1166}, {14934650266808366570, -1140},
{11127181549972568877, -1113}, {16580792590934885855, -1087},
{12353653155963782858, -1060}, {18408377700990114895, -1034},
{13715310171984221708, -1007}, {10218702384817765436, -980},
{15227053142812498563, -954}, {11345038669416679861, -927},
{16905424996341287883, -901}, {12595523146049147757, -874},
{9384396036005875287, -847}, {13983839803942852151, -821},
{10418772551374772303, -794}, {15525180923007089351, -768},
{11567161174868858868, -741}, {17236413322193710309, -715},
{12842128665889583758, -688}, {9568131466127621947, -661},
{14257626930069360058, -635}, {10622759856335341974, -608},
{15829145694278690180, -582}, {11793632577567316726, -555},
{17573882009934360870, -529}, {13093562431584567480, -502},
{9755464219737475723, -475}, {14536774485912137811, -449},
{10830740992659433045, -422}, {16139061738043178685, -396},
{12024538023802026127, -369}, {17917957937422433684, -343},
{13349918974505688015, -316}, {9946464728195732843, -289},
{14821387422376473014, -263}, {11042794154864902060, -236},
{16455045573212060422, -210}, {12259964326927110867, -183},
{18268770466636286478, -157}, {13611294676837538539, -130},
{10141204801825835212, -103}, {15111572745182864684, -77},
{11258999068426240000, -50}, {16777216000000000000, -24},
{12500000000000000000, 3}, {9313225746154785156, 30},
{13877787807814456755, 56}, {10339757656912845936, 83},
{15407439555097886824, 109}, {11479437019748901445, 136},
{17105694144590052135, 162}, {12744735289059618216, 189},
{9495567745759798747, 216}, {14149498560666738074, 242},
{10542197943230523224, 269}, {15709099088952724970, 295},
{11704190886730495818, 322}, {17440603504673385349, 348},
{12994262207056124023, 375}, {9681479787123295682, 402},
{14426529090290212157, 428}, {10748601772107342003, 455},
{16016664761464807395, 481}, {11933345169920330789, 508},
{17782069995880619868, 534}, {13248674568444952270, 561},
{9871031767461413346, 588}, {14708983551653345445, 614},
{10959046745042015199, 641}, {16330252207878254650, 667},
{12166986024289022870, 694}, {18130221999122236476, 720},
{13508068024458167312, 747}, {10064294952495520794, 774},
{14996968138956309548, 800}, {11173611982879273257, 827},
{16649979327439178909, 853}, {12405201291620119593, 880},
{9242595204427927429, 907}, {13772540099066387757, 933},
{10261342003245940623, 960}, {15290591125556738113, 986},
{11392378155556871081, 1013}, {16975966327722178521, 1039},
{12648080533535911531, 1066},
}
)
func find_cachedpow10(exp int64, k *int64) Fp {
one_log_ten := 0.30102999566398114
approx := int64(float64(-(exp + npowers)) * one_log_ten)
idx := int((approx - firstpower) / steppowers)
for {
current := int(exp + powers_ten[idx].exp + 64)
if current < expmin {
idx++
continue
}
if current > expmax {
idx--
continue
}
*k = (firstpower + int64(idx)*steppowers)
return powers_ten[idx]
}
}
+46
View File
@@ -0,0 +1,46 @@
package miniredis
import (
"math"
"github.com/alicebob/miniredis/v2/geohash"
)
func toGeohash(long, lat float64) uint64 {
return geohash.EncodeIntWithPrecision(lat, long, 52)
}
func fromGeohash(score uint64) (float64, float64) {
lat, long := geohash.DecodeIntWithPrecision(score, 52)
return long, lat
}
// haversin(θ) function
func hsin(theta float64) float64 {
return math.Pow(math.Sin(theta/2), 2)
}
// distance function returns the distance (in meters) between two points of
// a given longitude and latitude relatively accurately (using a spherical
// approximation of the Earth) through the Haversin Distance Formula for
// great arc distance on a sphere with accuracy for small distances
// point coordinates are supplied in degrees and converted into rad. in the func
// distance returned is meters
// http://en.wikipedia.org/wiki/Haversine_formula
// Source: https://gist.github.com/cdipaolo/d3f8db3848278b49db68
func distance(lat1, lon1, lat2, lon2 float64) float64 {
// convert to radians
// must cast radius as float to multiply later
var la1, lo1, la2, lo2 float64
la1 = lat1 * math.Pi / 180
lo1 = lon1 * math.Pi / 180
la2 = lat2 * math.Pi / 180
lo2 = lon2 * math.Pi / 180
earth := 6372797.560856 // Earth radius in METERS, according to src/geohash_helper.c
// calculate
h := hsin(la2-la1) + math.Cos(la1)*math.Cos(la2)*hsin(lo2-lo1)
return 2 * earth * math.Asin(math.Sqrt(h))
}
+22
View File
@@ -0,0 +1,22 @@
The MIT License (MIT)
Copyright (c) 2015 Michael McLoughlin
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+2
View File
@@ -0,0 +1,2 @@
This is a (selected) copy of github.com/mmcloughlin/geohash with the latitude
range changed from 90 to ~85, to align with the algorithm use by Redis.
+44
View File
@@ -0,0 +1,44 @@
package geohash
// encoding encapsulates an encoding defined by a given base32 alphabet.
type encoding struct {
encode string
decode [256]byte
}
// newEncoding constructs a new encoding defined by the given alphabet,
// which must be a 32-byte string.
func newEncoding(encoder string) *encoding {
e := new(encoding)
e.encode = encoder
for i := 0; i < len(e.decode); i++ {
e.decode[i] = 0xff
}
for i := 0; i < len(encoder); i++ {
e.decode[encoder[i]] = byte(i)
}
return e
}
// Decode string into bits of a 64-bit word. The string s may be at most 12
// characters.
func (e *encoding) Decode(s string) uint64 {
x := uint64(0)
for i := 0; i < len(s); i++ {
x = (x << 5) | uint64(e.decode[s[i]])
}
return x
}
// Encode bits of 64-bit word into a string.
func (e *encoding) Encode(x uint64) string {
b := [12]byte{}
for i := 0; i < 12; i++ {
b[11-i] = e.encode[x&0x1f]
x >>= 5
}
return string(b[:])
}
// Base32Encoding with the Geohash alphabet.
var base32encoding = newEncoding("0123456789bcdefghjkmnpqrstuvwxyz")
+269
View File
@@ -0,0 +1,269 @@
// Package geohash provides encoding and decoding of string and integer
// geohashes.
package geohash
import (
"math"
)
const (
ENC_LAT = 85.05112878
ENC_LONG = 180.0
)
// Direction represents directions in the latitute/longitude space.
type Direction int
// Cardinal and intercardinal directions
const (
North Direction = iota
NorthEast
East
SouthEast
South
SouthWest
West
NorthWest
)
// Encode the point (lat, lng) as a string geohash with the standard 12
// characters of precision.
func Encode(lat, lng float64) string {
return EncodeWithPrecision(lat, lng, 12)
}
// EncodeWithPrecision encodes the point (lat, lng) as a string geohash with
// the specified number of characters of precision (max 12).
func EncodeWithPrecision(lat, lng float64, chars uint) string {
bits := 5 * chars
inthash := EncodeIntWithPrecision(lat, lng, bits)
enc := base32encoding.Encode(inthash)
return enc[12-chars:]
}
// encodeInt provides a Go implementation of integer geohash. This is the
// default implementation of EncodeInt, but optimized versions are provided
// for certain architectures.
func EncodeInt(lat, lng float64) uint64 {
latInt := encodeRange(lat, ENC_LAT)
lngInt := encodeRange(lng, ENC_LONG)
return interleave(latInt, lngInt)
}
// EncodeIntWithPrecision encodes the point (lat, lng) to an integer with the
// specified number of bits.
func EncodeIntWithPrecision(lat, lng float64, bits uint) uint64 {
hash := EncodeInt(lat, lng)
return hash >> (64 - bits)
}
// Box represents a rectangle in latitude/longitude space.
type Box struct {
MinLat float64
MaxLat float64
MinLng float64
MaxLng float64
}
// Center returns the center of the box.
func (b Box) Center() (lat, lng float64) {
lat = (b.MinLat + b.MaxLat) / 2.0
lng = (b.MinLng + b.MaxLng) / 2.0
return
}
// Contains decides whether (lat, lng) is contained in the box. The
// containment test is inclusive of the edges and corners.
func (b Box) Contains(lat, lng float64) bool {
return (b.MinLat <= lat && lat <= b.MaxLat &&
b.MinLng <= lng && lng <= b.MaxLng)
}
// errorWithPrecision returns the error range in latitude and longitude for in
// integer geohash with bits of precision.
func errorWithPrecision(bits uint) (latErr, lngErr float64) {
b := int(bits)
latBits := b / 2
lngBits := b - latBits
latErr = math.Ldexp(180.0, -latBits)
lngErr = math.Ldexp(360.0, -lngBits)
return
}
// BoundingBox returns the region encoded by the given string geohash.
func BoundingBox(hash string) Box {
bits := uint(5 * len(hash))
inthash := base32encoding.Decode(hash)
return BoundingBoxIntWithPrecision(inthash, bits)
}
// BoundingBoxIntWithPrecision returns the region encoded by the integer
// geohash with the specified precision.
func BoundingBoxIntWithPrecision(hash uint64, bits uint) Box {
fullHash := hash << (64 - bits)
latInt, lngInt := deinterleave(fullHash)
lat := decodeRange(latInt, ENC_LAT)
lng := decodeRange(lngInt, ENC_LONG)
latErr, lngErr := errorWithPrecision(bits)
return Box{
MinLat: lat,
MaxLat: lat + latErr,
MinLng: lng,
MaxLng: lng + lngErr,
}
}
// BoundingBoxInt returns the region encoded by the given 64-bit integer
// geohash.
func BoundingBoxInt(hash uint64) Box {
return BoundingBoxIntWithPrecision(hash, 64)
}
// DecodeCenter decodes the string geohash to the central point of the bounding box.
func DecodeCenter(hash string) (lat, lng float64) {
box := BoundingBox(hash)
return box.Center()
}
// DecodeIntWithPrecision decodes the provided integer geohash with bits of
// precision to a (lat, lng) point.
func DecodeIntWithPrecision(hash uint64, bits uint) (lat, lng float64) {
box := BoundingBoxIntWithPrecision(hash, bits)
return box.Center()
}
// DecodeInt decodes the provided 64-bit integer geohash to a (lat, lng) point.
func DecodeInt(hash uint64) (lat, lng float64) {
return DecodeIntWithPrecision(hash, 64)
}
// Neighbors returns a slice of geohash strings that correspond to the provided
// geohash's neighbors.
func Neighbors(hash string) []string {
box := BoundingBox(hash)
lat, lng := box.Center()
latDelta := box.MaxLat - box.MinLat
lngDelta := box.MaxLng - box.MinLng
precision := uint(len(hash))
return []string{
// N
EncodeWithPrecision(lat+latDelta, lng, precision),
// NE,
EncodeWithPrecision(lat+latDelta, lng+lngDelta, precision),
// E,
EncodeWithPrecision(lat, lng+lngDelta, precision),
// SE,
EncodeWithPrecision(lat-latDelta, lng+lngDelta, precision),
// S,
EncodeWithPrecision(lat-latDelta, lng, precision),
// SW,
EncodeWithPrecision(lat-latDelta, lng-lngDelta, precision),
// W,
EncodeWithPrecision(lat, lng-lngDelta, precision),
// NW
EncodeWithPrecision(lat+latDelta, lng-lngDelta, precision),
}
}
// NeighborsInt returns a slice of uint64s that correspond to the provided hash's
// neighbors at 64-bit precision.
func NeighborsInt(hash uint64) []uint64 {
return NeighborsIntWithPrecision(hash, 64)
}
// NeighborsIntWithPrecision returns a slice of uint64s that correspond to the
// provided hash's neighbors at the given precision.
func NeighborsIntWithPrecision(hash uint64, bits uint) []uint64 {
box := BoundingBoxIntWithPrecision(hash, bits)
lat, lng := box.Center()
latDelta := box.MaxLat - box.MinLat
lngDelta := box.MaxLng - box.MinLng
return []uint64{
// N
EncodeIntWithPrecision(lat+latDelta, lng, bits),
// NE,
EncodeIntWithPrecision(lat+latDelta, lng+lngDelta, bits),
// E,
EncodeIntWithPrecision(lat, lng+lngDelta, bits),
// SE,
EncodeIntWithPrecision(lat-latDelta, lng+lngDelta, bits),
// S,
EncodeIntWithPrecision(lat-latDelta, lng, bits),
// SW,
EncodeIntWithPrecision(lat-latDelta, lng-lngDelta, bits),
// W,
EncodeIntWithPrecision(lat, lng-lngDelta, bits),
// NW
EncodeIntWithPrecision(lat+latDelta, lng-lngDelta, bits),
}
}
// Neighbor returns a geohash string that corresponds to the provided
// geohash's neighbor in the provided direction
func Neighbor(hash string, direction Direction) string {
return Neighbors(hash)[direction]
}
// NeighborInt returns a uint64 that corresponds to the provided hash's
// neighbor in the provided direction at 64-bit precision.
func NeighborInt(hash uint64, direction Direction) uint64 {
return NeighborsIntWithPrecision(hash, 64)[direction]
}
// NeighborIntWithPrecision returns a uint64s that corresponds to the
// provided hash's neighbor in the provided direction at the given precision.
func NeighborIntWithPrecision(hash uint64, bits uint, direction Direction) uint64 {
return NeighborsIntWithPrecision(hash, bits)[direction]
}
// precalculated for performance
var exp232 = math.Exp2(32)
// Encode the position of x within the range -r to +r as a 32-bit integer.
func encodeRange(x, r float64) uint32 {
p := (x + r) / (2 * r)
return uint32(p * exp232)
}
// Decode the 32-bit range encoding X back to a value in the range -r to +r.
func decodeRange(X uint32, r float64) float64 {
p := float64(X) / exp232
x := 2*r*p - r
return x
}
// Spread out the 32 bits of x into 64 bits, where the bits of x occupy even
// bit positions.
func spread(x uint32) uint64 {
X := uint64(x)
X = (X | (X << 16)) & 0x0000ffff0000ffff
X = (X | (X << 8)) & 0x00ff00ff00ff00ff
X = (X | (X << 4)) & 0x0f0f0f0f0f0f0f0f
X = (X | (X << 2)) & 0x3333333333333333
X = (X | (X << 1)) & 0x5555555555555555
return X
}
// Interleave the bits of x and y. In the result, x and y occupy even and odd
// bitlevels, respectively.
func interleave(x, y uint32) uint64 {
return spread(x) | (spread(y) << 1)
}
// Squash the even bitlevels of X into a 32-bit word. Odd bitlevels of X are
// ignored, and may take any value.
func squash(X uint64) uint32 {
X &= 0x5555555555555555
X = (X | (X >> 1)) & 0x3333333333333333
X = (X | (X >> 2)) & 0x0f0f0f0f0f0f0f0f
X = (X | (X >> 4)) & 0x00ff00ff00ff00ff
X = (X | (X >> 8)) & 0x0000ffff0000ffff
X = (X | (X >> 16)) & 0x00000000ffffffff
return uint32(X)
}
// Deinterleave the bits of X into 32-bit words containing the even and odd
// bitlevels of X, respectively.
func deinterleave(X uint64) (uint32, uint32) {
return squash(X), squash(X >> 1)
}
+24
View File
@@ -0,0 +1,24 @@
This is free and unencumbered software released into the public domain.
Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.
In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
For more information, please refer to <http://unlicense.org/>
+1
View File
@@ -0,0 +1 @@
Copied from https://github.com/layeh/gopher-json and https://github.com/alicebob/gopher-json
+189
View File
@@ -0,0 +1,189 @@
package json
import (
"encoding/json"
"errors"
"github.com/yuin/gopher-lua"
)
// Preload adds json to the given Lua state's package.preload table. After it
// has been preloaded, it can be loaded using require:
//
// local json = require("json")
func Preload(L *lua.LState) {
L.PreloadModule("json", Loader)
}
// Loader is the module loader function.
func Loader(L *lua.LState) int {
t := L.NewTable()
L.SetFuncs(t, api)
L.Push(t)
return 1
}
var api = map[string]lua.LGFunction{
"decode": apiDecode,
"encode": apiEncode,
}
func apiDecode(L *lua.LState) int {
if L.GetTop() != 1 {
L.Error(lua.LString("bad argument #1 to decode"), 1)
return 0
}
str := L.CheckString(1)
value, err := Decode(L, []byte(str))
if err != nil {
L.Push(lua.LNil)
L.Push(lua.LString(err.Error()))
return 2
}
L.Push(value)
return 1
}
func apiEncode(L *lua.LState) int {
if L.GetTop() != 1 {
L.Error(lua.LString("bad argument #1 to encode"), 1)
return 0
}
value := L.CheckAny(1)
data, err := Encode(value)
if err != nil {
L.Push(lua.LNil)
L.Push(lua.LString(err.Error()))
return 2
}
L.Push(lua.LString(string(data)))
return 1
}
var (
errNested = errors.New("cannot encode recursively nested tables to JSON")
errSparseArray = errors.New("cannot encode sparse array")
errInvalidKeys = errors.New("cannot encode mixed or invalid key types")
)
type invalidTypeError lua.LValueType
func (i invalidTypeError) Error() string {
return `cannot encode ` + lua.LValueType(i).String() + ` to JSON`
}
// Encode returns the JSON encoding of value.
func Encode(value lua.LValue) ([]byte, error) {
return json.Marshal(jsonValue{
LValue: value,
visited: make(map[*lua.LTable]bool),
})
}
type jsonValue struct {
lua.LValue
visited map[*lua.LTable]bool
}
func (j jsonValue) MarshalJSON() (data []byte, err error) {
switch converted := j.LValue.(type) {
case lua.LBool:
data, err = json.Marshal(bool(converted))
case lua.LNumber:
data, err = json.Marshal(float64(converted))
case *lua.LNilType:
data = []byte(`null`)
case lua.LString:
data, err = json.Marshal(string(converted))
case *lua.LTable:
if j.visited[converted] {
return nil, errNested
}
j.visited[converted] = true
key, value := converted.Next(lua.LNil)
switch key.Type() {
case lua.LTNil: // empty table
data = []byte(`[]`)
case lua.LTNumber:
arr := make([]jsonValue, 0, converted.Len())
expectedKey := lua.LNumber(1)
for key != lua.LNil {
if key.Type() != lua.LTNumber {
err = errInvalidKeys
return
}
if expectedKey != key {
err = errSparseArray
return
}
arr = append(arr, jsonValue{value, j.visited})
expectedKey++
key, value = converted.Next(key)
}
data, err = json.Marshal(arr)
case lua.LTString:
obj := make(map[string]jsonValue)
for key != lua.LNil {
if key.Type() != lua.LTString {
err = errInvalidKeys
return
}
obj[key.String()] = jsonValue{value, j.visited}
key, value = converted.Next(key)
}
data, err = json.Marshal(obj)
default:
err = errInvalidKeys
}
default:
err = invalidTypeError(j.LValue.Type())
}
return
}
// Decode converts the JSON encoded data to Lua values.
func Decode(L *lua.LState, data []byte) (lua.LValue, error) {
var value interface{}
err := json.Unmarshal(data, &value)
if err != nil {
return nil, err
}
return DecodeValue(L, value), nil
}
// DecodeValue converts the value to a Lua value.
//
// This function only converts values that the encoding/json package decodes to.
// All other values will return lua.LNil.
func DecodeValue(L *lua.LState, value interface{}) lua.LValue {
switch converted := value.(type) {
case bool:
return lua.LBool(converted)
case float64:
return lua.LNumber(converted)
case string:
return lua.LString(converted)
case json.Number:
return lua.LString(converted)
case []interface{}:
arr := L.CreateTable(len(converted), 0)
for _, item := range converted {
arr.Append(DecodeValue(L, item))
}
return arr
case map[string]interface{}:
tbl := L.CreateTable(0, len(converted))
for key, item := range converted {
tbl.RawSetH(lua.LString(key), DecodeValue(L, item))
}
return tbl
case nil:
return lua.LNil
}
return lua.LNil
}
+42
View File
@@ -0,0 +1,42 @@
package miniredis
import (
"github.com/alicebob/miniredis/v2/hyperloglog"
)
type hll struct {
inner *hyperloglog.Sketch
}
func newHll() *hll {
return &hll{
inner: hyperloglog.New14(),
}
}
// Add returns true if cardinality has been changed, or false otherwise.
func (h *hll) Add(item []byte) bool {
return h.inner.Insert(item)
}
// Count returns the estimation of a set cardinality.
func (h *hll) Count() int {
return int(h.inner.Estimate())
}
// Merge merges the other hll into original one (not making a copy but doing this in place).
func (h *hll) Merge(other *hll) {
_ = h.inner.Merge(other.inner)
}
// Bytes returns raw-bytes representation of hll data structure.
func (h *hll) Bytes() []byte {
dataBytes, _ := h.inner.MarshalBinary()
return dataBytes
}
func (h *hll) copy() *hll {
return &hll{
inner: h.inner.Clone(),
}
}
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2017 Axiom Inc. <[email protected]>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+1
View File
@@ -0,0 +1 @@
This is a copy of github.com/axiomhq/hyperloglog.
+180
View File
@@ -0,0 +1,180 @@
package hyperloglog
import "encoding/binary"
// Original author of this file is github.com/clarkduvall/hyperloglog
type iterable interface {
decode(i int, last uint32) (uint32, int)
Len() int
Iter() *iterator
}
type iterator struct {
i int
last uint32
v iterable
}
func (iter *iterator) Next() uint32 {
n, i := iter.v.decode(iter.i, iter.last)
iter.last = n
iter.i = i
return n
}
func (iter *iterator) Peek() uint32 {
n, _ := iter.v.decode(iter.i, iter.last)
return n
}
func (iter iterator) HasNext() bool {
return iter.i < iter.v.Len()
}
type compressedList struct {
count uint32
last uint32
b variableLengthList
}
func (v *compressedList) Clone() *compressedList {
if v == nil {
return nil
}
newV := &compressedList{
count: v.count,
last: v.last,
}
newV.b = make(variableLengthList, len(v.b))
copy(newV.b, v.b)
return newV
}
func (v *compressedList) MarshalBinary() (data []byte, err error) {
// Marshal the variableLengthList
bdata, err := v.b.MarshalBinary()
if err != nil {
return nil, err
}
// At least 4 bytes for the two fixed sized values plus the size of bdata.
data = make([]byte, 0, 4+4+len(bdata))
// Marshal the count and last values.
data = append(data, []byte{
// Number of items in the list.
byte(v.count >> 24),
byte(v.count >> 16),
byte(v.count >> 8),
byte(v.count),
// The last item in the list.
byte(v.last >> 24),
byte(v.last >> 16),
byte(v.last >> 8),
byte(v.last),
}...)
// Append the list
return append(data, bdata...), nil
}
func (v *compressedList) UnmarshalBinary(data []byte) error {
if len(data) < 12 {
return ErrorTooShort
}
// Set the count.
v.count, data = binary.BigEndian.Uint32(data[:4]), data[4:]
// Set the last value.
v.last, data = binary.BigEndian.Uint32(data[:4]), data[4:]
// Set the list.
sz, data := binary.BigEndian.Uint32(data[:4]), data[4:]
v.b = make([]uint8, sz)
if uint32(len(data)) < sz {
return ErrorTooShort
}
for i := uint32(0); i < sz; i++ {
v.b[i] = data[i]
}
return nil
}
func newCompressedList() *compressedList {
v := &compressedList{}
v.b = make(variableLengthList, 0)
return v
}
func (v *compressedList) Len() int {
return len(v.b)
}
func (v *compressedList) decode(i int, last uint32) (uint32, int) {
n, i := v.b.decode(i, last)
return n + last, i
}
func (v *compressedList) Append(x uint32) {
v.count++
v.b = v.b.Append(x - v.last)
v.last = x
}
func (v *compressedList) Iter() *iterator {
return &iterator{0, 0, v}
}
type variableLengthList []uint8
func (v variableLengthList) MarshalBinary() (data []byte, err error) {
// 4 bytes for the size of the list, and a byte for each element in the
// list.
data = make([]byte, 0, 4+v.Len())
// Length of the list. We only need 32 bits because the size of the set
// couldn't exceed that on 32 bit architectures.
sz := v.Len()
data = append(data, []byte{
byte(sz >> 24),
byte(sz >> 16),
byte(sz >> 8),
byte(sz),
}...)
// Marshal each element in the list.
for i := 0; i < sz; i++ {
data = append(data, v[i])
}
return data, nil
}
func (v variableLengthList) Len() int {
return len(v)
}
func (v *variableLengthList) Iter() *iterator {
return &iterator{0, 0, v}
}
func (v variableLengthList) decode(i int, last uint32) (uint32, int) {
var x uint32
j := i
for ; v[j]&0x80 != 0; j++ {
x |= uint32(v[j]&0x7f) << (uint(j-i) * 7)
}
x |= uint32(v[j]) << (uint(j-i) * 7)
return x, j + 1
}
func (v variableLengthList) Append(x uint32) variableLengthList {
for x&0xffffff80 != 0 {
v = append(v, uint8((x&0x7f)|0x80))
x >>= 7
}
return append(v, uint8(x&0x7f))
}
+424
View File
@@ -0,0 +1,424 @@
package hyperloglog
import (
"encoding/binary"
"errors"
"fmt"
"math"
"sort"
)
const (
capacity = uint8(16)
pp = uint8(25)
mp = uint32(1) << pp
version = 1
)
// Sketch is a HyperLogLog data-structure for the count-distinct problem,
// approximating the number of distinct elements in a multiset.
type Sketch struct {
p uint8
b uint8
m uint32
alpha float64
tmpSet set
sparseList *compressedList
regs *registers
}
// New returns a HyperLogLog Sketch with 2^14 registers (precision 14)
func New() *Sketch {
return New14()
}
// New14 returns a HyperLogLog Sketch with 2^14 registers (precision 14)
func New14() *Sketch {
sk, _ := newSketch(14, true)
return sk
}
// New16 returns a HyperLogLog Sketch with 2^16 registers (precision 16)
func New16() *Sketch {
sk, _ := newSketch(16, true)
return sk
}
// NewNoSparse returns a HyperLogLog Sketch with 2^14 registers (precision 14)
// that will not use a sparse representation
func NewNoSparse() *Sketch {
sk, _ := newSketch(14, false)
return sk
}
// New16NoSparse returns a HyperLogLog Sketch with 2^16 registers (precision 16)
// that will not use a sparse representation
func New16NoSparse() *Sketch {
sk, _ := newSketch(16, false)
return sk
}
// newSketch returns a HyperLogLog Sketch with 2^precision registers
func newSketch(precision uint8, sparse bool) (*Sketch, error) {
if precision < 4 || precision > 18 {
return nil, fmt.Errorf("p has to be >= 4 and <= 18")
}
m := uint32(math.Pow(2, float64(precision)))
s := &Sketch{
m: m,
p: precision,
alpha: alpha(float64(m)),
}
if sparse {
s.tmpSet = set{}
s.sparseList = newCompressedList()
} else {
s.regs = newRegisters(m)
}
return s, nil
}
func (sk *Sketch) sparse() bool {
return sk.sparseList != nil
}
// Clone returns a deep copy of sk.
func (sk *Sketch) Clone() *Sketch {
return &Sketch{
b: sk.b,
p: sk.p,
m: sk.m,
alpha: sk.alpha,
tmpSet: sk.tmpSet.Clone(),
sparseList: sk.sparseList.Clone(),
regs: sk.regs.clone(),
}
}
// Converts to normal if the sparse list is too large.
func (sk *Sketch) maybeToNormal() {
if uint32(len(sk.tmpSet))*100 > sk.m {
sk.mergeSparse()
if uint32(sk.sparseList.Len()) > sk.m {
sk.toNormal()
}
}
}
// Merge takes another Sketch and combines it with Sketch h.
// If Sketch h is using the sparse Sketch, it will be converted
// to the normal Sketch.
func (sk *Sketch) Merge(other *Sketch) error {
if other == nil {
// Nothing to do
return nil
}
cpOther := other.Clone()
if sk.p != cpOther.p {
return errors.New("precisions must be equal")
}
if sk.sparse() && other.sparse() {
for k := range other.tmpSet {
sk.tmpSet.add(k)
}
for iter := other.sparseList.Iter(); iter.HasNext(); {
sk.tmpSet.add(iter.Next())
}
sk.maybeToNormal()
return nil
}
if sk.sparse() {
sk.toNormal()
}
if cpOther.sparse() {
for k := range cpOther.tmpSet {
i, r := decodeHash(k, cpOther.p, pp)
sk.insert(i, r)
}
for iter := cpOther.sparseList.Iter(); iter.HasNext(); {
i, r := decodeHash(iter.Next(), cpOther.p, pp)
sk.insert(i, r)
}
} else {
if sk.b < cpOther.b {
sk.regs.rebase(cpOther.b - sk.b)
sk.b = cpOther.b
} else {
cpOther.regs.rebase(sk.b - cpOther.b)
cpOther.b = sk.b
}
for i, v := range cpOther.regs.tailcuts {
v1 := v.get(0)
if v1 > sk.regs.get(uint32(i)*2) {
sk.regs.set(uint32(i)*2, v1)
}
v2 := v.get(1)
if v2 > sk.regs.get(1+uint32(i)*2) {
sk.regs.set(1+uint32(i)*2, v2)
}
}
}
return nil
}
// Convert from sparse Sketch to dense Sketch.
func (sk *Sketch) toNormal() {
if len(sk.tmpSet) > 0 {
sk.mergeSparse()
}
sk.regs = newRegisters(sk.m)
for iter := sk.sparseList.Iter(); iter.HasNext(); {
i, r := decodeHash(iter.Next(), sk.p, pp)
sk.insert(i, r)
}
sk.tmpSet = nil
sk.sparseList = nil
}
func (sk *Sketch) insert(i uint32, r uint8) bool {
changed := false
if r-sk.b >= capacity {
//overflow
db := sk.regs.min()
if db > 0 {
sk.b += db
sk.regs.rebase(db)
changed = true
}
}
if r > sk.b {
val := r - sk.b
if c1 := capacity - 1; c1 < val {
val = c1
}
if val > sk.regs.get(i) {
sk.regs.set(i, val)
changed = true
}
}
return changed
}
// Insert adds element e to sketch
func (sk *Sketch) Insert(e []byte) bool {
x := hash(e)
return sk.InsertHash(x)
}
// InsertHash adds hash x to sketch
func (sk *Sketch) InsertHash(x uint64) bool {
if sk.sparse() {
changed := sk.tmpSet.add(encodeHash(x, sk.p, pp))
if !changed {
return false
}
if uint32(len(sk.tmpSet))*100 > sk.m/2 {
sk.mergeSparse()
if uint32(sk.sparseList.Len()) > sk.m/2 {
sk.toNormal()
}
}
return true
} else {
i, r := getPosVal(x, sk.p)
return sk.insert(uint32(i), r)
}
}
// Estimate returns the cardinality of the Sketch
func (sk *Sketch) Estimate() uint64 {
if sk.sparse() {
sk.mergeSparse()
return uint64(linearCount(mp, mp-sk.sparseList.count))
}
sum, ez := sk.regs.sumAndZeros(sk.b)
m := float64(sk.m)
var est float64
var beta func(float64) float64
if sk.p < 16 {
beta = beta14
} else {
beta = beta16
}
if sk.b == 0 {
est = (sk.alpha * m * (m - ez) / (sum + beta(ez)))
} else {
est = (sk.alpha * m * m / sum)
}
return uint64(est + 0.5)
}
func (sk *Sketch) mergeSparse() {
if len(sk.tmpSet) == 0 {
return
}
keys := make(uint64Slice, 0, len(sk.tmpSet))
for k := range sk.tmpSet {
keys = append(keys, k)
}
sort.Sort(keys)
newList := newCompressedList()
for iter, i := sk.sparseList.Iter(), 0; iter.HasNext() || i < len(keys); {
if !iter.HasNext() {
newList.Append(keys[i])
i++
continue
}
if i >= len(keys) {
newList.Append(iter.Next())
continue
}
x1, x2 := iter.Peek(), keys[i]
if x1 == x2 {
newList.Append(iter.Next())
i++
} else if x1 > x2 {
newList.Append(x2)
i++
} else {
newList.Append(iter.Next())
}
}
sk.sparseList = newList
sk.tmpSet = set{}
}
// MarshalBinary implements the encoding.BinaryMarshaler interface.
func (sk *Sketch) MarshalBinary() (data []byte, err error) {
// Marshal a version marker.
data = append(data, version)
// Marshal p.
data = append(data, sk.p)
// Marshal b
data = append(data, sk.b)
if sk.sparse() {
// It's using the sparse Sketch.
data = append(data, byte(1))
// Add the tmp_set
tsdata, err := sk.tmpSet.MarshalBinary()
if err != nil {
return nil, err
}
data = append(data, tsdata...)
// Add the sparse Sketch
sdata, err := sk.sparseList.MarshalBinary()
if err != nil {
return nil, err
}
return append(data, sdata...), nil
}
// It's using the dense Sketch.
data = append(data, byte(0))
// Add the dense sketch Sketch.
sz := len(sk.regs.tailcuts)
data = append(data, []byte{
byte(sz >> 24),
byte(sz >> 16),
byte(sz >> 8),
byte(sz),
}...)
// Marshal each element in the list.
for i := 0; i < len(sk.regs.tailcuts); i++ {
data = append(data, byte(sk.regs.tailcuts[i]))
}
return data, nil
}
// ErrorTooShort is an error that UnmarshalBinary try to parse too short
// binary.
var ErrorTooShort = errors.New("too short binary")
// UnmarshalBinary implements the encoding.BinaryUnmarshaler interface.
func (sk *Sketch) UnmarshalBinary(data []byte) error {
if len(data) < 8 {
return ErrorTooShort
}
// Unmarshal version. We may need this in the future if we make
// non-compatible changes.
_ = data[0]
// Unmarshal p.
p := data[1]
// Unmarshal b.
sk.b = data[2]
// Determine if we need a sparse Sketch
sparse := data[3] == byte(1)
// Make a newSketch Sketch if the precision doesn't match or if the Sketch was used
if sk.p != p || sk.regs != nil || len(sk.tmpSet) > 0 || (sk.sparseList != nil && sk.sparseList.Len() > 0) {
newh, err := newSketch(p, sparse)
if err != nil {
return err
}
newh.b = sk.b
*sk = *newh
}
// h is now initialised with the correct p. We just need to fill the
// rest of the details out.
if sparse {
// Using the sparse Sketch.
// Unmarshal the tmp_set.
tssz := binary.BigEndian.Uint32(data[4:8])
sk.tmpSet = make(map[uint32]struct{}, tssz)
// We need to unmarshal tssz values in total, and each value requires us
// to read 4 bytes.
tsLastByte := int((tssz * 4) + 8)
for i := 8; i < tsLastByte; i += 4 {
k := binary.BigEndian.Uint32(data[i : i+4])
sk.tmpSet[k] = struct{}{}
}
// Unmarshal the sparse Sketch.
return sk.sparseList.UnmarshalBinary(data[tsLastByte:])
}
// Using the dense Sketch.
sk.sparseList = nil
sk.tmpSet = nil
dsz := binary.BigEndian.Uint32(data[4:8])
sk.regs = newRegisters(dsz * 2)
data = data[8:]
for i, val := range data {
sk.regs.tailcuts[i] = reg(val)
if uint8(sk.regs.tailcuts[i]<<4>>4) > 0 {
sk.regs.nz--
}
if uint8(sk.regs.tailcuts[i]>>4) > 0 {
sk.regs.nz--
}
}
return nil
}
+114
View File
@@ -0,0 +1,114 @@
package hyperloglog
import (
"math"
)
type reg uint8
type tailcuts []reg
type registers struct {
tailcuts
nz uint32
}
func (r *reg) set(offset, val uint8) bool {
var isZero bool
if offset == 0 {
isZero = *r < 16
tmpVal := uint8((*r) << 4 >> 4)
*r = reg(tmpVal | (val << 4))
} else {
isZero = *r&0x0f == 0
tmpVal := uint8((*r) >> 4 << 4)
*r = reg(tmpVal | val)
}
return isZero
}
func (r *reg) get(offset uint8) uint8 {
if offset == 0 {
return uint8((*r) >> 4)
}
return uint8((*r) << 4 >> 4)
}
func newRegisters(size uint32) *registers {
return &registers{
tailcuts: make(tailcuts, size/2),
nz: size,
}
}
func (rs *registers) clone() *registers {
if rs == nil {
return nil
}
tc := make([]reg, len(rs.tailcuts))
copy(tc, rs.tailcuts)
return &registers{
tailcuts: tc,
nz: rs.nz,
}
}
func (rs *registers) rebase(delta uint8) {
nz := uint32(len(rs.tailcuts)) * 2
for i := range rs.tailcuts {
for j := uint8(0); j < 2; j++ {
val := rs.tailcuts[i].get(j)
if val >= delta {
rs.tailcuts[i].set(j, val-delta)
if val-delta > 0 {
nz--
}
}
}
}
rs.nz = nz
}
func (rs *registers) set(i uint32, val uint8) {
offset, index := uint8(i)&1, i/2
if rs.tailcuts[index].set(offset, val) {
rs.nz--
}
}
func (rs *registers) get(i uint32) uint8 {
offset, index := uint8(i)&1, i/2
return rs.tailcuts[index].get(offset)
}
func (rs *registers) sumAndZeros(base uint8) (res, ez float64) {
for _, r := range rs.tailcuts {
for j := uint8(0); j < 2; j++ {
v := float64(base + r.get(j))
if v == 0 {
ez++
}
res += 1.0 / math.Pow(2.0, v)
}
}
rs.nz = uint32(ez)
return res, ez
}
func (rs *registers) min() uint8 {
if rs.nz > 0 {
return 0
}
min := uint8(math.MaxUint8)
for _, r := range rs.tailcuts {
if r == 0 || min == 0 {
return 0
}
if val := uint8(r << 4 >> 4); val < min {
min = val
}
if val := uint8(r >> 4); val < min {
min = val
}
}
return min
}
+92
View File
@@ -0,0 +1,92 @@
package hyperloglog
import (
"math/bits"
)
func getIndex(k uint32, p, pp uint8) uint32 {
if k&1 == 1 {
return bextr32(k, 32-p, p)
}
return bextr32(k, pp-p+1, p)
}
// Encode a hash to be used in the sparse representation.
func encodeHash(x uint64, p, pp uint8) uint32 {
idx := uint32(bextr(x, 64-pp, pp))
if bextr(x, 64-pp, pp-p) == 0 {
zeros := bits.LeadingZeros64((bextr(x, 0, 64-pp)<<pp)|(1<<pp-1)) + 1
return idx<<7 | uint32(zeros<<1) | 1
}
return idx << 1
}
// Decode a hash from the sparse representation.
func decodeHash(k uint32, p, pp uint8) (uint32, uint8) {
var r uint8
if k&1 == 1 {
r = uint8(bextr32(k, 1, 6)) + pp - p
} else {
// We can use the 64bit clz implementation and reduce the result
// by 32 to get a clz for a 32bit word.
r = uint8(bits.LeadingZeros64(uint64(k<<(32-pp+p-1))) - 31) // -32 + 1
}
return getIndex(k, p, pp), r
}
type set map[uint32]struct{}
func (s set) add(v uint32) bool {
_, ok := s[v]
if ok {
return false
}
s[v] = struct{}{}
return true
}
func (s set) Clone() set {
if s == nil {
return nil
}
newS := make(map[uint32]struct{}, len(s))
for k, v := range s {
newS[k] = v
}
return newS
}
func (s set) MarshalBinary() (data []byte, err error) {
// 4 bytes for the size of the set, and 4 bytes for each key.
// list.
data = make([]byte, 0, 4+(4*len(s)))
// Length of the set. We only need 32 bits because the size of the set
// couldn't exceed that on 32 bit architectures.
sl := len(s)
data = append(data, []byte{
byte(sl >> 24),
byte(sl >> 16),
byte(sl >> 8),
byte(sl),
}...)
// Marshal each element in the set.
for k := range s {
data = append(data, []byte{
byte(k >> 24),
byte(k >> 16),
byte(k >> 8),
byte(k),
}...)
}
return data, nil
}
type uint64Slice []uint32
func (p uint64Slice) Len() int { return len(p) }
func (p uint64Slice) Less(i, j int) bool { return p[i] < p[j] }
func (p uint64Slice) Swap(i, j int) { p[i], p[j] = p[j], p[i] }
+69
View File
@@ -0,0 +1,69 @@
package hyperloglog
import (
"github.com/alicebob/miniredis/v2/metro"
"math"
"math/bits"
)
var hash = hashFunc
func beta14(ez float64) float64 {
zl := math.Log(ez + 1)
return -0.370393911*ez +
0.070471823*zl +
0.17393686*math.Pow(zl, 2) +
0.16339839*math.Pow(zl, 3) +
-0.09237745*math.Pow(zl, 4) +
0.03738027*math.Pow(zl, 5) +
-0.005384159*math.Pow(zl, 6) +
0.00042419*math.Pow(zl, 7)
}
func beta16(ez float64) float64 {
zl := math.Log(ez + 1)
return -0.37331876643753059*ez +
-1.41704077448122989*zl +
0.40729184796612533*math.Pow(zl, 2) +
1.56152033906584164*math.Pow(zl, 3) +
-0.99242233534286128*math.Pow(zl, 4) +
0.26064681399483092*math.Pow(zl, 5) +
-0.03053811369682807*math.Pow(zl, 6) +
0.00155770210179105*math.Pow(zl, 7)
}
func alpha(m float64) float64 {
switch m {
case 16:
return 0.673
case 32:
return 0.697
case 64:
return 0.709
}
return 0.7213 / (1 + 1.079/m)
}
func getPosVal(x uint64, p uint8) (uint64, uint8) {
i := bextr(x, 64-p, p) // {x63,...,x64-p}
w := x<<p | 1<<(p-1) // {x63-p,...,x0}
rho := uint8(bits.LeadingZeros64(w)) + 1
return i, rho
}
func linearCount(m uint32, v uint32) float64 {
fm := float64(m)
return fm * math.Log(fm/float64(v))
}
func bextr(v uint64, start, length uint8) uint64 {
return (v >> start) & ((1 << length) - 1)
}
func bextr32(v uint32, start, length uint8) uint32 {
return (v >> start) & ((1 << length) - 1)
}
func hashFunc(e []byte) uint64 {
return metro.Hash64(e, 1337)
}
+83
View File
@@ -0,0 +1,83 @@
package miniredis
// Translate the 'KEYS' or 'PSUBSCRIBE' argument ('foo*', 'f??', &c.) into a regexp.
import (
"bytes"
"regexp"
)
// patternRE compiles a glob to a regexp. Returns nil if the given
// pattern will never match anything.
// The general strategy is to sandwich all non-meta characters between \Q...\E.
func patternRE(k string) *regexp.Regexp {
re := bytes.Buffer{}
re.WriteString(`(?s)^\Q`)
for i := 0; i < len(k); i++ {
p := k[i]
switch p {
case '*':
re.WriteString(`\E.*\Q`)
case '?':
re.WriteString(`\E.\Q`)
case '[':
charClass := bytes.Buffer{}
i++
for ; i < len(k); i++ {
if k[i] == ']' {
break
}
if k[i] == '\\' {
if i == len(k)-1 {
// Ends with a '\'. U-huh.
return nil
}
charClass.WriteByte(k[i])
i++
charClass.WriteByte(k[i])
continue
}
charClass.WriteByte(k[i])
}
if charClass.Len() == 0 {
// '[]' is valid in Redis, but matches nothing.
return nil
}
re.WriteString(`\E[`)
re.Write(charClass.Bytes())
re.WriteString(`]\Q`)
case '\\':
if i == len(k)-1 {
// Ends with a '\'. U-huh.
return nil
}
// Forget the \, keep the next char.
i++
re.WriteByte(k[i])
continue
default:
re.WriteByte(p)
}
}
re.WriteString(`\E$`)
return regexp.MustCompile(re.String())
}
// matchKeys filters only matching keys.
// The returned boolean is whether the match pattern was valid
func matchKeys(keys []string, match string) ([]string, bool) {
re := patternRE(match)
if re == nil {
// Special case: the given pattern won't match anything or is invalid.
return nil, false
}
var res []string
for _, k := range keys {
if !re.MatchString(k) {
continue
}
res = append(res, k)
}
return res, true
}
+281
View File
@@ -0,0 +1,281 @@
package miniredis
import (
"bufio"
"bytes"
"fmt"
"strings"
lua "github.com/yuin/gopher-lua"
"github.com/alicebob/miniredis/v2/server"
)
var luaRedisConstants = map[string]lua.LValue{
"LOG_DEBUG": lua.LNumber(0),
"LOG_VERBOSE": lua.LNumber(1),
"LOG_NOTICE": lua.LNumber(2),
"LOG_WARNING": lua.LNumber(3),
}
func mkLua(srv *server.Server, c *server.Peer, sha string) (map[string]lua.LGFunction, map[string]lua.LValue) {
mkCall := func(failFast bool) func(l *lua.LState) int {
// one server.Ctx for a single Lua run
pCtx := &connCtx{}
if getCtx(c).authenticated {
pCtx.authenticated = true
}
pCtx.nested = true
pCtx.nestedSHA = sha
pCtx.selectedDB = getCtx(c).selectedDB
return func(l *lua.LState) int {
top := l.GetTop()
if top == 0 {
l.Error(lua.LString(fmt.Sprintf("Please specify at least one argument for this redis lib call script: %s, &c.", sha)), 1)
return 0
}
var args []string
for i := 1; i <= top; i++ {
switch a := l.Get(i).(type) {
case lua.LNumber:
args = append(args, a.String())
case lua.LString:
args = append(args, string(a))
default:
l.Error(lua.LString(fmt.Sprintf("Lua redis lib command arguments must be strings or integers script: %s, &c.", sha)), 1)
return 0
}
}
if len(args) == 0 {
l.Error(lua.LString(msgNotFromScripts(sha)), 1)
return 0
}
buf := &bytes.Buffer{}
wr := bufio.NewWriter(buf)
peer := server.NewPeer(wr)
peer.Ctx = pCtx
srv.Dispatch(peer, args)
wr.Flush()
res, err := server.ParseReply(bufio.NewReader(buf))
if err != nil {
if failFast {
// call() mode
if strings.Contains(err.Error(), "ERR unknown command") {
l.Error(lua.LString(fmt.Sprintf("Unknown Redis command called from script script: %s, &c.", sha)), 1)
} else {
l.Error(lua.LString(err.Error()), 1)
}
return 0
}
// pcall() mode
l.Push(lua.LNil)
return 1
}
if res == nil {
l.Push(lua.LFalse)
} else {
switch r := res.(type) {
case int64:
l.Push(lua.LNumber(r))
case int:
l.Push(lua.LNumber(r))
case []uint8:
l.Push(lua.LString(string(r)))
case []interface{}:
l.Push(redisToLua(l, r))
case server.Simple:
l.Push(luaStatusReply(string(r)))
case string:
l.Push(lua.LString(r))
case error:
l.Error(lua.LString(r.Error()), 1)
return 0
default:
panic(fmt.Sprintf("type not handled (%T)", r))
}
}
return 1
}
}
return map[string]lua.LGFunction{
"call": mkCall(true),
"pcall": mkCall(false),
"error_reply": func(l *lua.LState) int {
v := l.Get(1)
msg, ok := v.(lua.LString)
if !ok {
l.Error(lua.LString("wrong number or type of arguments"), 1)
return 0
}
res := &lua.LTable{}
parts := strings.SplitN(msg.String(), " ", 2)
// '-' at the beginging will be added as a part of error response
if parts[0] != "" && parts[0][0] == '-' {
parts[0] = parts[0][1:]
}
var final_msg string
if len(parts) == 2 {
final_msg = fmt.Sprintf("%s %s", parts[0], parts[1])
} else {
final_msg = fmt.Sprintf("ERR %s", parts[0])
}
res.RawSetString("err", lua.LString(final_msg))
l.Push(res)
return 1
},
"log": func(l *lua.LState) int {
level := l.CheckInt(1)
msg := l.CheckString(2)
_, _ = level, msg
// do nothing by default. To see logs uncomment:
// fmt.Printf("%v: %v", level, msg)
return 0
},
"status_reply": func(l *lua.LState) int {
v := l.Get(1)
msg, ok := v.(lua.LString)
if !ok {
l.Error(lua.LString("wrong number or type of arguments"), 1)
return 0
}
res := luaStatusReply(string(msg))
l.Push(res)
return 1
},
"sha1hex": func(l *lua.LState) int {
top := l.GetTop()
if top != 1 {
l.Error(lua.LString("wrong number of arguments"), 1)
return 0
}
msg := lua.LVAsString(l.Get(1))
l.Push(lua.LString(sha1Hex(msg)))
return 1
},
"replicate_commands": func(l *lua.LState) int {
// always succeeds since 7.0.0
l.Push(lua.LTrue)
return 1
},
"set_repl": func(l *lua.LState) int {
top := l.GetTop()
if top != 1 {
l.Error(lua.LString("wrong number of arguments"), 1)
return 0
}
// ignored
return 1
},
"setresp": func(l *lua.LState) int {
level := l.CheckInt(1)
toresp3 := false
switch level {
case 2:
toresp3 = false
case 3:
toresp3 = true
default:
l.Error(lua.LString("RESP version must be 2 or 3"), 1)
return 0
}
c.SwitchResp3 = &toresp3
return 0
},
}, luaRedisConstants
}
func luaToRedis(l *lua.LState, c *server.Peer, value lua.LValue) {
if value == nil {
c.WriteNull()
return
}
switch t := value.(type) {
case *lua.LNilType:
c.WriteNull()
case lua.LBool:
if lua.LVAsBool(value) {
c.WriteInt(1)
} else {
c.WriteNull()
}
case lua.LNumber:
c.WriteInt(int(lua.LVAsNumber(value)))
case lua.LString:
s := lua.LVAsString(value)
c.WriteBulk(s)
case *lua.LTable:
// special case for tables with an 'err' or 'ok' field
// note: according to the docs this only counts when 'err' or 'ok' is
// the only field.
if s := t.RawGetString("err"); s.Type() != lua.LTNil {
c.WriteError(s.String())
return
}
if s := t.RawGetString("ok"); s.Type() != lua.LTNil {
c.WriteInline(s.String())
return
}
result := []lua.LValue{}
for j := 1; true; j++ {
val := l.GetTable(value, lua.LNumber(j))
if val == nil {
result = append(result, val)
continue
}
if val.Type() == lua.LTNil {
break
}
result = append(result, val)
}
c.WriteLen(len(result))
for _, r := range result {
luaToRedis(l, c, r)
}
default:
panic(fmt.Sprintf("wat: %T", t))
}
}
func redisToLua(l *lua.LState, res []interface{}) *lua.LTable {
rettb := l.NewTable()
for _, e := range res {
var v lua.LValue
if e == nil {
v = lua.LFalse
} else {
switch et := e.(type) {
case int:
v = lua.LNumber(et)
case int64:
v = lua.LNumber(et)
case []uint8:
v = lua.LString(string(et))
case []interface{}:
v = redisToLua(l, et)
case string:
v = lua.LString(et)
default:
// TODO: oops?
v = lua.LString(e.(string))
}
}
l.RawSet(rettb, lua.LNumber(rettb.Len()+1), v)
}
return rettb
}
func luaStatusReply(msg string) *lua.LTable {
tab := &lua.LTable{}
tab.RawSetString("ok", lua.LString(msg))
return tab
}
+24
View File
@@ -0,0 +1,24 @@
This package is a mechanical translation of the reference C++ code for
MetroHash, available at https://github.com/jandrewrogers/MetroHash
The MIT License (MIT)
Copyright (c) 2016 Damian Gryski
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+1
View File
@@ -0,0 +1 @@
This is a partial copy of github.com/dgryski/go-metro.
+87
View File
@@ -0,0 +1,87 @@
package metro
import "encoding/binary"
func Hash64(buffer []byte, seed uint64) uint64 {
const (
k0 = 0xD6D018F5
k1 = 0xA2AA033B
k2 = 0x62992FC1
k3 = 0x30BC5B29
)
ptr := buffer
hash := (seed + k2) * k0
if len(ptr) >= 32 {
v := [4]uint64{hash, hash, hash, hash}
for len(ptr) >= 32 {
v[0] += binary.LittleEndian.Uint64(ptr[:8]) * k0
v[0] = rotate_right(v[0], 29) + v[2]
v[1] += binary.LittleEndian.Uint64(ptr[8:16]) * k1
v[1] = rotate_right(v[1], 29) + v[3]
v[2] += binary.LittleEndian.Uint64(ptr[16:24]) * k2
v[2] = rotate_right(v[2], 29) + v[0]
v[3] += binary.LittleEndian.Uint64(ptr[24:32]) * k3
v[3] = rotate_right(v[3], 29) + v[1]
ptr = ptr[32:]
}
v[2] ^= rotate_right(((v[0]+v[3])*k0)+v[1], 37) * k1
v[3] ^= rotate_right(((v[1]+v[2])*k1)+v[0], 37) * k0
v[0] ^= rotate_right(((v[0]+v[2])*k0)+v[3], 37) * k1
v[1] ^= rotate_right(((v[1]+v[3])*k1)+v[2], 37) * k0
hash += v[0] ^ v[1]
}
if len(ptr) >= 16 {
v0 := hash + (binary.LittleEndian.Uint64(ptr[:8]) * k2)
v0 = rotate_right(v0, 29) * k3
v1 := hash + (binary.LittleEndian.Uint64(ptr[8:16]) * k2)
v1 = rotate_right(v1, 29) * k3
v0 ^= rotate_right(v0*k0, 21) + v1
v1 ^= rotate_right(v1*k3, 21) + v0
hash += v1
ptr = ptr[16:]
}
if len(ptr) >= 8 {
hash += binary.LittleEndian.Uint64(ptr[:8]) * k3
ptr = ptr[8:]
hash ^= rotate_right(hash, 55) * k1
}
if len(ptr) >= 4 {
hash += uint64(binary.LittleEndian.Uint32(ptr[:4])) * k3
hash ^= rotate_right(hash, 26) * k1
ptr = ptr[4:]
}
if len(ptr) >= 2 {
hash += uint64(binary.LittleEndian.Uint16(ptr[:2])) * k3
ptr = ptr[2:]
hash ^= rotate_right(hash, 48) * k1
}
if len(ptr) >= 1 {
hash += uint64(ptr[0]) * k3
hash ^= rotate_right(hash, 37) * k1
}
hash ^= rotate_right(hash, 28)
hash *= k0
hash ^= rotate_right(hash, 29)
return hash
}
func Hash64Str(buffer string, seed uint64) uint64 {
return Hash64([]byte(buffer), seed)
}
func rotate_right(v uint64, k uint) uint64 {
return (v >> k) | (v << (64 - k))
}
+759
View File
@@ -0,0 +1,759 @@
// Package miniredis is a pure Go Redis test server, for use in Go unittests.
// There are no dependencies on system binaries, and every server you start
// will be empty.
//
// import "github.com/alicebob/miniredis/v2"
//
// Start a server with `s := miniredis.RunT(t)`, it'll be shutdown via a t.Cleanup().
// Or do everything manual: `s, err := miniredis.Run(); defer s.Close()`
//
// Point your Redis client to `s.Addr()` or `s.Host(), s.Port()`.
//
// Set keys directly via s.Set(...) and similar commands, or use a Redis client.
//
// For direct use you can select a Redis database with either `s.Select(12);
// s.Get("foo")` or `s.DB(12).Get("foo")`.
package miniredis
import (
"context"
"crypto/tls"
"fmt"
"math/rand"
"strconv"
"strings"
"sync"
"time"
"github.com/alicebob/miniredis/v2/proto"
"github.com/alicebob/miniredis/v2/server"
)
var DumpMaxLineLen = 60
type hashKey map[string]string
type listKey []string
type setKey map[string]struct{}
// RedisDB holds a single (numbered) Redis database.
type RedisDB struct {
master *Miniredis // pointer to the lock in Miniredis
id int // db id
keys map[string]string // Master map of keys with their type
stringKeys map[string]string // GET/SET &c. keys
hashKeys map[string]hashKey // MGET/MSET &c. keys
listKeys map[string]listKey // LPUSH &c. keys
setKeys map[string]setKey // SADD &c. keys
hllKeys map[string]*hll // PFADD &c. keys
sortedsetKeys map[string]sortedSet // ZADD &c. keys
streamKeys map[string]*streamKey // XADD &c. keys
ttl map[string]time.Duration // effective TTL values
lru map[string]time.Time // last recently used ( read or written to )
keyVersion map[string]uint // used to watch values
}
// Miniredis is a Redis server implementation.
type Miniredis struct {
sync.Mutex
srv *server.Server
port int
passwords map[string]string // username password
dbs map[int]*RedisDB
selectedDB int // DB id used in the direct Get(), Set() &c.
scripts map[string]string // sha1 -> lua src
signal *sync.Cond
now time.Time // time.Now() if not set.
subscribers map[*Subscriber]struct{}
rand *rand.Rand
Ctx context.Context
CtxCancel context.CancelFunc
}
type txCmd func(*server.Peer, *connCtx)
// database id + key combo
type dbKey struct {
db int
key string
}
// connCtx has all state for a single connection.
// (this struct was named before context.Context existed)
type connCtx struct {
selectedDB int // selected DB
authenticated bool // auth enabled and a valid AUTH seen
transaction []txCmd // transaction callbacks. Or nil.
dirtyTransaction bool // any error during QUEUEing
watch map[dbKey]uint // WATCHed keys
subscriber *Subscriber // client is in PUBSUB mode if not nil
nested bool // this is called via Lua
nestedSHA string // set to the SHA of the nesting function
}
// NewMiniRedis makes a new, non-started, Miniredis object.
func NewMiniRedis() *Miniredis {
m := Miniredis{
dbs: map[int]*RedisDB{},
scripts: map[string]string{},
subscribers: map[*Subscriber]struct{}{},
}
m.Ctx, m.CtxCancel = context.WithCancel(context.Background())
m.signal = sync.NewCond(&m)
return &m
}
func newRedisDB(id int, m *Miniredis) RedisDB {
return RedisDB{
id: id,
master: m,
keys: map[string]string{},
lru: map[string]time.Time{},
stringKeys: map[string]string{},
hashKeys: map[string]hashKey{},
listKeys: map[string]listKey{},
setKeys: map[string]setKey{},
hllKeys: map[string]*hll{},
sortedsetKeys: map[string]sortedSet{},
streamKeys: map[string]*streamKey{},
ttl: map[string]time.Duration{},
keyVersion: map[string]uint{},
}
}
// Run creates and Start()s a Miniredis.
func Run() (*Miniredis, error) {
m := NewMiniRedis()
return m, m.Start()
}
// Run creates and Start()s a Miniredis, TLS version.
func RunTLS(cfg *tls.Config) (*Miniredis, error) {
m := NewMiniRedis()
return m, m.StartTLS(cfg)
}
// Tester is a minimal version of a testing.T
type Tester interface {
Fatalf(string, ...interface{})
Cleanup(func())
Logf(format string, args ...interface{})
}
// RunT start a new miniredis, pass it a testing.T. It also registers the cleanup after your test is done.
func RunT(t Tester) *Miniredis {
m := NewMiniRedis()
if err := m.Start(); err != nil {
t.Fatalf("could not start miniredis: %s", err)
// not reached
}
t.Cleanup(m.Close)
return m
}
func runWithClient(t Tester) (*Miniredis, *proto.Client) {
m := RunT(t)
c, err := proto.Dial(m.Addr())
if err != nil {
t.Fatalf("could not connect to miniredis: %s", err)
}
t.Cleanup(func() {
if err = c.Close(); err != nil {
t.Logf("error closing connection to miniredis: %s", err)
}
})
return m, c
}
// Start starts a server. It listens on a random port on localhost. See also
// Addr().
func (m *Miniredis) Start() error {
s, err := server.NewServer(fmt.Sprintf("127.0.0.1:%d", m.port))
if err != nil {
return err
}
return m.start(s)
}
// Start starts a server, TLS version.
func (m *Miniredis) StartTLS(cfg *tls.Config) error {
s, err := server.NewServerTLS(fmt.Sprintf("127.0.0.1:%d", m.port), cfg)
if err != nil {
return err
}
return m.start(s)
}
// StartAddr runs miniredis with a given addr. Examples: "127.0.0.1:6379",
// ":6379", or "127.0.0.1:0"
func (m *Miniredis) StartAddr(addr string) error {
s, err := server.NewServer(addr)
if err != nil {
return err
}
return m.start(s)
}
// StartAddrTLS runs miniredis with a given addr, TLS version.
func (m *Miniredis) StartAddrTLS(addr string, cfg *tls.Config) error {
s, err := server.NewServerTLS(addr, cfg)
if err != nil {
return err
}
return m.start(s)
}
func (m *Miniredis) start(s *server.Server) error {
m.Lock()
defer m.Unlock()
m.srv = s
m.port = s.Addr().Port
commandsConnection(m)
commandsGeneric(m)
commandsServer(m)
commandsString(m)
commandsHash(m)
commandsList(m)
commandsPubsub(m)
commandsSet(m)
commandsSortedSet(m)
commandsStream(m)
commandsTransaction(m)
commandsScripting(m)
commandsGeo(m)
commandsCluster(m)
commandsHll(m)
commandsClient(m)
commandsObject(m)
return nil
}
// Restart restarts a Close()d server on the same port. Values will be
// preserved.
func (m *Miniredis) Restart() error {
return m.Start()
}
// Close shuts down a Miniredis.
func (m *Miniredis) Close() {
m.Lock()
if m.srv == nil {
m.Unlock()
return
}
srv := m.srv
m.srv = nil
m.CtxCancel()
m.Unlock()
// the OnDisconnect callbacks can lock m, so run Close() outside the lock.
srv.Close()
}
// RequireAuth makes every connection need to AUTH first. This is the old 'AUTH [password] command.
// Remove it by setting an empty string.
func (m *Miniredis) RequireAuth(pw string) {
m.RequireUserAuth("default", pw)
}
// Add a username/password, for use with 'AUTH [username] [password]'.
// There are currently no access controls for commands implemented.
// Disable access for the user with an empty password.
func (m *Miniredis) RequireUserAuth(username, pw string) {
m.Lock()
defer m.Unlock()
if m.passwords == nil {
m.passwords = map[string]string{}
}
if pw == "" {
delete(m.passwords, username)
return
}
m.passwords[username] = pw
}
// DB returns a DB by ID.
func (m *Miniredis) DB(i int) *RedisDB {
m.Lock()
defer m.Unlock()
return m.db(i)
}
// get DB. No locks!
func (m *Miniredis) db(i int) *RedisDB {
if db, ok := m.dbs[i]; ok {
return db
}
db := newRedisDB(i, m) // main miniredis has our mutex.
m.dbs[i] = &db
return &db
}
// SwapDB swaps DBs by IDs.
func (m *Miniredis) SwapDB(i, j int) bool {
m.Lock()
defer m.Unlock()
return m.swapDB(i, j)
}
// swap DB. No locks!
func (m *Miniredis) swapDB(i, j int) bool {
db1 := m.db(i)
db2 := m.db(j)
db1.id = j
db2.id = i
m.dbs[i] = db2
m.dbs[j] = db1
return true
}
// Addr returns '127.0.0.1:12345'. Can be given to a Dial(). See also Host()
// and Port(), which return the same things.
func (m *Miniredis) Addr() string {
m.Lock()
defer m.Unlock()
return m.srv.Addr().String()
}
// Host returns the host part of Addr().
func (m *Miniredis) Host() string {
m.Lock()
defer m.Unlock()
return m.srv.Addr().IP.String()
}
// Port returns the (random) port part of Addr().
func (m *Miniredis) Port() string {
m.Lock()
defer m.Unlock()
return strconv.Itoa(m.srv.Addr().Port)
}
// CommandCount returns the number of processed commands.
func (m *Miniredis) CommandCount() int {
m.Lock()
defer m.Unlock()
return int(m.srv.TotalCommands())
}
// CurrentConnectionCount returns the number of currently connected clients.
func (m *Miniredis) CurrentConnectionCount() int {
m.Lock()
defer m.Unlock()
return m.srv.ClientsLen()
}
// TotalConnectionCount returns the number of client connections since server start.
func (m *Miniredis) TotalConnectionCount() int {
m.Lock()
defer m.Unlock()
return int(m.srv.TotalConnections())
}
// FastForward decreases all TTLs by the given duration. All TTLs <= 0 will be
// expired.
func (m *Miniredis) FastForward(duration time.Duration) {
m.Lock()
defer m.Unlock()
for _, db := range m.dbs {
db.fastForward(duration)
}
}
// Server returns the underlying server to allow custom commands to be implemented
func (m *Miniredis) Server() *server.Server {
return m.srv
}
// Dump returns a text version of the selected DB, usable for debugging.
//
// Dump limits the maximum length of each key:value to "DumpMaxLineLen" characters.
// To increase that, call something like:
//
// miniredis.DumpMaxLineLen = 1024
// mr, _ = miniredis.Run()
// mr.Dump()
func (m *Miniredis) Dump() string {
m.Lock()
defer m.Unlock()
var (
maxLen = DumpMaxLineLen
indent = " "
db = m.db(m.selectedDB)
r = ""
v = func(s string) string {
suffix := ""
if len(s) > maxLen {
suffix = fmt.Sprintf("...(%d)", len(s))
s = s[:maxLen-len(suffix)]
}
return fmt.Sprintf("%q%s", s, suffix)
}
)
for _, k := range db.allKeys() {
r += fmt.Sprintf("- %s\n", k)
t := db.t(k)
switch t {
case keyTypeString:
r += fmt.Sprintf("%s%s\n", indent, v(db.stringKeys[k]))
case keyTypeHash:
for _, hk := range db.hashFields(k) {
r += fmt.Sprintf("%s%s: %s\n", indent, hk, v(db.hashGet(k, hk)))
}
case keyTypeList:
for _, lk := range db.listKeys[k] {
r += fmt.Sprintf("%s%s\n", indent, v(lk))
}
case keyTypeSet:
for _, mk := range db.setMembers(k) {
r += fmt.Sprintf("%s%s\n", indent, v(mk))
}
case keyTypeSortedSet:
for _, el := range db.ssetElements(k) {
r += fmt.Sprintf("%s%f: %s\n", indent, el.score, v(el.member))
}
case keyTypeStream:
for _, entry := range db.streamKeys[k].entries {
r += fmt.Sprintf("%s%s\n", indent, entry.ID)
ev := entry.Values
for i := 0; i < len(ev)/2; i++ {
r += fmt.Sprintf("%s%s%s: %s\n", indent, indent, v(ev[2*i]), v(ev[2*i+1]))
}
}
case keyTypeHll:
for _, entry := range db.hllKeys {
r += fmt.Sprintf("%s%s\n", indent, v(string(entry.Bytes())))
}
default:
r += fmt.Sprintf("%s(a %s, fixme!)\n", indent, t)
}
}
return r
}
// SetTime sets the time against which EXPIREAT values are compared, and the
// time used in stream entry IDs. Will use time.Now() if this is not set.
func (m *Miniredis) SetTime(t time.Time) {
m.Lock()
defer m.Unlock()
m.now = t
}
// make every command return this message. For example:
//
// LOADING Redis is loading the dataset in memory
// MASTERDOWN Link with MASTER is down and replica-serve-stale-data is set to 'no'.
//
// Clear it with an empty string. Don't add newlines.
func (m *Miniredis) SetError(msg string) {
cb := server.Hook(nil)
if msg != "" {
cb = func(c *server.Peer, cmd string, args ...string) bool {
c.WriteError(msg)
return true
}
}
m.srv.SetPreHook(cb)
}
// isValidCMD returns true if command is valid and can be executed.
func (m *Miniredis) isValidCMD(c *server.Peer, cmd string) bool {
if !m.handleAuth(c) {
return false
}
if m.checkPubsub(c, cmd) {
return false
}
return true
}
// handleAuth returns false if connection has no access. It sends the reply.
func (m *Miniredis) handleAuth(c *server.Peer) bool {
if getCtx(c).nested {
return true
}
m.Lock()
defer m.Unlock()
if len(m.passwords) == 0 {
return true
}
if !getCtx(c).authenticated {
c.WriteError("NOAUTH Authentication required.")
return false
}
return true
}
// handlePubsub sends an error to the user if the connection is in PUBSUB mode.
// It'll return true if it did.
func (m *Miniredis) checkPubsub(c *server.Peer, cmd string) bool {
if getCtx(c).nested {
return false
}
m.Lock()
defer m.Unlock()
ctx := getCtx(c)
if ctx.subscriber == nil {
return false
}
prefix := "ERR "
if strings.ToLower(cmd) == "exec" {
prefix = "EXECABORT Transaction discarded because of: "
}
c.WriteError(fmt.Sprintf(
"%sCan't execute '%s': only (P)SUBSCRIBE / (P)UNSUBSCRIBE / PING / QUIT are allowed in this context",
prefix,
strings.ToLower(cmd),
))
return true
}
func getCtx(c *server.Peer) *connCtx {
if c.Ctx == nil {
c.Ctx = &connCtx{}
}
return c.Ctx.(*connCtx)
}
func startTx(ctx *connCtx) {
ctx.transaction = []txCmd{}
ctx.dirtyTransaction = false
}
func stopTx(ctx *connCtx) {
ctx.transaction = nil
unwatch(ctx)
}
func inTx(ctx *connCtx) bool {
return ctx.transaction != nil
}
func addTxCmd(ctx *connCtx, cb txCmd) {
ctx.transaction = append(ctx.transaction, cb)
}
func watch(db *RedisDB, ctx *connCtx, key string) {
if ctx.watch == nil {
ctx.watch = map[dbKey]uint{}
}
ctx.watch[dbKey{db: db.id, key: key}] = db.keyVersion[key] // Can be 0.
}
func unwatch(ctx *connCtx) {
ctx.watch = nil
}
// setDirty can be called even when not in an tx. Is an no-op then.
func setDirty(c *server.Peer) {
if c.Ctx == nil {
// No transaction. Not relevant.
return
}
getCtx(c).dirtyTransaction = true
}
func (m *Miniredis) addSubscriber(s *Subscriber) {
m.subscribers[s] = struct{}{}
}
// closes and remove the subscriber.
func (m *Miniredis) removeSubscriber(s *Subscriber) {
_, ok := m.subscribers[s]
delete(m.subscribers, s)
if ok {
s.Close()
}
}
func (m *Miniredis) publish(c, msg string) int {
n := 0
for s := range m.subscribers {
n += s.Publish(c, msg)
}
return n
}
// enter 'subscribed state', or return the existing one.
func (m *Miniredis) subscribedState(c *server.Peer) *Subscriber {
ctx := getCtx(c)
sub := ctx.subscriber
if sub != nil {
return sub
}
sub = newSubscriber()
m.addSubscriber(sub)
c.OnDisconnect(func() {
m.Lock()
m.removeSubscriber(sub)
m.Unlock()
})
ctx.subscriber = sub
go monitorPublish(c, sub.publish)
go monitorPpublish(c, sub.ppublish)
return sub
}
// whenever the p?sub count drops to 0 subscribed state should be stopped, and
// all redis commands are allowed again.
func endSubscriber(m *Miniredis, c *server.Peer) {
ctx := getCtx(c)
if sub := ctx.subscriber; sub != nil {
m.removeSubscriber(sub) // will Close() the sub
}
ctx.subscriber = nil
}
// Start a new pubsub subscriber. It can (un) subscribe to channels and
// patterns, and has a channel to get published messages. Close it with
// Close().
// Does not close itself when there are no subscriptions left.
func (m *Miniredis) NewSubscriber() *Subscriber {
sub := newSubscriber()
m.Lock()
m.addSubscriber(sub)
m.Unlock()
return sub
}
func (m *Miniredis) allSubscribers() []*Subscriber {
var subs []*Subscriber
for s := range m.subscribers {
subs = append(subs, s)
}
return subs
}
func (m *Miniredis) Seed(seed int) {
m.Lock()
defer m.Unlock()
// m.rand is not safe for concurrent use.
m.rand = rand.New(rand.NewSource(int64(seed)))
}
func (m *Miniredis) randIntn(n int) int {
if m.rand == nil {
return rand.Intn(n)
}
return m.rand.Intn(n)
}
// shuffle shuffles a list of strings. Kinda.
func (m *Miniredis) shuffle(l []string) {
for range l {
i := m.randIntn(len(l))
j := m.randIntn(len(l))
l[i], l[j] = l[j], l[i]
}
}
func (m *Miniredis) effectiveNow() time.Time {
if !m.now.IsZero() {
return m.now
}
return time.Now().UTC()
}
// convert a unixtimestamp to a duration, to use an absolute time as TTL.
// d can be either time.Second or time.Millisecond.
func (m *Miniredis) at(i int, d time.Duration) time.Duration {
var ts time.Time
switch d {
case time.Millisecond:
ts = time.Unix(int64(i/1000), 1000000*int64(i%1000))
case time.Second:
ts = time.Unix(int64(i), 0)
default:
panic("invalid time unit (d). Fixme!")
}
now := m.effectiveNow()
return ts.Sub(now)
}
// copy does not mind if dst already exists.
func (m *Miniredis) copy(
srcDB *RedisDB, src string,
destDB *RedisDB, dst string,
) error {
if !srcDB.exists(src) {
return ErrKeyNotFound
}
switch srcDB.t(src) {
case keyTypeString:
destDB.stringKeys[dst] = srcDB.stringKeys[src]
case keyTypeHash:
destDB.hashKeys[dst] = copyHashKey(srcDB.hashKeys[src])
case keyTypeList:
destDB.listKeys[dst] = copyListKey(srcDB.listKeys[src])
case keyTypeSet:
destDB.setKeys[dst] = copySetKey(srcDB.setKeys[src])
case keyTypeSortedSet:
destDB.sortedsetKeys[dst] = copySortedSet(srcDB.sortedsetKeys[src])
case keyTypeStream:
destDB.streamKeys[dst] = srcDB.streamKeys[src].copy()
case keyTypeHll:
destDB.hllKeys[dst] = srcDB.hllKeys[src].copy()
default:
panic("missing case")
}
destDB.keys[dst] = srcDB.keys[src]
destDB.incr(dst)
if v, ok := srcDB.ttl[src]; ok {
destDB.ttl[dst] = v
}
return nil
}
func copyHashKey(orig hashKey) hashKey {
cpy := hashKey{}
for k, v := range orig {
cpy[k] = v
}
return cpy
}
func copyListKey(orig listKey) listKey {
cpy := make(listKey, len(orig))
copy(cpy, orig)
return cpy
}
func copySetKey(orig setKey) setKey {
cpy := setKey{}
for k, v := range orig {
cpy[k] = v
}
return cpy
}
func copySortedSet(orig sortedSet) sortedSet {
cpy := sortedSet{}
for k, v := range orig {
cpy[k] = v
}
return cpy
}
+60
View File
@@ -0,0 +1,60 @@
package miniredis
import (
"errors"
"math"
"strconv"
"time"
"github.com/alicebob/miniredis/v2/server"
)
// optInt parses an int option in a command.
// Writes "invalid integer" error to c if it's not a valid integer. Returns
// whether or not things were okay.
func optInt(c *server.Peer, src string, dest *int) bool {
return optIntErr(c, src, dest, msgInvalidInt)
}
func optIntErr(c *server.Peer, src string, dest *int, errMsg string) bool {
n, err := strconv.Atoi(src)
if err != nil {
setDirty(c)
c.WriteError(errMsg)
return false
}
*dest = n
return true
}
// optIntSimple sets dest or returns an error
func optIntSimple(src string, dest *int) error {
n, err := strconv.Atoi(src)
if err != nil {
return errors.New(msgInvalidInt)
}
*dest = n
return nil
}
func optDuration(c *server.Peer, src string, dest *time.Duration) bool {
n, err := strconv.ParseFloat(src, 64)
if err != nil {
setDirty(c)
c.WriteError(msgInvalidTimeout)
return false
}
if n < 0 {
setDirty(c)
c.WriteError(msgTimeoutNegative)
return false
}
if math.IsInf(n, 0) {
setDirty(c)
c.WriteError(msgTimeoutIsOutOfRange)
return false
}
*dest = time.Duration(n*1_000_000) * time.Microsecond
return true
}
+2
View File
@@ -0,0 +1,2 @@
test:
go test
+60
View File
@@ -0,0 +1,60 @@
package proto
import (
"bufio"
"crypto/tls"
"net"
)
type Client struct {
c net.Conn
r *bufio.Reader
}
func Dial(addr string) (*Client, error) {
c, err := net.Dial("tcp", addr)
if err != nil {
return nil, err
}
return &Client{
c: c,
r: bufio.NewReader(c),
}, nil
}
func DialTLS(addr string, cfg *tls.Config) (*Client, error) {
c, err := tls.Dial("tcp", addr, cfg)
if err != nil {
return nil, err
}
return &Client{
c: c,
r: bufio.NewReader(c),
}, nil
}
func (c *Client) Close() error {
return c.c.Close()
}
func (c *Client) Do(cmd ...string) (string, error) {
if err := Write(c.c, cmd); err != nil {
return "", err
}
return Read(c.r)
}
func (c *Client) Read() (string, error) {
return Read(c.r)
}
// Do() + ReadStrings()
func (c *Client) DoStrings(cmd ...string) ([]string, error) {
res, err := c.Do(cmd...)
if err != nil {
return nil, err
}
return ReadStrings(res)
}
+288
View File
@@ -0,0 +1,288 @@
package proto
import (
"bufio"
"errors"
"fmt"
"io"
"strconv"
"strings"
)
var (
ErrProtocol = errors.New("unsupported protocol")
ErrUnexpected = errors.New("not what you asked for")
)
func readLine(r *bufio.Reader) (string, error) {
line, err := r.ReadString('\n')
if err != nil {
return "", err
}
if len(line) < 3 {
return "", ErrProtocol
}
return line, nil
}
// Read an array, with all elements are the raw redis commands
// Also reads sets and maps.
func ReadArray(b string) ([]string, error) {
r := bufio.NewReader(strings.NewReader(b))
line, err := readLine(r)
if err != nil {
return nil, err
}
elems := 0
switch line[0] {
default:
return nil, ErrUnexpected
case '*', '>', '~':
// *: array
// >: push data
// ~: set
length, err := strconv.Atoi(line[1 : len(line)-2])
if err != nil {
return nil, err
}
elems = length
case '%':
// we also read maps.
length, err := strconv.Atoi(line[1 : len(line)-2])
if err != nil {
return nil, err
}
elems = length * 2
}
var res []string
for i := 0; i < elems; i++ {
next, err := Read(r)
if err != nil {
return nil, err
}
res = append(res, next)
}
return res, nil
}
func ReadString(b string) (string, error) {
r := bufio.NewReader(strings.NewReader(b))
line, err := readLine(r)
if err != nil {
return "", err
}
switch line[0] {
default:
return "", ErrUnexpected
case '$':
// bulk strings are: `$5\r\nhello\r\n`
length, err := strconv.Atoi(line[1 : len(line)-2])
if err != nil {
return "", err
}
if length < 0 {
// -1 is a nil response
return line, nil
}
var (
buf = make([]byte, length+2)
pos = 0
)
for pos < length+2 {
n, err := r.Read(buf[pos:])
if err != nil {
return "", err
}
pos += n
}
return string(buf[:len(buf)-2]), nil
}
}
func readInline(b string) (string, error) {
if len(b) < 3 {
return "", ErrUnexpected
}
return b[1 : len(b)-2], nil
}
func ReadError(b string) (string, error) {
if len(b) < 1 {
return "", ErrUnexpected
}
switch b[0] {
default:
return "", ErrUnexpected
case '-':
return readInline(b)
}
}
func ReadStrings(b string) ([]string, error) {
elems, err := ReadArray(b)
if err != nil {
return nil, err
}
var res []string
for _, e := range elems {
s, err := ReadString(e)
if err != nil {
return nil, err
}
res = append(res, s)
}
return res, nil
}
// Read a single command, returning it raw. Used to read replies from redis.
// Understands RESP3 proto.
func Read(r *bufio.Reader) (string, error) {
line, err := readLine(r)
if err != nil {
return "", err
}
switch line[0] {
default:
return "", ErrProtocol
case '+', '-', ':', ',', '_':
// +: inline string
// -: errors
// :: integer
// ,: float
// _: null
// Simple line based replies.
return line, nil
case '$':
// bulk strings are: `$5\r\nhello\r\n`
length, err := strconv.Atoi(line[1 : len(line)-2])
if err != nil {
return "", err
}
if length < 0 {
// -1 is a nil response
return line, nil
}
var (
buf = make([]byte, length+2)
pos = 0
)
for pos < length+2 {
n, err := r.Read(buf[pos:])
if err != nil {
return "", err
}
pos += n
}
return line + string(buf), nil
case '*', '>', '~':
// arrays are: `*6\r\n...`
// pushdata is: `>6\r\n...`
// sets are: `~6\r\n...`
length, err := strconv.Atoi(line[1 : len(line)-2])
if err != nil {
return "", err
}
for i := 0; i < length; i++ {
next, err := Read(r)
if err != nil {
return "", err
}
line += next
}
return line, nil
case '%':
// maps are: `%3\r\n...`
length, err := strconv.Atoi(line[1 : len(line)-2])
if err != nil {
return "", err
}
for i := 0; i < length*2; i++ {
next, err := Read(r)
if err != nil {
return "", err
}
line += next
}
return line, nil
}
}
// Write a command in RESP3 proto. Used to write commands to redis.
// Currently only supports string arrays.
func Write(w io.Writer, cmd []string) error {
if _, err := fmt.Fprintf(w, "*%d\r\n", len(cmd)); err != nil {
return err
}
for _, c := range cmd {
if _, err := fmt.Fprintf(w, "$%d\r\n%s\r\n", len(c), c); err != nil {
return err
}
}
return nil
}
// Parse into interfaces. `b` must contain exactly a single command (which can be nested).
func Parse(b string) (interface{}, error) {
if len(b) < 1 {
return nil, ErrUnexpected
}
switch b[0] {
default:
return "", ErrProtocol
case '+':
return readInline(b)
case '-':
e, err := readInline(b)
if err != nil {
return nil, err
}
return errors.New(e), nil
case ':':
e, err := readInline(b)
if err != nil {
return nil, err
}
return strconv.Atoi(e)
case '$':
return ReadString(b)
case '*':
elems, err := ReadArray(b)
if err != nil {
return nil, err
}
var res []interface{}
for _, elem := range elems {
e, err := Parse(elem)
if err != nil {
return nil, err
}
res = append(res, e)
}
return res, nil
case '%':
elems, err := ReadArray(b)
if err != nil {
return nil, err
}
var res = map[interface{}]interface{}{}
for len(elems) > 1 {
key, err := Parse(elems[0])
if err != nil {
return nil, err
}
value, err := Parse(elems[1])
if err != nil {
return nil, err
}
res[key] = value
elems = elems[2:]
}
return res, nil
}
}
+102
View File
@@ -0,0 +1,102 @@
package proto
import (
"fmt"
"strings"
)
// Byte-safe string
func String(s string) string {
return fmt.Sprintf("$%d\r\n%s\r\n", len(s), s)
}
// Inline string
func Inline(s string) string {
return inline('+', s)
}
// Error
func Error(s string) string {
return inline('-', s)
}
func inline(r rune, s string) string {
return fmt.Sprintf("%s%s\r\n", string(r), s)
}
// Int
func Int(n int) string {
return fmt.Sprintf(":%d\r\n", n)
}
// Float
func Float(n float64) string {
return fmt.Sprintf(",%g\r\n", n)
}
const (
Nil = "$-1\r\n"
NilResp3 = "_\r\n"
NilList = "*-1\r\n"
)
// Array assembles the args in a list. Args should be raw redis commands.
// Example: Array(String("foo"), String("bar"))
func Array(args ...string) string {
return fmt.Sprintf("*%d\r\n", len(args)) + strings.Join(args, "")
}
// Push assembles the args for push-data. Args should be raw redis commands.
// Example: Push(String("foo"), String("bar"))
func Push(args ...string) string {
return fmt.Sprintf(">%d\r\n", len(args)) + strings.Join(args, "")
}
// Strings is a helper to build 1 dimensional string arrays.
func Strings(args ...string) string {
var strings []string
for _, a := range args {
strings = append(strings, String(a))
}
return Array(strings...)
}
// Ints is a helper to build 1 dimensional int arrays.
func Ints(args ...int) string {
var ints []string
for _, a := range args {
ints = append(ints, Int(a))
}
return Array(ints...)
}
// Map assembles the args in a map. Args should be raw redis commands.
// Must be an even number of arguments.
// Example: Map(String("foo"), String("bar"))
func Map(args ...string) string {
return fmt.Sprintf("%%%d\r\n", len(args)/2) + strings.Join(args, "")
}
// StringMap is is a wrapper to get a map of (bulk)strings.
func StringMap(args ...string) string {
var strings []string
for _, a := range args {
strings = append(strings, String(a))
}
return Map(strings...)
}
// Set assembles the args in a map. Args should be raw redis commands.
// Example: Set(String("foo"), String("bar"))
func Set(args ...string) string {
return fmt.Sprintf("~%d\r\n", len(args)) + strings.Join(args, "")
}
// StringSet is is a wrapper to get a set of (bulk)strings.
func StringSet(args ...string) string {
var strings []string
for _, a := range args {
strings = append(strings, String(a))
}
return Set(strings...)
}
+240
View File
@@ -0,0 +1,240 @@
package miniredis
import (
"regexp"
"sort"
"sync"
"github.com/alicebob/miniredis/v2/server"
)
// PubsubMessage is what gets broadcasted over pubsub channels.
type PubsubMessage struct {
Channel string
Message string
}
type PubsubPmessage struct {
Pattern string
Channel string
Message string
}
// Subscriber has the (p)subscriptions.
type Subscriber struct {
publish chan PubsubMessage
ppublish chan PubsubPmessage
channels map[string]struct{}
patterns map[string]*regexp.Regexp
mu sync.Mutex
}
// Make a new subscriber. The channel is not buffered, so you will need to keep
// reading using Messages(). Use Close() when done, or unsubscribe.
func newSubscriber() *Subscriber {
return &Subscriber{
publish: make(chan PubsubMessage),
ppublish: make(chan PubsubPmessage),
channels: map[string]struct{}{},
patterns: map[string]*regexp.Regexp{},
}
}
// Close the listening channel
func (s *Subscriber) Close() {
close(s.publish)
close(s.ppublish)
}
// Count the total number of channels and patterns
func (s *Subscriber) Count() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.count()
}
func (s *Subscriber) count() int {
return len(s.channels) + len(s.patterns)
}
// Subscribe to a channel. Returns the total number of (p)subscriptions after
// subscribing.
func (s *Subscriber) Subscribe(c string) int {
s.mu.Lock()
defer s.mu.Unlock()
s.channels[c] = struct{}{}
return s.count()
}
// Unsubscribe a channel. Returns the total number of (p)subscriptions after
// unsubscribing.
func (s *Subscriber) Unsubscribe(c string) int {
s.mu.Lock()
defer s.mu.Unlock()
delete(s.channels, c)
return s.count()
}
// Subscribe to a pattern. Returns the total number of (p)subscriptions after
// subscribing.
func (s *Subscriber) Psubscribe(pat string) int {
s.mu.Lock()
defer s.mu.Unlock()
s.patterns[pat] = patternRE(pat)
return s.count()
}
// Unsubscribe a pattern. Returns the total number of (p)subscriptions after
// unsubscribing.
func (s *Subscriber) Punsubscribe(pat string) int {
s.mu.Lock()
defer s.mu.Unlock()
delete(s.patterns, pat)
return s.count()
}
// List all subscribed channels, in alphabetical order
func (s *Subscriber) Channels() []string {
s.mu.Lock()
defer s.mu.Unlock()
var cs []string
for c := range s.channels {
cs = append(cs, c)
}
sort.Strings(cs)
return cs
}
// List all subscribed patterns, in alphabetical order
func (s *Subscriber) Patterns() []string {
s.mu.Lock()
defer s.mu.Unlock()
var ps []string
for p := range s.patterns {
ps = append(ps, p)
}
sort.Strings(ps)
return ps
}
// Publish a message. Will return return how often we sent the message (can be
// a match for a subscription and for a psubscription.
func (s *Subscriber) Publish(c, msg string) int {
s.mu.Lock()
defer s.mu.Unlock()
found := 0
subs:
for sub := range s.channels {
if sub == c {
s.publish <- PubsubMessage{c, msg}
found++
break subs
}
}
pats:
for orig, pat := range s.patterns {
if pat != nil && pat.MatchString(c) {
s.ppublish <- PubsubPmessage{orig, c, msg}
found++
break pats
}
}
return found
}
// The channel to read messages for this subscriber. Only for messages matching
// a SUBSCRIBE.
func (s *Subscriber) Messages() <-chan PubsubMessage {
return s.publish
}
// The channel to read messages for this subscriber. Only for messages matching
// a PSUBSCRIBE.
func (s *Subscriber) Pmessages() <-chan PubsubPmessage {
return s.ppublish
}
// List all pubsub channels. If `pat` isn't empty channels names must match the
// pattern. Channels are returned alphabetically.
func activeChannels(subs []*Subscriber, pat string) []string {
channels := map[string]struct{}{}
for _, s := range subs {
for c := range s.channels {
channels[c] = struct{}{}
}
}
var cpat *regexp.Regexp
if pat != "" {
cpat = patternRE(pat)
}
var cs []string
for k := range channels {
if cpat != nil && !cpat.MatchString(k) {
continue
}
cs = append(cs, k)
}
sort.Strings(cs)
return cs
}
// Count all subscribed (not psubscribed) clients for the given channel
// pattern. Channels are returned alphabetically.
func countSubs(subs []*Subscriber, channel string) int {
n := 0
for _, p := range subs {
for c := range p.channels {
if c == channel {
n++
break
}
}
}
return n
}
// Count the total of all client psubscriptions.
func countPsubs(subs []*Subscriber) int {
n := 0
for _, p := range subs {
n += len(p.patterns)
}
return n
}
func monitorPublish(conn *server.Peer, msgs <-chan PubsubMessage) {
for msg := range msgs {
conn.Block(func(c *server.Writer) {
c.WritePushLen(3)
c.WriteBulk("message")
c.WriteBulk(msg.Channel)
c.WriteBulk(msg.Message)
c.Flush()
})
}
}
func monitorPpublish(conn *server.Peer, msgs <-chan PubsubPmessage) {
for msg := range msgs {
conn.Block(func(c *server.Writer) {
c.WritePushLen(4)
c.WriteBulk("pmessage")
c.WriteBulk(msg.Pattern)
c.WriteBulk(msg.Channel)
c.WriteBulk(msg.Message)
c.Flush()
})
}
}
+264
View File
@@ -0,0 +1,264 @@
package miniredis
import (
"context"
"fmt"
"math"
"math/big"
"strings"
"sync"
"time"
"github.com/alicebob/miniredis/v2/server"
)
const (
keyTypeString = "string"
keyTypeHash = "hash"
keyTypeList = "list"
keyTypeSet = "set"
keyTypeHll = "hll"
keyTypeSortedSet = "zset"
keyTypeStream = "stream"
)
const (
msgWrongType = "WRONGTYPE Operation against a key holding the wrong kind of value"
msgNotValidHllValue = "WRONGTYPE Key is not a valid HyperLogLog string value."
msgInvalidInt = "ERR value is not an integer or out of range"
msgIntOverflow = "ERR increment or decrement would overflow"
msgInvalidFloat = "ERR value is not a valid float"
msgInvalidMinMax = "ERR min or max is not a float"
msgInvalidRangeItem = "ERR min or max not valid string range item"
msgInvalidTimeout = "ERR timeout is not a float or out of range"
msgInvalidRange = "ERR value is out of range, must be positive"
msgSyntaxError = "ERR syntax error"
msgKeyNotFound = "ERR no such key"
msgOutOfRange = "ERR index out of range"
msgInvalidCursor = "ERR invalid cursor"
msgXXandNX = "ERR XX and NX options at the same time are not compatible"
msgTimeoutNegative = "ERR timeout is negative"
msgTimeoutIsOutOfRange = "ERR timeout is out of range"
msgInvalidSETime = "ERR invalid expire time in set"
msgInvalidSETEXTime = "ERR invalid expire time in setex"
msgInvalidPSETEXTime = "ERR invalid expire time in psetex"
msgInvalidKeysNumber = "ERR Number of keys can't be greater than number of args"
msgNegativeKeysNumber = "ERR Number of keys can't be negative"
msgFScriptUsage = "ERR unknown subcommand or wrong number of arguments for '%s'. Try SCRIPT HELP."
msgFScriptUsageSimple = "ERR unknown subcommand '%s'. Try SCRIPT HELP."
msgFPubsubUsage = "ERR unknown subcommand or wrong number of arguments for '%s'. Try PUBSUB HELP."
msgFPubsubUsageSimple = "ERR unknown subcommand '%s'. Try PUBSUB HELP."
msgFObjectUsage = "ERR unknown subcommand '%s'. Try OBJECT HELP."
msgScriptFlush = "ERR SCRIPT FLUSH only support SYNC|ASYNC option"
msgSingleElementPair = "ERR INCR option supports a single increment-element pair"
msgGTLTandNX = "ERR GT, LT, and/or NX options at the same time are not compatible"
msgInvalidStreamID = "ERR Invalid stream ID specified as stream command argument"
msgStreamIDTooSmall = "ERR The ID specified in XADD is equal or smaller than the target stream top item"
msgStreamIDZero = "ERR The ID specified in XADD must be greater than 0-0"
msgNoScriptFound = "NOSCRIPT No matching script. Please use EVAL."
msgUnsupportedUnit = "ERR unsupported unit provided. please use M, KM, FT, MI"
msgXreadUnbalanced = "ERR Unbalanced 'xread' list of streams: for each stream key an ID or '$' must be specified."
msgXgroupKeyNotFound = "ERR The XGROUP subcommand requires the key to exist. Note that for CREATE you may want to use the MKSTREAM option to create an empty stream automatically."
msgXtrimInvalidStrategy = "ERR unsupported XTRIM strategy. Please use MAXLEN, MINID"
msgXtrimInvalidMaxLen = "ERR value is not an integer or out of range"
msgXtrimInvalidLimit = "ERR syntax error, LIMIT cannot be used without the special ~ option"
msgDBIndexOutOfRange = "ERR DB index is out of range"
msgLimitCombination = "ERR syntax error, LIMIT is only supported in combination with either BYSCORE or BYLEX"
msgRankIsZero = "ERR RANK can't be zero: use 1 to start from the first match, 2 from the second ... or use negative to start from the end of the list"
msgCountIsNegative = "ERR COUNT can't be negative"
msgMaxLengthIsNegative = "ERR MAXLEN can't be negative"
msgLimitIsNegative = "ERR LIMIT can't be negative"
msgMemorySubcommand = "ERR unknown subcommand '%s'. Try MEMORY HELP."
)
func errWrongNumber(cmd string) string {
return fmt.Sprintf("ERR wrong number of arguments for '%s' command", strings.ToLower(cmd))
}
func errLuaParseError(err error) string {
return fmt.Sprintf("ERR Error compiling script (new function): %s", err.Error())
}
func errReadgroup(key, group string) error {
return fmt.Errorf("NOGROUP No such key '%s' or consumer group '%s'", key, group)
}
func errXreadgroup(key, group string) error {
return fmt.Errorf("NOGROUP No such key '%s' or consumer group '%s' in XREADGROUP with GROUP option", key, group)
}
func msgNotFromScripts(sha string) string {
return fmt.Sprintf("This Redis command is not allowed from script script: %s, &c", sha)
}
// withTx wraps the non-argument-checking part of command handling code in
// transaction logic.
func withTx(
m *Miniredis,
c *server.Peer,
cb txCmd,
) {
ctx := getCtx(c)
if ctx.nested {
// this is a call via Lua's .call(). It's already locked.
cb(c, ctx)
m.signal.Broadcast()
return
}
if inTx(ctx) {
addTxCmd(ctx, cb)
c.WriteInline("QUEUED")
return
}
m.Lock()
cb(c, ctx)
// done, wake up anyone who waits on anything.
m.signal.Broadcast()
m.Unlock()
}
// blockCmd is executed returns whether it is done
type blockCmd func(*server.Peer, *connCtx) bool
// blocking keeps trying a command until the callback returns true. Calls
// onTimeout after the timeout (or when we call this in a transaction).
func blocking(
m *Miniredis,
c *server.Peer,
timeout time.Duration,
cb blockCmd,
onTimeout func(*server.Peer),
) {
var (
ctx = getCtx(c)
)
if inTx(ctx) {
addTxCmd(ctx, func(c *server.Peer, ctx *connCtx) {
if !cb(c, ctx) {
onTimeout(c)
}
})
c.WriteInline("QUEUED")
return
}
localCtx, cancel := context.WithCancel(m.Ctx)
defer cancel()
timedOut := false
if timeout != 0 {
go setCondTimer(localCtx, m.signal, &timedOut, timeout)
}
go func() {
<-localCtx.Done()
m.signal.Broadcast() // main loop might miss this signal
}()
if !ctx.nested {
// this is a call via Lua's .call(). It's already locked.
m.Lock()
defer m.Unlock()
}
for {
if c.Closed() {
return
}
if m.Ctx.Err() != nil {
return
}
done := cb(c, ctx)
if done {
return
}
if timedOut {
onTimeout(c)
return
}
m.signal.Wait()
}
}
func setCondTimer(ctx context.Context, sig *sync.Cond, timedOut *bool, timeout time.Duration) {
dl := time.NewTimer(timeout)
defer dl.Stop()
select {
case <-dl.C:
sig.L.Lock() // for timedOut
*timedOut = true
sig.Broadcast() // main loop might miss this signal
sig.L.Unlock()
case <-ctx.Done():
}
}
// formatBig formats a float the way redis does
func formatBig(v *big.Float) string {
// Format with %f and strip trailing 0s.
if v.IsInf() {
return "inf"
}
// if math.IsInf(v, -1) {
// return "-inf"
// }
return stripZeros(fmt.Sprintf("%.17f", v))
}
func stripZeros(sv string) string {
for strings.Contains(sv, ".") {
if sv[len(sv)-1] != '0' {
break
}
// Remove trailing 0s.
sv = sv[:len(sv)-1]
// Ends with a '.'.
if sv[len(sv)-1] == '.' {
sv = sv[:len(sv)-1]
break
}
}
return sv
}
// redisRange gives Go offsets for something l long with start/end in
// Redis semantics. Both start and end can be negative.
// Used for string range and list range things.
// The results can be used as: v[start:end]
// Note that GETRANGE (on a string key) never returns an empty string when end
// is a large negative number.
func redisRange(l, start, end int, stringSymantics bool) (int, int) {
if start < 0 {
start = l + start
if start < 0 {
start = 0
}
}
if start > l {
start = l
}
if end < 0 {
end = l + end
if end < 0 {
end = -1
if stringSymantics {
end = 0
}
}
}
if end < math.MaxInt32 {
end++ // end argument is inclusive in Redis.
}
if end > l {
end = l
}
if end < start {
return 0, 0
}
return start, end
}
+9
View File
@@ -0,0 +1,9 @@
.PHONY: all build test
all: build test
build:
go build
test:
go test
+157
View File
@@ -0,0 +1,157 @@
package server
import (
"bufio"
"errors"
"strconv"
)
type Simple string
// ErrProtocol is the general error for unexpected input
var ErrProtocol = errors.New("invalid request")
// client always sends arrays with bulk strings
func readArray(rd *bufio.Reader) ([]string, error) {
line, err := rd.ReadString('\n')
if err != nil {
return nil, err
}
if len(line) < 3 {
return nil, ErrProtocol
}
switch line[0] {
default:
return nil, ErrProtocol
case '*':
l, err := strconv.Atoi(line[1 : len(line)-2])
if err != nil {
return nil, err
}
// l can be -1
var fields []string
for ; l > 0; l-- {
s, err := readString(rd)
if err != nil {
return nil, err
}
fields = append(fields, s)
}
return fields, nil
}
}
func readString(rd *bufio.Reader) (string, error) {
line, err := rd.ReadString('\n')
if err != nil {
return "", err
}
if len(line) < 3 {
return "", ErrProtocol
}
switch line[0] {
default:
return "", ErrProtocol
case '+', '-', ':':
// +: simple string
// -: errors
// :: integer
// Simple line based replies.
return string(line[1 : len(line)-2]), nil
case '$':
// bulk strings are: `$5\r\nhello\r\n`
length, err := strconv.Atoi(line[1 : len(line)-2])
if err != nil {
return "", err
}
if length < 0 {
// -1 is a nil response
return "", nil
}
var (
buf = make([]byte, length+2)
pos = 0
)
for pos < length+2 {
n, err := rd.Read(buf[pos:])
if err != nil {
return "", err
}
pos += n
}
return string(buf[:length]), nil
}
}
// parse a reply
func ParseReply(rd *bufio.Reader) (interface{}, error) {
line, err := rd.ReadString('\n')
if err != nil {
return nil, err
}
if len(line) < 3 {
return nil, ErrProtocol
}
switch line[0] {
default:
return nil, ErrProtocol
case '+':
// +: simple string
return Simple(line[1 : len(line)-2]), nil
case '-':
// -: errors
return nil, errors.New(string(line[1 : len(line)-2]))
case ':':
// :: integer
v := line[1 : len(line)-2]
if v == "" {
return 0, nil
}
n, err := strconv.Atoi(v)
if err != nil {
return nil, ErrProtocol
}
return n, nil
case '$':
// bulk strings are: `$5\r\nhello\r\n`
length, err := strconv.Atoi(line[1 : len(line)-2])
if err != nil {
return "", err
}
if length < 0 {
// -1 is a nil response
return nil, nil
}
var (
buf = make([]byte, length+2)
pos = 0
)
for pos < length+2 {
n, err := rd.Read(buf[pos:])
if err != nil {
return "", err
}
pos += n
}
return string(buf[:length]), nil
case '*':
// array
l, err := strconv.Atoi(line[1 : len(line)-2])
if err != nil {
return nil, ErrProtocol
}
// l can be -1
var fields []interface{}
for ; l > 0; l-- {
s, err := ParseReply(rd)
if err != nil {
return nil, err
}
fields = append(fields, s)
}
return fields, nil
}
}
+490
View File
@@ -0,0 +1,490 @@
package server
import (
"bufio"
"crypto/tls"
"fmt"
"net"
"strings"
"sync"
"unicode"
"github.com/alicebob/miniredis/v2/fpconv"
)
func errUnknownCommand(cmd string, args []string) string {
s := fmt.Sprintf("ERR unknown command `%s`, with args beginning with: ", cmd)
if len(args) > 20 {
args = args[:20]
}
for _, a := range args {
s += fmt.Sprintf("`%s`, ", a)
}
return s
}
// Cmd is what Register expects
type Cmd func(c *Peer, cmd string, args []string)
type DisconnectHandler func(c *Peer)
// Hook is can be added to run before every cmd. Return true if the command is done.
type Hook func(*Peer, string, ...string) bool
// Server is a simple redis server
type Server struct {
l net.Listener
cmds map[string]Cmd
preHook Hook
peers map[net.Conn]struct{}
mu sync.Mutex
wg sync.WaitGroup
infoConns int
infoCmds int
}
// NewServer makes a server listening on addr. Close with .Close().
func NewServer(addr string) (*Server, error) {
l, err := net.Listen("tcp", addr)
if err != nil {
return nil, err
}
return newServer(l), nil
}
func NewServerTLS(addr string, cfg *tls.Config) (*Server, error) {
l, err := tls.Listen("tcp", addr, cfg)
if err != nil {
return nil, err
}
return newServer(l), nil
}
func newServer(l net.Listener) *Server {
s := Server{
cmds: map[string]Cmd{},
peers: map[net.Conn]struct{}{},
l: l,
}
s.wg.Add(1)
go func() {
defer s.wg.Done()
s.serve(l)
s.mu.Lock()
for c := range s.peers {
c.Close()
}
s.mu.Unlock()
}()
return &s
}
// (un)set a hook which is ran before every call. It returns true if the command is done.
func (s *Server) SetPreHook(h Hook) {
s.mu.Lock()
s.preHook = h
s.mu.Unlock()
}
func (s *Server) serve(l net.Listener) {
for {
conn, err := l.Accept()
if err != nil {
return
}
s.ServeConn(conn)
}
}
// ServeConn handles a net.Conn. Nice with net.Pipe()
func (s *Server) ServeConn(conn net.Conn) {
s.wg.Add(1)
s.mu.Lock()
s.peers[conn] = struct{}{}
s.infoConns++
s.mu.Unlock()
go func() {
defer s.wg.Done()
defer conn.Close()
s.servePeer(conn)
s.mu.Lock()
delete(s.peers, conn)
s.mu.Unlock()
}()
}
// Addr has the net.Addr struct
func (s *Server) Addr() *net.TCPAddr {
s.mu.Lock()
defer s.mu.Unlock()
if s.l == nil {
return nil
}
return s.l.Addr().(*net.TCPAddr)
}
// Close a server started with NewServer. It will wait until all clients are
// closed.
func (s *Server) Close() {
s.mu.Lock()
if s.l != nil {
s.l.Close()
}
s.l = nil
s.mu.Unlock()
s.wg.Wait()
}
// Register a command. It can't have been registered before. Safe to call on a
// running server.
func (s *Server) Register(cmd string, f Cmd) error {
s.mu.Lock()
defer s.mu.Unlock()
cmd = strings.ToUpper(cmd)
if _, ok := s.cmds[cmd]; ok {
return fmt.Errorf("command already registered: %s", cmd)
}
s.cmds[cmd] = f
return nil
}
func (s *Server) servePeer(c net.Conn) {
r := bufio.NewReader(c)
peer := &Peer{
w: bufio.NewWriter(c),
}
defer func() {
for _, f := range peer.onDisconnect {
f()
}
}()
readCh := make(chan []string)
go func() {
defer close(readCh)
for {
args, err := readArray(r)
if err != nil {
peer.Close()
return
}
readCh <- args
}
}()
for args := range readCh {
s.Dispatch(peer, args)
peer.Flush()
if peer.Closed() {
c.Close()
}
}
}
func (s *Server) Dispatch(c *Peer, args []string) {
cmd, args := args[0], args[1:]
cmdUp := strings.ToUpper(cmd)
s.mu.Lock()
h := s.preHook
s.mu.Unlock()
if h != nil {
if h(c, cmdUp, args...) {
return
}
}
s.mu.Lock()
cb, ok := s.cmds[cmdUp]
s.mu.Unlock()
if !ok {
c.WriteError(errUnknownCommand(cmd, args))
return
}
s.mu.Lock()
s.infoCmds++
s.mu.Unlock()
cb(c, cmdUp, args)
if c.SwitchResp3 != nil {
c.Resp3 = *c.SwitchResp3
c.SwitchResp3 = nil
}
}
// TotalCommands is total (known) commands since this the server started
func (s *Server) TotalCommands() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.infoCmds
}
// ClientsLen gives the number of connected clients right now
func (s *Server) ClientsLen() int {
s.mu.Lock()
defer s.mu.Unlock()
return len(s.peers)
}
// TotalConnections give the number of clients connected since the server
// started, including the currently connected ones
func (s *Server) TotalConnections() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.infoConns
}
// Peer is a client connected to the server
type Peer struct {
w *bufio.Writer
closed bool
Resp3 bool
SwitchResp3 *bool // we'll switch to this version _after_ the command
Ctx interface{} // anything goes, server won't touch this
onDisconnect []func() // list of callbacks
mu sync.Mutex // for Block()
ClientName string // client name set by CLIENT SETNAME
}
func NewPeer(w *bufio.Writer) *Peer {
return &Peer{
w: w,
}
}
// Flush the write buffer. Called automatically after every redis command
func (c *Peer) Flush() {
c.mu.Lock()
defer c.mu.Unlock()
c.w.Flush()
}
// Close the client connection after the current command is done.
func (c *Peer) Close() {
c.mu.Lock()
defer c.mu.Unlock()
c.closed = true
}
// Return true if the peer connection closed.
func (c *Peer) Closed() bool {
c.mu.Lock()
defer c.mu.Unlock()
return c.closed
}
// Register a function to execute on disconnect. There can be multiple
// functions registered.
func (c *Peer) OnDisconnect(f func()) {
c.onDisconnect = append(c.onDisconnect, f)
}
// issue multiple calls, guarded with a mutex
func (c *Peer) Block(f func(*Writer)) {
c.mu.Lock()
defer c.mu.Unlock()
f(&Writer{c.w, c.Resp3})
}
// WriteError writes a redis 'Error'
func (c *Peer) WriteError(e string) {
c.Block(func(w *Writer) {
w.WriteError(e)
})
}
// WriteInline writes a redis inline string
func (c *Peer) WriteInline(s string) {
c.Block(func(w *Writer) {
w.WriteInline(s)
})
}
// WriteOK write the inline string `OK`
func (c *Peer) WriteOK() {
c.WriteInline("OK")
}
// WriteBulk writes a bulk string
func (c *Peer) WriteBulk(s string) {
c.Block(func(w *Writer) {
w.WriteBulk(s)
})
}
// WriteNull writes a redis Null element
func (c *Peer) WriteNull() {
c.Block(func(w *Writer) {
w.WriteNull()
})
}
// WriteLen starts an array with the given length
func (c *Peer) WriteLen(n int) {
c.Block(func(w *Writer) {
w.WriteLen(n)
})
}
// WriteMapLen starts a map with the given length (number of keys)
func (c *Peer) WriteMapLen(n int) {
c.Block(func(w *Writer) {
w.WriteMapLen(n)
})
}
// WriteSetLen starts a set with the given length (number of elements)
func (c *Peer) WriteSetLen(n int) {
c.Block(func(w *Writer) {
w.WriteSetLen(n)
})
}
// WritePushLen starts a push-data array with the given length
func (c *Peer) WritePushLen(n int) {
c.Block(func(w *Writer) {
w.WritePushLen(n)
})
}
// WriteInt writes an integer
func (c *Peer) WriteInt(n int) {
c.Block(func(w *Writer) {
w.WriteInt(n)
})
}
// WriteFloat writes a float
func (c *Peer) WriteFloat(n float64) {
c.Block(func(w *Writer) {
w.WriteFloat(n)
})
}
// WriteRaw writes a raw redis response
func (c *Peer) WriteRaw(s string) {
c.Block(func(w *Writer) {
w.WriteRaw(s)
})
}
// WriteStrings is a helper to (bulk)write a string list
func (c *Peer) WriteStrings(strs []string) {
c.Block(func(w *Writer) {
w.WriteStrings(strs)
})
}
func toInline(s string) string {
return strings.Map(func(r rune) rune {
if unicode.IsSpace(r) {
return ' '
}
return r
}, s)
}
// A Writer is given to the callback in Block()
type Writer struct {
w *bufio.Writer
resp3 bool
}
// WriteError writes a redis 'Error'
func (w *Writer) WriteError(e string) {
fmt.Fprintf(w.w, "-%s\r\n", toInline(e))
}
func (w *Writer) WriteLen(n int) {
fmt.Fprintf(w.w, "*%d\r\n", n)
}
func (w *Writer) WriteMapLen(n int) {
if w.resp3 {
fmt.Fprintf(w.w, "%%%d\r\n", n)
return
}
w.WriteLen(n * 2)
}
func (w *Writer) WriteSetLen(n int) {
if w.resp3 {
fmt.Fprintf(w.w, "~%d\r\n", n)
return
}
w.WriteLen(n)
}
func (w *Writer) WritePushLen(n int) {
if w.resp3 {
fmt.Fprintf(w.w, ">%d\r\n", n)
return
}
w.WriteLen(n)
}
// WriteBulk writes a bulk string
func (w *Writer) WriteBulk(s string) {
fmt.Fprintf(w.w, "$%d\r\n%s\r\n", len(s), s)
}
// WriteStrings writes a list of strings (bulk)
func (w *Writer) WriteStrings(strs []string) {
w.WriteLen(len(strs))
for _, s := range strs {
w.WriteBulk(s)
}
}
// WriteInt writes an integer
func (w *Writer) WriteInt(n int) {
fmt.Fprintf(w.w, ":%d\r\n", n)
}
// WriteFloat writes a float
func (w *Writer) WriteFloat(n float64) {
if w.resp3 {
fmt.Fprintf(w.w, ",%s\r\n", formatFloat(n))
return
}
w.WriteBulk(formatFloat(n))
}
// WriteNull writes a redis Null element
func (w *Writer) WriteNull() {
if w.resp3 {
fmt.Fprint(w.w, "_\r\n")
return
}
fmt.Fprintf(w.w, "$-1\r\n")
}
// WriteInline writes a redis inline string
func (w *Writer) WriteInline(s string) {
fmt.Fprintf(w.w, "+%s\r\n", toInline(s))
}
// WriteRaw writes a raw redis response
func (w *Writer) WriteRaw(s string) {
fmt.Fprint(w.w, s)
}
func (w *Writer) Flush() {
w.w.Flush()
}
// formatFloat formats a float the way redis does.
// Redis uses a method called "grisu2", which we ported from C.
func formatFloat(v float64) string {
return fpconv.Dtoa(v)
}
+2
View File
@@ -0,0 +1,2 @@
Credits to DmitriyVTitov on his package https://github.com/DmitriyVTitov/size
+138
View File
@@ -0,0 +1,138 @@
package size
import (
"reflect"
"unsafe"
)
// Of returns the size of 'v' in bytes.
// If there is an error during calculation, Of returns -1.
func Of(v interface{}) int {
// Cache with every visited pointer so we don't count two pointers
// to the same memory twice.
cache := make(map[uintptr]bool)
return sizeOf(reflect.Indirect(reflect.ValueOf(v)), cache)
}
// sizeOf returns the number of bytes the actual data represented by v occupies in memory.
// If there is an error, sizeOf returns -1.
func sizeOf(v reflect.Value, cache map[uintptr]bool) int {
switch v.Kind() {
case reflect.Array:
sum := 0
for i := 0; i < v.Len(); i++ {
s := sizeOf(v.Index(i), cache)
if s < 0 {
return -1
}
sum += s
}
return sum + (v.Cap()-v.Len())*int(v.Type().Elem().Size())
case reflect.Slice:
// return 0 if this node has been visited already
if cache[v.Pointer()] {
return 0
}
cache[v.Pointer()] = true
sum := 0
for i := 0; i < v.Len(); i++ {
s := sizeOf(v.Index(i), cache)
if s < 0 {
return -1
}
sum += s
}
sum += (v.Cap() - v.Len()) * int(v.Type().Elem().Size())
return sum + int(v.Type().Size())
case reflect.Struct:
sum := 0
for i, n := 0, v.NumField(); i < n; i++ {
s := sizeOf(v.Field(i), cache)
if s < 0 {
return -1
}
sum += s
}
// Look for struct padding.
padding := int(v.Type().Size())
for i, n := 0, v.NumField(); i < n; i++ {
padding -= int(v.Field(i).Type().Size())
}
return sum + padding
case reflect.String:
s := v.String()
hdr := (*reflect.StringHeader)(unsafe.Pointer(&s))
if cache[hdr.Data] {
return int(v.Type().Size())
}
cache[hdr.Data] = true
return len(s) + int(v.Type().Size())
case reflect.Ptr:
// return Ptr size if this node has been visited already (infinite recursion)
if cache[v.Pointer()] {
return int(v.Type().Size())
}
cache[v.Pointer()] = true
if v.IsNil() {
return int(reflect.New(v.Type()).Type().Size())
}
s := sizeOf(reflect.Indirect(v), cache)
if s < 0 {
return -1
}
return s + int(v.Type().Size())
case reflect.Bool,
reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64,
reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64,
reflect.Int, reflect.Uint,
reflect.Chan,
reflect.Uintptr,
reflect.Float32, reflect.Float64, reflect.Complex64, reflect.Complex128,
reflect.Func:
return int(v.Type().Size())
case reflect.Map:
// return 0 if this node has been visited already (infinite recursion)
if cache[v.Pointer()] {
return 0
}
cache[v.Pointer()] = true
sum := 0
keys := v.MapKeys()
for i := range keys {
val := v.MapIndex(keys[i])
// calculate size of key and value separately
sv := sizeOf(val, cache)
if sv < 0 {
return -1
}
sum += sv
sk := sizeOf(keys[i], cache)
if sk < 0 {
return -1
}
sum += sk
}
// Include overhead due to unused map buckets. 10.79 comes
// from https://golang.org/src/runtime/map.go.
return sum + int(v.Type().Size()) + int(float64(len(keys))*10.79)
case reflect.Interface:
return sizeOf(v.Elem(), cache) + int(v.Type().Size())
}
return -1
}
+98
View File
@@ -0,0 +1,98 @@
package miniredis
// The most KISS way to implement a sorted set. Luckily we don't care about
// performance that much.
import (
"sort"
)
type direction int
const (
unsorted direction = iota
asc
desc
)
type sortedSet map[string]float64
type ssElem struct {
score float64
member string
}
type ssElems []ssElem
type byScore ssElems
func (sse byScore) Len() int { return len(sse) }
func (sse byScore) Swap(i, j int) { sse[i], sse[j] = sse[j], sse[i] }
func (sse byScore) Less(i, j int) bool {
if sse[i].score != sse[j].score {
return sse[i].score < sse[j].score
}
return sse[i].member < sse[j].member
}
func newSortedSet() sortedSet {
return sortedSet{}
}
func (ss *sortedSet) card() int {
return len(*ss)
}
func (ss *sortedSet) set(score float64, member string) {
(*ss)[member] = score
}
func (ss *sortedSet) get(member string) (float64, bool) {
v, ok := (*ss)[member]
return v, ok
}
// elems gives the list of ssElem, ready to sort.
func (ss *sortedSet) elems() ssElems {
elems := make(ssElems, 0, len(*ss))
for e, s := range *ss {
elems = append(elems, ssElem{s, e})
}
return elems
}
func (ss *sortedSet) byScore(d direction) ssElems {
elems := ss.elems()
sort.Sort(byScore(elems))
if d == desc {
reverseElems(elems)
}
return ssElems(elems)
}
// rankByScore gives the (0-based) index of member, or returns false.
func (ss *sortedSet) rankByScore(member string, d direction) (int, bool) {
if _, ok := (*ss)[member]; !ok {
return 0, false
}
for i, e := range ss.byScore(d) {
if e.member == member {
return i, true
}
}
// Can't happen
return 0, false
}
func reverseSlice(o []string) {
for i := range make([]struct{}, len(o)/2) {
other := len(o) - 1 - i
o[i], o[other] = o[other], o[i]
}
}
func reverseElems(o ssElems) {
for i := range make([]struct{}, len(o)/2) {
other := len(o) - 1 - i
o[i], o[other] = o[other], o[i]
}
}
+507
View File
@@ -0,0 +1,507 @@
// Basic stream implementation.
package miniredis
import (
"errors"
"fmt"
"math"
"sort"
"strconv"
"strings"
"sync"
"time"
)
// a Stream is a list of entries, lowest ID (oldest) first, and all "groups".
type streamKey struct {
entries []StreamEntry
groups map[string]*streamGroup
lastAllocatedID string
mu sync.Mutex
}
// a StreamEntry is an entry in a stream. The ID is always of the form
// "123-123".
// Values is an ordered list of key-value pairs.
type StreamEntry struct {
ID string
Values []string
}
type streamGroup struct {
stream *streamKey
lastID string
pending []pendingEntry
consumers map[string]*consumer
}
type consumer struct {
numPendingEntries int
// these timestamps aren't tracked perfectly
lastSeen time.Time // "idle" XINFO key
lastSuccess time.Time // "inactive" XINFO key
}
type pendingEntry struct {
id string
consumer string
deliveryCount int
lastDelivery time.Time
}
func newStreamKey() *streamKey {
return &streamKey{
groups: map[string]*streamGroup{},
}
}
// generateID doesn't lock the mutex
func (s *streamKey) generateID(now time.Time) string {
ts := uint64(now.UnixNano()) / 1_000_000
next := fmt.Sprintf("%d-%d", ts, 0)
if s.lastAllocatedID != "" && streamCmp(s.lastAllocatedID, next) >= 0 {
last, _ := parseStreamID(s.lastAllocatedID)
next = fmt.Sprintf("%d-%d", last[0], last[1]+1)
}
lastID := s.lastIDUnlocked()
if streamCmp(lastID, next) >= 0 {
last, _ := parseStreamID(lastID)
next = fmt.Sprintf("%d-%d", last[0], last[1]+1)
}
s.lastAllocatedID = next
return next
}
// lastID locks the mutex
func (s *streamKey) lastID() string {
s.mu.Lock()
defer s.mu.Unlock()
return s.lastIDUnlocked()
}
// lastID doesn't lock the mutex
func (s *streamKey) lastIDUnlocked() string {
if len(s.entries) == 0 {
return "0-0"
}
return s.entries[len(s.entries)-1].ID
}
func (s *streamKey) copy() *streamKey {
s.mu.Lock()
defer s.mu.Unlock()
cpy := &streamKey{
entries: s.entries,
}
groups := map[string]*streamGroup{}
for k, v := range s.groups {
gr := v.copy()
gr.stream = cpy
groups[k] = gr
}
cpy.groups = groups
return cpy
}
func parseStreamID(id string) ([2]uint64, error) {
var (
res [2]uint64
err error
)
parts := strings.SplitN(id, "-", 2)
res[0], err = strconv.ParseUint(parts[0], 10, 64)
if err != nil {
return res, errors.New(msgInvalidStreamID)
}
if len(parts) == 2 {
res[1], err = strconv.ParseUint(parts[1], 10, 64)
if err != nil {
return res, errors.New(msgInvalidStreamID)
}
}
return res, nil
}
// compares two stream IDs (of the full format: "123-123"). Returns: -1, 0, 1
// The given IDs should be valid stream IDs.
func streamCmp(a, b string) int {
ap, _ := parseStreamID(a)
bp, _ := parseStreamID(b)
switch {
case ap[0] < bp[0]:
return -1
case ap[0] > bp[0]:
return 1
case ap[1] < bp[1]:
return -1
case ap[1] > bp[1]:
return 1
default:
return 0
}
}
// formatStreamID makes a full id ("42-42") out of a partial one ("42")
func formatStreamID(id string) (string, error) {
var ts [2]uint64
parts := strings.SplitN(id, "-", 2)
if len(parts) > 0 {
p, err := strconv.ParseUint(parts[0], 10, 64)
if err != nil {
return "", errInvalidEntryID
}
ts[0] = p
}
if len(parts) > 1 {
p, err := strconv.ParseUint(parts[1], 10, 64)
if err != nil {
return "", errInvalidEntryID
}
ts[1] = p
}
return fmt.Sprintf("%d-%d", ts[0], ts[1]), nil
}
func formatStreamRangeBound(id string, start bool, reverse bool) (string, error) {
if id == "-" {
return "0-0", nil
}
if id == "+" {
return fmt.Sprintf("%d-%d", uint64(math.MaxUint64), uint64(math.MaxUint64)), nil
}
if id == "0" {
return "0-0", nil
}
parts := strings.Split(id, "-")
if len(parts) == 2 {
return formatStreamID(id)
}
// Incomplete IDs case
ts, err := strconv.ParseUint(parts[0], 10, 64)
if err != nil {
return "", errInvalidEntryID
}
if (!start && !reverse) || (start && reverse) {
return fmt.Sprintf("%d-%d", ts, uint64(math.MaxUint64)), nil
}
return fmt.Sprintf("%d-%d", ts, 0), nil
}
func reversedStreamEntries(o []StreamEntry) []StreamEntry {
newStream := make([]StreamEntry, len(o))
for i, e := range o {
newStream[len(o)-i-1] = e
}
return newStream
}
func (s *streamKey) createGroup(group, id string) error {
s.mu.Lock()
defer s.mu.Unlock()
if _, ok := s.groups[group]; ok {
return errors.New("BUSYGROUP Consumer Group name already exists")
}
if id == "$" {
id = s.lastIDUnlocked()
}
s.groups[group] = &streamGroup{
stream: s,
lastID: id,
consumers: map[string]*consumer{},
}
return nil
}
// streamAdd adds an entry to a stream. Returns the new entry ID.
// If id is empty or "*" the ID will be generated automatically.
// `values` should have an even length.
func (s *streamKey) add(entryID string, values []string, now time.Time) (string, error) {
s.mu.Lock()
defer s.mu.Unlock()
if entryID == "" || entryID == "*" {
entryID = s.generateID(now)
}
entryID, err := formatStreamID(entryID)
if err != nil {
return "", err
}
if entryID == "0-0" {
return "", errors.New(msgStreamIDZero)
}
if streamCmp(s.lastIDUnlocked(), entryID) != -1 {
return "", errors.New(msgStreamIDTooSmall)
}
s.entries = append(s.entries, StreamEntry{
ID: entryID,
Values: values,
})
return entryID, nil
}
func (s *streamKey) trim(n int) {
s.mu.Lock()
defer s.mu.Unlock()
if len(s.entries) > n {
s.entries = s.entries[len(s.entries)-n:]
}
}
// trimBefore deletes entries with an id less than the provided id
// and returns the number of entries deleted
func (s *streamKey) trimBefore(id string) int {
s.mu.Lock()
var delete []string
for _, entry := range s.entries {
if streamCmp(entry.ID, id) < 0 {
delete = append(delete, entry.ID)
} else {
break
}
}
s.mu.Unlock()
s.delete(delete)
return len(delete)
}
// all entries after "id"
func (s *streamKey) after(id string) []StreamEntry {
s.mu.Lock()
defer s.mu.Unlock()
pos := sort.Search(len(s.entries), func(i int) bool {
return streamCmp(id, s.entries[i].ID) < 0
})
return s.entries[pos:]
}
// get a stream entry by ID
// Also returns the position in the entries slice, if found.
func (s *streamKey) get(id string) (int, *StreamEntry) {
s.mu.Lock()
defer s.mu.Unlock()
pos := sort.Search(len(s.entries), func(i int) bool {
return streamCmp(id, s.entries[i].ID) <= 0
})
if len(s.entries) <= pos || s.entries[pos].ID != id {
return 0, nil
}
return pos, &s.entries[pos]
}
func (g *streamGroup) readGroup(
now time.Time,
consumerID,
id string,
count int,
noack bool,
) []StreamEntry {
if id == ">" {
// undelivered messages
msgs := g.stream.after(g.lastID)
if len(msgs) == 0 {
return nil
}
if count > 0 && len(msgs) > count {
msgs = msgs[:count]
}
if !noack {
shouldAppend := len(g.pending) == 0
for _, msg := range msgs {
if !shouldAppend {
shouldAppend = streamCmp(msg.ID, g.pending[len(g.pending)-1].id) == 1
}
var entry *pendingEntry
if shouldAppend {
g.pending = append(g.pending, pendingEntry{})
entry = &g.pending[len(g.pending)-1]
} else {
var pos int
pos, entry = g.searchPending(msg.ID)
if entry == nil {
g.pending = append(g.pending[:pos+1], g.pending[pos:]...)
entry = &g.pending[pos]
} else {
g.consumers[entry.consumer].numPendingEntries--
}
}
*entry = pendingEntry{
id: msg.ID,
consumer: consumerID,
deliveryCount: 1,
lastDelivery: now,
}
}
}
if _, ok := g.consumers[consumerID]; !ok {
g.consumers[consumerID] = &consumer{}
}
g.consumers[consumerID].numPendingEntries += len(msgs)
g.lastID = msgs[len(msgs)-1].ID
return msgs
}
// re-deliver messages from the pending list.
// con := gr.consumers[consumerID]
msgs := g.pendingAfter(id)
var res []StreamEntry
for i, p := range msgs {
if p.consumer != consumerID {
continue
}
_, entry := g.stream.get(p.id)
// not found. Weird?
if entry == nil {
continue
}
p.deliveryCount += 1
p.lastDelivery = now
msgs[i] = p
res = append(res, *entry)
}
return res
}
func (g *streamGroup) searchPending(id string) (int, *pendingEntry) {
pos := sort.Search(len(g.pending), func(i int) bool {
return streamCmp(id, g.pending[i].id) <= 0
})
if pos >= len(g.pending) || g.pending[pos].id != id {
return pos, nil
}
return pos, &g.pending[pos]
}
func (g *streamGroup) ack(ids []string) (int, error) {
count := 0
for _, id := range ids {
if _, err := parseStreamID(id); err != nil {
return 0, errors.New(msgInvalidStreamID)
}
pos, entry := g.searchPending(id)
if entry == nil {
continue
}
consumer := g.consumers[entry.consumer]
consumer.numPendingEntries--
g.pending = append(g.pending[:pos], g.pending[pos+1:]...)
// don't count deleted entries
if _, e := g.stream.get(id); e == nil {
continue
}
count++
}
return count, nil
}
func (s *streamKey) delete(ids []string) (int, error) {
count := 0
for _, id := range ids {
if _, err := parseStreamID(id); err != nil {
return 0, errors.New(msgInvalidStreamID)
}
i, entry := s.get(id)
if entry == nil {
continue
}
s.entries = append(s.entries[:i], s.entries[i+1:]...)
count++
}
return count, nil
}
func (g *streamGroup) pendingAfterOrEqual(id string) []pendingEntry {
pos := sort.Search(len(g.pending), func(i int) bool {
return streamCmp(id, g.pending[i].id) <= 0
})
return g.pending[pos:]
}
func (g *streamGroup) pendingAfter(id string) []pendingEntry {
pos := sort.Search(len(g.pending), func(i int) bool {
return streamCmp(id, g.pending[i].id) < 0
})
return g.pending[pos:]
}
func (g *streamGroup) pendingCount(consumer string) int {
n := 0
for _, p := range g.activePending() {
if p.consumer == consumer {
n++
}
}
return n
}
// pending entries without the entries deleted from the group
func (g *streamGroup) activePending() []pendingEntry {
var pe []pendingEntry
for _, p := range g.pending {
// drop deleted ones
if _, e := g.stream.get(p.id); e == nil {
continue
}
p := p
pe = append(pe, p)
}
return pe
}
func (g *streamGroup) copy() *streamGroup {
cns := map[string]*consumer{}
for k, v := range g.consumers {
c := *v
cns[k] = &c
}
return &streamGroup{
// don't copy stream
lastID: g.lastID,
pending: g.pending,
consumers: cns,
}
}
func (g *streamGroup) setLastSeen(c string, t time.Time) {
cons, ok := g.consumers[c]
if !ok {
cons = &consumer{}
}
cons.lastSeen = t
g.consumers[c] = cons
}
func (g *streamGroup) setLastSuccess(c string, t time.Time) {
g.setLastSeen(c, t)
g.consumers[c].lastSuccess = t
}