Created
February 8, 2020 13:52
-
-
Save cblgh/787e78a3479a4b0801bee2768933ffda to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
18:29:19 karissa | noffle: relatedly, i want to get your thoughts on how to think about kappa-cores/hypercores with media attached. we did some brainstorming on it in | |
| the cabal meeting | |
18:29:34 noffle | like, media in the hypercore? | |
18:29:46 karissa | https://hackmd.io/BeZ49Q2ARZavH1v2JbbJKw# | |
18:30:04 karissa | under 'how do we do binary stuff in cabal? ideas below ' | |
18:30:06 * | noffle reads | |
18:30:35 noffle | wow, really good notes | |
18:32:02 karissa | yeah it was a fun meeting! | |
18:32:32 karissa | seeing a pattern here that could overlap between mapeo and cabal --> media attached to messages/media attached to observations | |
18:32:52 noffle | nod | |
18:32:58 karissa | with also some user control around 'sparseness' of it | |
18:33:10 noffle | I like the idea of keeping blobs in seprate hypercores | |
18:33:59 karissa | wondering if message -> blob store or device(user) -> blob store makes more sense. i think mapeo uses the latter right now | |
18:33:59 noffle | one cost that can seem invisible is that you need to manage a swarm around EACH hypercore, so with N blobs, if you do 1 blob per hypercore that's N | |
| swarms. but if it was, say, per-user, that's M swarms (assuming M users) | |
18:34:26 noffle | maintaining an ongoing membership in a swarm costs network connections | |
18:34:27 karissa | yeah that's a limitation i was thinking of too. | |
18:34:37 karissa | which makes me lean towards device/user-> blob store | |
18:34:54 noffle | what we could do is: one *multifeed* of blobs per cabal | |
18:34:59 noffle | so it's still M hypercores | |
18:35:03 noffle | but one swarm for blobs | |
18:35:08 karissa | oh nice | |
18:35:08 noffle | one swarm for chat data | |
18:35:38 noffle | it means expsoing sparse replication on multifeed | |
18:35:43 noffle | at some point | |
18:35:52 karissa | will putting blob data in a hypercore get slow after a while? i think hyperdrive writes it to an external blob store right? | |
18:36:08 noffle | slow how? | |
18:36:12 noffle | like, appending locally? syncing? | |
18:36:40 noffle | the hyperdb-based hyperdrive writes to a hypercore | |
18:36:47 karissa | not sure exactly where the slowdown happens, but I remember this being a key point of how hyperdrive can handle upwards of GB/TB | |
18:36:47 noffle | not sure about the single-writer version | |
18:36:58 noffle | oh interesting, hadn't heard about this | |
18:37:06 karissa | i think it writes the ranges to the hypercore but not the blob data | |
18:37:11 karissa | kinda like sqlite blobs | |
18:37:26 noffle | is this a hypercore issue or hyperdrive issue? | |
18:38:09 karissa | exactly thats the question kinda raised in the cabal meeting, whether to have a 'media message' link to a hyperdrive | |
18:38:14 karissa | or to a hypercore | |
18:38:33 karissa | cause you can imagine wanting to share like, a folder or bunch of images | |
18:38:40 karissa | or having a user account or cabal have the 'media' | |
18:39:12 noffle | oh sorry I meant is that scaling issue on hypercore or hyperdrive? | |
18:40:03 karissa | ah i think it's a limitation of a merkle dag, which is why git is slow, iirc | |
18:40:31 noffle | huh. wouldn't that be a limit on # of entries then, not on total amount of data stored? | |
18:40:36 noffle | one merkle entry per node | |
18:40:47 noffle | node/file | |
18:40:54 noffle | would be interested in more bg on that | |
18:41:09 karissa | yeah it'd be great to have this documentation on hypercore | |
18:41:16 karissa | perhaps that's something you and i could work on one day | |
18:41:23 noffle | it sounds like something that hyper{core,drive} wants to fix, so maybe it's OK to rely on it & expect it to be fixed upstream as dat grows | |
18:41:23 karissa | 'book sprint' | |
18:41:31 noffle | I like docs :) | |
18:42:16 karissa | noffle: sorry i meant its not really a limitation in hyperdrive as it's architected, and it can handle lots of data -- it does currently have an | |
| issue with tons of *files* though, (like, thousands/hundreds of thousands in the same level) | |
18:42:19 noffle | yeah maybe we could allowing one-blob links like "KEY@SEQ" but also directory shares, which are maybe just "KEY/HASH" | |
18:42:31 * | noffle nod | |
18:42:42 noffle | "we could allow" | |
18:43:17 noffle | hyperblob://<KEY|HASH> and dat://<KEY/HASH> maybe | |
18:43:18 karissa | noffle: it would be cool to imagine media messages as just links to the outside world, and if a client supports reading it, that's great. so we can | |
| evolve this and try different things | |
18:43:22 noffle | yeah! | |
18:43:30 noffle | unixy | |
18:43:47 noffle | ha we should've had this chat in #cabal-club | |
18:43:48 karissa | yeah. also keep messages as tiny as possible in case a particular cabal gets super popular | |
18:43:54 noffle | yes | |
18:44:12 karissa | haha. well anyway, it relates to Mapeo because I *think* we want to do similar kinds of stuff. and probably other apps.. | |
18:45:05 noffle | nod | |
18:45:06 noffle | good chat | |
18:45:30 noffle | I'm going to go over these meeting minutes & will include these notes | |
18:45:36 karissa | rad |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment