Send out ActivityPub mentions (T627) #195

qwazix · 2019-10-09T06:35:03Z

This pull request adds code that scans the text of each post to be published for handles of the type @username@domain.tld, uses webfinger to ask ActivityPub endpoints about them, gets their inbox and actor IRI's and adds mention tags to the post. Finally it adds the users to the CC list of the recipients of the activity.

This works in pleroma but not in mastodon unless we set the object type to "Note" instead of "Article"

T627

be Note to work but I will open an issue for them and see what their reaction will be.

thebaer · 2019-10-09T12:12:17Z

Latest changes look good 👍 Confirmed that mentions work with Pleroma.

to the last because mastodon doesn't like https://my.instance/@me but https://my.instance/users/me

intermediate page (WIP) that shows the user profile page url

there it updates the table to include the handle. migrations WIP

thebaer

Just a few changes to make. Also getting this error during compilation:

./database.go:2476:61: remoteActor.GetSharedInbox undefined (type activityserve.RemoteActor has no field or method GetSharedInbox)
./database.go:2477:157: remoteActor.GetSharedInbox undefined (type activityserve.RemoteActor has no field or method GetSharedInbox)

routes.go

sqlite.sql

schema.sql

collections.go

activitypub.go

qwazix · 2019-10-10T16:59:18Z

Ack to all, will do.

thebaer · 2019-10-10T17:01:56Z

Also, just saw the comments on the Mastodon issue. Is that fixed on our end?

qwazix · 2019-10-10T17:13:25Z

Yes it works as expected, Gargron was right, I was thrown off because the first alias that webfinger returned was the @ version.

thebaer · 2019-10-10T18:23:15Z

Awesome -- works great with Mastodon 👍

Did some more testing and crashed the app with this post. Logs:

panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 7763 [running]:
github.com/writeas/activityserve.NewRemoteActor(0xc0009a1860, 0x22, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/mattbaer/projects/go/src/github.com/writeas/activityserve/remoteActor.go:34 +0x41e
github.com/writeas/writefreely.(*datastore).getProfilePageFromHandle(0xc000687f00, 0xc000021ef0, 0xc000e20dcf, 0x15, 0xc000e6ee60, 0x2, 0xa, 0xc0009a0570)
	/home/mattbaer/projects/go/src/github.com/writeas/writefreely/database.go:2472 +0x11a
github.com/writeas/writefreely.(*PublicPost).ActivityObject(0xc00059e540, 0xc000021ef0, 0x1)
	/home/mattbaer/projects/go/src/github.com/writeas/writefreely/posts.go:1120 +0xc67
github.com/writeas/writefreely.federatePost(0xc000021ef0, 0xc00059e540, 0x65, 0x2cd9a00, 0x0, 0x0)
	/home/mattbaer/projects/go/src/github.com/writeas/writefreely/activitypub.go:574 +0xf6
created by github.com/writeas/writefreely.newPost
	/home/mattbaer/projects/go/src/github.com/writeas/writefreely/posts.go:593 +0x792

(#195)

qwazix · 2019-10-11T07:09:39Z

The crash was due to prismo and pixelfed not having sharedInboxes. I made GetSharedInbox() to return the inbox of the user so implementation won't need to care whether there is a sharedInbox or not (this is in activityserve)

webfinger.go

(#195 (comment))

This fixes the `undefined: strings.ReplaceAll` build error.

thebaer · 2019-11-26T19:55:13Z

Got everything up to date with develop; this is almost ready to merge! Last pieces I'm thinking about:

Maybe using a different path for remote users. Would just like something short and clean.
Maybe changing the style of mention links. Initial thoughts are to make the domain part of a user's handle faded out a little (but not completely hidden like on Mastodon). I'm also open to any ideas anyone has here

thebaer · 2019-11-26T20:01:00Z

Ah, one more crash caused by this post:

panic: interface conversion: interface {} is nil, not string
goroutine 1153 [running]:
github.com/writeas/activityserve.NewRemoteActor(0xc000d83e00, 0x33, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/matt/go/pkg/mod/github.com/writeas/activityserve@v0.0.0-20191011072627-3a81f7784d5b/remoteActor.go:37 +0x444
github.com/writeas/writefreely.(*datastore).getProfilePageFromHandle(0xc0006acb40, 0xc000085f40, 0xc00139a44f, 0x1c, 0xc000aaec80, 0x2, 0xa, 0xc000d82bc0)
        /home/matt/go/src/github.com/writeas/writefreely/database.go:2506 +0x11a
github.com/writeas/writefreely.(*PublicPost).ActivityObject(0xc000512240, 0xc000085f40, 0x1)
        /home/matt/go/src/github.com/writeas/writefreely/posts.go:1177 +0xc6c
github.com/writeas/writefreely.federatePost(0xc000085f40, 0xc000512240, 0x1, 0x29fbd00, 0x0, 0x0)
        /home/matt/go/src/github.com/writeas/writefreely/activitypub.go:614 +0xf6
created by github.com/writeas/writefreely.newPost
        /home/matt/go/src/github.com/writeas/writefreely/posts.go:612 +0x81f

And the mention links in that post cause a 500 and panic:

ERROR: 2019/11/26 19:58:17 handle.go:469: User: devlog
interface conversion: interface {} is nil, not string: goroutine 204 [running]:
runtime/debug.Stack(0xc000085f40, 0xc000579100, 0xc0004dbd80)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/writeas/writefreely.(*Handler).Web.func1.1.1(0xc0006a6d20, 0xc000579100, 0x1936e40, 0xc0000e01c0, 0xc000515960, 0xbf6f7ed657a0817f, 0x18d536fd42, 0x29fbda0)
        /home/matt/go/src/github.com/writeas/writefreely/handle.go:469 +0x16f
panic(0x161a700, 0xc000628ba0)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/writeas/activityserve.NewRemoteActor(0xc0005a2db0, 0x25, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/matt/go/pkg/mod/github.com/writeas/activityserve@v0.0.0-20191011072627-3a81f7784d5b/remoteActor.go:37 +0x444
github.com/writeas/writefreely.(*datastore).getProfilePageFromHandle(0xc0006a6b40, 0xc000085f40, 0xc00048006d, 0xe, 0xc000443048, 0x403faa, 0x15d7020, 0x191bbb0)
        /home/matt/go/src/github.com/writeas/writefreely/database.go:2506 +0x11a
github.com/writeas/writefreely.handleViewMention(0xc000085f40, 0x1936e40, 0xc0000e01c0, 0xc000579100, 0xc000085180, 0x0)
        /home/matt/go/src/github.com/writeas/writefreely/collections.go:847 +0x9f
github.com/writeas/writefreely.(*Handler).Web.func1.1(0xc0006a6d20, 0xc000579100, 0x1936e40, 0xc0000e01c0, 0x17f67d8, 0x17f6930, 0x0, 0x0)
        /home/matt/go/src/github.com/writeas/writefreely/handle.go:501 +0x166
github.com/writeas/writefreely.(*Handler).Web.func1(0x1936e40, 0xc0000e01c0, 0xc000579100)
        /home/matt/go/src/github.com/writeas/writefreely/handle.go:515 +0x63
net/http.HandlerFunc.ServeHTTP(0xc0006a7180, 0x1936e40, 0xc0000e01c0, 0xc000579100)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0000da240, 0x1936e40, 0xc0000e01c0, 0xc000395e00)
        /home/matt/go/pkg/mod/github.com/gorilla/mux@v1.7.0/mux.go:212 +0xe3
net/http.serverHandler.ServeHTTP(0xc0006c2a90, 0x1936e40, 0xc0000e01c0, 0xc000395e00)
        /usr/local/go/src/net/http/server.go:2774 +0xa8
net/http.(*conn).serve(0xc000706a00, 0x1938a00, 0xc000625240)
        /usr/local/go/src/net/http/server.go:1878 +0x851
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2884 +0x2f4
2019/11/26 19:58:17 Web deferred internal error render
2019/11/26 19:58:17 "GET /mention:@matt@write.as" 500 324.540684ms "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"

qwazix · 2019-11-27T08:01:20Z

This is already fixed in upstream activityserve but that one uses upstream httpsig and not writeas/httpsig (I made those changes as a result of the review of pherephone) so I can't update it. Should I make the required changes to writefreely to support upstream httpsig?

qwazix · 2019-11-27T08:12:22Z

It turns out this is not necessary. If you do

go get github.com/go-fed/httpsig@master
go get -u github.com/writeas/activityserve

everything should work.

However the new httpsig removes the burden of setting the digest header manually and signs that header too so it might be a good idea to use that throughout.

thebaer · 2019-11-27T14:22:58Z

Awesome, just verified that that fixes the issue. I'll update go.mod, and we can work on moving to httpsig in the future.

Did more testing with this post. Last issue I'll address:

Invalid mentions link takes user to a blank page. I'll make it so user gets better feedback ("Profile not found").

For the future, a few notes:

Funkwhale's response:

2019/11/27 14:08:05 Status  : 403 Forbidden
2019/11/27 14:08:05 Response: {"detail":"{'summary': [ErrorDetail(string='This field may not be blank.', code='blank')]}"

Funkwhale mentions link to raw Actor data. Not sure if this is something we need to eventually fix on our end, though (e.g. in our case, when web visitors go to the raw actor URL, WF redirects them to the web-based profile).
PeerTube's response: 403 Forbidden
Pherephone mention link works correctly 👍
When POSTing to the Pherephone inbox, maybe it should also return a 403 Forbidden like PeerTube and Funkwhale? Not sure.
microblog.pub seems to work correctly (haven't tested on receiving end), returning a 201 Created

dariusk · 2019-11-27T19:41:17Z

Funkwhale mentions link to raw Actor data. Not sure if this is something we need to eventually fix on our end, though

My two cents: this is an issue on their end, they should redirect based on the request headers if it's a request from a web browser. Especially since they don't provide a link to the human-rendering URL in the raw JSON data.

thebaer · 2019-11-27T20:23:24Z

@dariusk Good point -- just opened an issue for this: funkwhale#979.

This fixes activityserve crashes caused by mentioning WriteFreely instances.

@me

Previously, links would go to /user/mention:@me@this.tld instead of /mention:@me@this.tld

gytisrepecka · 2019-12-12T15:36:19Z

Maybe changing the style of mention links. Initial thoughts are to make the domain part of a user's handle faded out a little (but not completely hidden like on Mastodon). I'm also open to any ideas anyone has here

Wouldn't suggest hiding domain part - it would be the same as hiding domain part in e-mail address. Perhaps only makes sense to show only username when it is on the same instance (domain), but that wouldn't be the case - I believe in most scenarios users will be mentioning Mastodon account while actually posting from WriteFreely blog.

- Export the func - Remove commented-out code - Use log, not fmt for debug messages - Remove named return parameters - Use standard var naming schemes - Fix spacing in queries and remove unnecessary chars

by making them absolute, not relative.

This notifies the user that the remote user doesn't exist, instead of showing a blank page. Ref T627

from /mention: Ref T627

Ref T627

thebaer · 2020-02-08T18:11:41Z

I merged our latest changes, did some cleanup, moved the remote user route to /@/ instead of /mention:, and changed it so the user gets visual feedback if they mention a remote user that doesn't exist.

I think there's still some work to be done around the datastore.GetProfilePageFromHandle() func, particularly when it comes to users that don't exist. In short, we need to be able to communicate that information back up through the stack, and really we shouldn't have all that network logic inside a database function. So it still needs a lot of refactoring and logic clean up. But we can address that later. For now, this gets the job done. Merging now!

Michael Demetriou added 2 commits October 8, 2019 15:58

Mentioning pleroma accounts works! Mastodon still needs the type to b

dccfae7

be Note to work but I will open an issue for them and see what their reaction will be.

Fix @thebaer's comments in dccfae7#commitcomment-35410380

3eb638b

qwazix mentioned this pull request Oct 9, 2019

Mentions don't work for federated "Articles" mastodon/mastodon#12129

Closed

Michael Demetriou added 4 commits October 10, 2019 10:59

Change the result that webfinger returns from the first alias

e5bbd45

to the last because mastodon doesn't like https://my.instance/@me but https://my.instance/users/me

Handles are saved in remoteusers while the links take you to an

99bb771

intermediate page (WIP) that shows the user profile page url

Redirects from the intermediate page work and if there's an old mention

db14f04

there it updates the table to include the handle. migrations WIP

Fix missing commit statement in migrations/v3.go

bc2016f

thebaer requested changes Oct 10, 2019

View reviewed changes

routes.go Outdated Show resolved Hide resolved

sqlite.sql Outdated Show resolved Hide resolved

schema.sql Outdated Show resolved Hide resolved

collections.go Outdated Show resolved Hide resolved

activitypub.go Outdated Show resolved Hide resolved

thebaer added this to the 0.11 milestone Oct 10, 2019

Fix comments on T627 pull request

b9d2689

(#195)

Update dependencies and add a comment

972ec00

thebaer requested changes Oct 14, 2019

View reviewed changes

webfinger.go Outdated Show resolved Hide resolved

Unmarshal to webfinger.Resource instead of interface{}

1bda043

(#195 (comment))

thebaer modified the milestones: 0.11, 0.12 Nov 8, 2019

thebaer added 2 commits November 26, 2019 13:19

Merge branch 'develop' into activitypub-mentions

d8df158

Bump Travis build to Go 1.12

bbb7b28

This fixes the `undefined: strings.ReplaceAll` build error.

thebaer approved these changes Nov 26, 2019

View reviewed changes

thebaer added 4 commits November 27, 2019 16:37

Update httpsig and activityserve

181af8c

This fixes activityserve crashes caused by mentioning WriteFreely instances.

Fix mention URL on multi-user instances

ae5bbd2

Previously, links would go to /user/mention:@me@this.tld instead of /mention:@me@this.tld

Merge branch 'hotfix-0.11.2' into activitypub-mentions

a513c99

Merge branch 'markdown-API' into activitypub-mentions

8c1bf2d

thebaer added 8 commits February 8, 2020 11:51

Merge branch 'develop' into activitypub-mentions

68d63d3

Clean up getProfilePageFromHandle

bb63e64

- Export the func - Remove commented-out code - Use log, not fmt for debug messages - Remove named return parameters - Use standard var naming schemes - Fix spacing in queries and remove unnecessary chars

Fix mention links

81edb73

by making them absolute, not relative.

Show 404 when remote user not found

867eb53

This notifies the user that the remote user doesn't exist, instead of showing a blank page. Ref T627

Move remote user URL to /@/

eac2231

from /mention: Ref T627

Add u-url class and span in mention link

4570511

Ref T627

Fix error logging format in RemoteLookup

ca4b0ac

Add TODOs for improving GetProfilePageFromHandle()

9589612

thebaer merged commit 9be05ef into develop Feb 8, 2020

thebaer deleted the activitypub-mentions branch February 8, 2020 18:13

thebaer mentioned this pull request Mar 18, 2020

Clean up ActivityPub mentions #282

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Send out ActivityPub mentions (T627) #195

Send out ActivityPub mentions (T627) #195

qwazix commented Oct 9, 2019

thebaer commented Oct 9, 2019

thebaer left a comment

qwazix commented Oct 10, 2019

thebaer commented Oct 10, 2019

qwazix commented Oct 10, 2019

thebaer commented Oct 10, 2019

qwazix commented Oct 11, 2019

thebaer commented Nov 26, 2019

thebaer commented Nov 26, 2019

qwazix commented Nov 27, 2019

qwazix commented Nov 27, 2019

thebaer commented Nov 27, 2019

dariusk commented Nov 27, 2019

thebaer commented Nov 27, 2019

gytisrepecka commented Dec 12, 2019

thebaer commented Feb 8, 2020

Navigation Menu

Send out ActivityPub mentions (T627) #195

Send out ActivityPub mentions (T627) #195

Conversation

qwazix commented Oct 9, 2019

thebaer commented Oct 9, 2019

thebaer left a comment

Choose a reason for hiding this comment

qwazix commented Oct 10, 2019

thebaer commented Oct 10, 2019

qwazix commented Oct 10, 2019

thebaer commented Oct 10, 2019

qwazix commented Oct 11, 2019

thebaer commented Nov 26, 2019

thebaer commented Nov 26, 2019

qwazix commented Nov 27, 2019

qwazix commented Nov 27, 2019

thebaer commented Nov 27, 2019

dariusk commented Nov 27, 2019

thebaer commented Nov 27, 2019

gytisrepecka commented Dec 12, 2019

thebaer commented Feb 8, 2020