Packaging issue: "panic: read pages: is a directory"

[tokudan]

Describe the bug

I'm trying to package writefreely for Nixpkgs but having trouble due to the mixed build systems.
I've got a working binary and I'm copying the {pages,static,templates} directory from your released binaries to verify my built binary.
The writefreely binary that I've built seems to be working fine for configuring (writefreely --config works), but after linking in the {pages,static,templates} directories, writefreely refuses to start and instead panics:

[user@host:/tmp/tmp.IoaQUiyMpS]$ l
insgesamt 136K
drwx------  3 user users  160 15. Mär 13:09 .
drwxrwxrwt 25 root   root   620 15. Mär 13:08 ..
-rw-r--r--  1 user users  977 15. Mär 11:40 config.ini
drwx------  2 user users  100 15. Mär 11:40 keys
lrwxrwxrwx  1 user users   72 15. Mär 13:09 pages -> /nix/store/0z59069wc0jf4s82fza7x12c05rg0bh9-writefreely-0.11.2/lib/pages
lrwxrwxrwx  1 user users   73 15. Mär 13:09 static -> /nix/store/0z59069wc0jf4s82fza7x12c05rg0bh9-writefreely-0.11.2/lib/static
lrwxrwxrwx  1 user users   76 15. Mär 13:09 templates -> /nix/store/0z59069wc0jf4s82fza7x12c05rg0bh9-writefreely-0.11.2/lib/templates
-rw-r--r--  1 user users 132K 15. Mär 11:40 writefreely.db

[user@host:/tmp/tmp.IoaQUiyMpS]$ l pages/
insgesamt 62K
dr-xr-xr-x 2 root root   12  1. Jan 1970  .
dr-xr-xr-x 5 root root    5  1. Jan 1970  ..
-r--r--r-- 5 root root  226  1. Jan 1970  404-general.tmpl
-r--r--r-- 5 root root  515  1. Jan 1970  404.tmpl
-r--r--r-- 5 root root  294  1. Jan 1970  410.tmpl
-r--r--r-- 5 root root  649  1. Jan 1970  500.tmpl
-r--r--r-- 5 root root 1,4K  1. Jan 1970  about.tmpl
-r--r--r-- 5 root root  149  1. Jan 1970  blank.tmpl
-r--r--r-- 5 root root 5,3K  1. Jan 1970  landing.tmpl
-r--r--r-- 5 root root 1,4K  1. Jan 1970  login.tmpl
-r--r--r-- 5 root root  312  1. Jan 1970  privacy.tmpl
-r--r--r-- 5 root root 4,7K  1. Jan 1970  signup.tmpl

[user@host:/tmp/tmp.IoaQUiyMpS]$ /nix/store/0z59069wc0jf4s82fza7x12c05rg0bh9-writefreely-0.11.2/bin/writefreely -debug
2020/03/15 13:10:17 Starting WriteFreely 0.11.2...
2020/03/15 13:10:17 Loading config.ini configuration...
2020/03/15 13:10:17 Loading templates...
2020/03/15 13:10:17   templates/bare.tmpl
2020/03/15 13:10:17   templates/base.tmpl
2020/03/15 13:10:17   templates/chorus-collection-post.tmpl
2020/03/15 13:10:17   templates/chorus-collection.tmpl
2020/03/15 13:10:17   templates/collection-post.tmpl
2020/03/15 13:10:17   templates/collection-tags.tmpl
2020/03/15 13:10:17   templates/collection.tmpl
2020/03/15 13:10:17   templates/edit-meta.tmpl
2020/03/15 13:10:17   templates/pad.tmpl
2020/03/15 13:10:17   templates/password-collection.tmpl
2020/03/15 13:10:17   templates/post.tmpl
2020/03/15 13:10:17   templates/read.tmpl
2020/03/15 13:10:17 Loading pages...
2020/03/15 13:10:17   [pages] pages
panic: read pages: is a directory

goroutine 1 [running]:
html/template.Must(...)
        /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-go-1.13.8/share/go/src/html/template/template.go:372
github.com/writeas/writefreely.initPage(0x0, 0x0, 0x1889b31, 0x5, 0x1889b31, 0x5)
        /build/source/templates.go:87 +0x4d7
github.com/writeas/writefreely.InitTemplates.func1(0x1889b31, 0x5, 0x1a376a0, 0xc00026ed00, 0x0, 0x0, 0x4d9849, 0xc00026ed00)
        /build/source/templates.go:129 +0xb4
path/filepath.walk(0x1889b31, 0x5, 0x1a376a0, 0xc00026ed00, 0xc0004b7d38, 0x0, 0x5)
        /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-go-1.13.8/share/go/src/path/filepath/path.go:358 +0x425
path/filepath.Walk(0x1889b31, 0x5, 0xc0004b7d38, 0x1889b31, 0x5)
        /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-go-1.13.8/share/go/src/path/filepath/path.go:404 +0xff
github.com/writeas/writefreely.InitTemplates(0xc00014a1c0, 0xc0000ea640, 0x0)
        /build/source/templates.go:126 +0x259
github.com/writeas/writefreely.Initialize(0x1a35200, 0xc0000ea640, 0x1, 0xc0004b7f40, 0x1, 0x1)
        /build/source/app.go:355 +0x66
main.main()
        /build/source/cmd/writefreely/main.go:117 +0x640

The error message does not make any sense for me. Of course pages is a directory. What's the actual issue here or how can I find out?

Expected behavior

Writefreely should have started

Application configuration

• Single mode or Multi-user mode? Single mode
• Database? sqlite
• Open registration? no
• Federation enabled? yes

Version or last commit: 0.11.2

[thebaer]

Are you able to get WF running on NixOS with the normal setup instructions? Just to rule out any application issues.

Based on the debug logs, it's trying to read pages when it should be skipping it and going to the files inside of it. Here are my logs when WF successfully starts:

2020/03/15 08:59:48 Loading pages...
2020/03/15 08:59:48   [404-general.tmpl] pages/404-general.tmpl
2020/03/15 08:59:48   [404.tmpl] pages/404.tmpl
2020/03/15 08:59:48   [410.tmpl] pages/410.tmpl
2020/03/15 08:59:48   [500.tmpl] pages/500.tmpl
2020/03/15 08:59:48   [about.tmpl] pages/about.tmpl
2020/03/15 08:59:48   [blank.tmpl] pages/blank.tmpl
2020/03/15 08:59:48   [landing.tmpl] pages/landing.tmpl
2020/03/15 08:59:48   [login.tmpl] pages/login.tmpl
2020/03/15 08:59:48   [privacy.tmpl] pages/privacy.tmpl
2020/03/15 08:59:48   [signup-oauth.tmpl] pages/signup-oauth.tmpl
2020/03/15 08:59:48   [signup.tmpl] pages/signup.tmpl

My first thought is that it might have to do with the symbolic link to the pages dir. But you might also try using the pages_parent_dir field in the [server] config section to see if that helps.

[tokudan]

Binaries built for other Linux systems need to have their binaries patched to work on NixOS, so they won't work out of the box.

The symlinks actually seem to be causing the issue here. I set the three options

templates_parent_dir = /nix/store/0z59069wc0jf4s82fza7x12c05rg0bh9-writefreely-0.11.2/lib
static_parent_dir    = /nix/store/0z59069wc0jf4s82fza7x12c05rg0bh9-writefreely-0.11.2/lib
pages_parent_dir     = /nix/store/0z59069wc0jf4s82fza7x12c05rg0bh9-writefreely-0.11.2/lib

Not supporting symlinks is somewhat unexpected for me, but I'll be able to work around that for nixpkgs/NixOS.

Does it make sense to drill into why symlinks are not supported? If not, I'd consider this issue closed.

[thebaer]

It looks like the filepath.Walk Go func we use simply doesn't follow symbolic links (see those docs). And there doesn't seem to be a way to check for symlinks as we load up the templates so we might prevent this panic or show a better error. So I'd say if the built-in config options work for you, yeah we can close this.

[diamondburned]

Just a 2021 update: if the code switches to using io/fs.WalkDir, then this should work:

WalkDir does not follow symbolic links found in directories, but if root itself is a symbolic link, its target will be walked.