Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import zips #173

Draft
wants to merge 16 commits into
base: develop
Choose a base branch
from
Draft

Import zips #173

wants to merge 16 commits into from

Conversation

ghost
Copy link

@ghost ghost commented Aug 21, 2019

This is just a draft for early discussion, it is dependent on #172 and will likely need changes.

Expect force pushes while work continues on text imports.


  • I have signed the CLA

Rob Loranger added 8 commits August 16, 2019 14:27
this adds basic support for importing files as blog posts.

.txt and .md are supported at this time and the
collection is selectable, defaulting to draft.

if a collection is specified the post is federated.
different template action for partial or complete import success
handler for post request to import is now under /api/me/import
form target updated

also allow all plaintext files in form
this changes the import handler to use CreatePost instead of
CreateOwnedPost which required generation of non expiring access tokens
in favor of library side generation to support zip files
this updates to parse the time from the imported file, using v0.1.1 of
the wfimport library
@ghost
Copy link
Author

ghost commented Aug 22, 2019

the only thing really missing, other than pretty front end things, is importing just .txt files. that's a quick change though, a new ZipFunc to filter on only .txt and use FromZipDirsFunc instead.

@joyeusenoelle
Copy link
Contributor

Hang on, Rob, we may be duplicating effort. I believe I have zip imports working in https://github.com/writeas/wp-import/tree/writefreely-import .

@ghost
Copy link
Author

ghost commented Aug 22, 2019

We have indeed, rather I have. I should have jumped over there before writing the zip part.

I think we can merge the general purpose zip parsing logic into writeas/import, not right away but eventually. My goal with import was to have very general purpose utilities to get posts from different files.

Then we will have a library for parsing posts from various files and formats. We can still then have a wp-import tool and also use it in writefreely itself. Then later we could add import functionality into the CLI as well, and others can create their own tools if they wish.

@joyeusenoelle
Copy link
Contributor

That was my eventual goal as well, if I'm understanding you correctly, so we're on the same page.

A note, while we're talking about it: although Medium's export format is ZIP, it needs a little more finagling than Write.as/WriteFreely's ZIPs, since there's a bunch of stuff in there we don't care about and the posts are in HTML format.

Do we have existing code, or a recommended library, for converting HTML to markdown? I don't remember offhand.

@ghost
Copy link
Author

ghost commented Aug 22, 2019

👍

In the import library there is a function FromZipDirsByFunc which allows filtering based on any logic you wish, including parsing HTML to markdown, or skipping certain files.

So we could add, say FromZipMedium or similar, and that could use a ZipFunc to filter and process accordingly.

AFAIK there is not any existing code we have for conversion, there is probably a nice library somewhere.

@ghost
Copy link
Author

ghost commented Aug 26, 2019

form is now dynamic depending on files selected, only for users with javascript. those with it disabled see everything needed to understand and operate the form.

see discussion on phabricator for video and screenshots.

now checking for and returning invalid content type errors
Rob Loranger added 4 commits August 26, 2019 15:04
this allows zip files in the import upload form

plain files are still uploaded to the selected collection but zip
contents are considered drafts with the exception of subdirectories,
these are added to the corresponding collections. if no collection is
found then it will be created.
by default all content is enabled and displayed, this allows users
without javascript enabled to still use the form and understand what is
happening.

users with javascript enabled will see actions and options
disabled until the files selected allow, and show information and
warnings about zip files in needed.
Rob Loranger added 3 commits August 26, 2019 15:10
user feedback logic was updated to report if zero posts were found in a
zip and form submissions disable the submit button until the form input
for files changes again, preventing possible duplicate submissions on
large zip uploads.

updated to v0.2.1 wfimport to prevent early error returns when an
invalid file is present in a zip.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant