fub

The problem

A Mastodon server keeps all entries it “knows” in its own database. That means that every entry received by every other federate social media server is stored in the database. That database can get quite big, and as someone has to pay for all of the storage costs, the database can’t be allowed to grow unchecked. Which is why Mastodon has a configuration that specifies after how many days external posts (so posts that originated on other servers) are deleted from the database. It’s in Administration | Server Settings | Content retention, as the setting “Content cache retention period”. On MuBlog, this is set to 100 — so after 100 days, posts from other servers are deleted. (Posts that were made on MuBlog are not under this policy, they keep existing.)

If you have bookmarked a post, which you do on your ‘home server’, you are actually bookmarking the copy of the post that is in your server’s database. And you can see where this is going: after the content cache retention period has expired, the local copy is removed — and with that, you have lost your bookmark as well! But the original link, on the originating server, still exists — so if you had a way to preserve your Mastodon bookmarks somewhere else, that would also record the original entry URL, you’d be safe.

Step one: Get Shaarli

I decided I wanted something to move my Mastodon bookmarks to an external bookmark manager, to preserve them. As I am a fan of self-hosted solutions, I selected Shaarli as my bookmark-manager. It is written in PHP and doesn’t use an external database, which means you can just drop the files through FTP on your webhost, open the page in your browser and configure. I installed it on my host in a separate directory.

After it all works (play around with it for a bit to get used to it!), you must go to Tools | Configure, and enable the REST API — this is how we will create bookmarks in Shaarli! Take a note of the REST API secret: this is how we will authorise ourselves to Shaarli.

Step two: Prepare your Mastodon account

Our script also has to authenticate itself with your Mastodon server in order to read your bookmarks, as those are not available to the public!

On the server, go to Preferences and then click on Development. Click on the button ‘New Application’. Give the application a name that makes sense to you (I used ‘Bookmark importer’), and enter the URL — you might want to use the URL where you installed Shaarli. Keep the default Redirect URI, since we don’t need a redirect. Then, set the correct Scope: the things the application is allowed to do. The only scopes the application needs are read:bookmarks and write:bookmarks. I suggest you only select these (deselecting the defaults of the ‘Read’ and ‘Write’ categories and ‘Follow’).

You now see the list of applications you have created. Click on the name of your bookmark-sync application and take note of the access token. This is how we will authorise our script to the Mastodon server.

Step three: The script

I have shared my script that does the syncing here. It’s in PHP, and I recommend that you try the script out from the commandline to make sure that it all works. Linux distributions often come with PHP CLI (the Ubuntu package is called php-cli), Windows users will have to find it out for themselves. Your PHP install should have at least the cURL and JSON modules included, but every self-respecting PHP version will have that out of the box. If not: good luck, figure it out.

Take the script, and fill out the four global constants at the top of the file: your Mastodon server, the Shaarli secret, the Mastodon token and the Shaarli endpoint. It is now ready to run.

The script will iterate through your bookmarks on Mastodon, create the bookmark in Shaarli, and delete the bookmark on Mastodon. Of course, you might not want to do all of that right on the first try, so there are three arguments you can give on the commandline. DONT_DELETE won’t delete the bookmarks from Mastodon, DONT_ADD won’t add the bookmarks to Shaarli and DONT_ITERATE will only take the first five bookmarks from Mastodon.

I would suggest the following order:

php mastobookmark.php DONT_ITERATE DONT_ADD DONT_DELETE, to check if the connection to the Mastodon server works;

php mastobookmark.php DONT_DELETE, to check if the connection to Shaarli works.

Note that the bookmarks are created in Shaarli with the tag source:mastodon, and any tag the original post might have. All of the bookmarks are also created as private bookmarks. The URL for the bookmark will be the URL of the bookmark on your home server. If you get around to reading the bookmark before it expires, you can read it on your home server and reply, like or boost directly. The original URL of the bookmark is also recorded, so if the entry has already expired on your home server, you can use the original link to get to the post after all. Interacting with the post will be a bit more complicated, but still possible.

Step four: Schedule the script

When you are satisfied that everything works, you can schedule running the script. If you have web hosting (which you have, because you used it to install Shaarli on) then there is a good chance you have the possibility to run cronjobs. (‘cron’ being the name of the server process that schedules scripts on Unix systems.) You can use FTP to upload the script somewhere — I strongly advise to not do this in the tree of your webserver (anything under public_html on my host, might be something else on yours, do check!), where it is accessible through any browser, but somewhere else — and then configure the cronjob to call it on a schedule that makes sense to you. Personally, I run it every day at 5 past midnight, but YMMV.

Optional: Tweak Shaarli

It irked me that Shaarli does not open the links of the bookmarks in a new tab by default. I use the default template, so I edited the file tpl/default/linklist.html in my Shaarli tree. That file is not HTML, but a template that gets interpreted. Find the link to “{$value.real_url}” with the CSS class linklist-real-url and add target="_blank" to the link tag. This will open the bookmarked links in a new tab. Mind you that links in the description of the bookmark (which includes the link on the original server!) will not be opened in a new tab. I’m sure that’s possible to do with some post-processing, but I’m too lazy to find out how to do that.

In closing

I made this script, and you are welcome to use it. I will not offer support, and if you mess things up, I won’t help you fix it. If you don’t know what you’re doing, it’s probably best not to do it. You’re welcome to suggest improvements, but I am under no obligation to implement them for you. I’m sure my coding style is horrible, I don’t care.

Good luck.

Crossposted from my blog. Comment here or at the original post.